Monitor system resources during runtime and automatically adjust settings

Created by: Zapotek

• Add Arachni::Supervisor (or something) to take system readings every few time intervals, or intervals based on system actions (like page audit, browser-cluster completed job etc.).
  • Should accept conditions (based on Proc objects probably) which will be evaluated at those intervals to gauge whether or not action needs to be taken.
• Conditions should be added by component-specific supervisors, for example Framework::Supervisor, BrowserCluster::Supervisor, etc.

The general idea is to monitor RAM utilization during scans and perform the following actions when resources start to run low:

• Adjust HTTP request queue size.
• Adjust buffer sizes to dump more entries to disk.
• Adjust cache sizes.
• Adjust the amount of browser workers.
• ...and pretty much anything else that can affect resource utilization.

The inverse should also be true, i.e. increasing lowered buffers to their default size once system health has been restored -- let's not ignore outside influences, Arachni will not be the only process on the machine.