Connections in CLOSE_WAIT cause a hang

Created by: ordinaehs

I started a scan on a remote website, only to come back the next day with the OS (Kali Linux up-to-date; I was using Arachni 1.5.1 downloaded from the website) in an almost unusable state. Memory usage was extreme (one of the Ruby instances was using more than 2G of memory), sending the OS into swapping death. Arachni was reporting time outs in handling some Javascript events. I managed to get a CTRL+C to Arachni, which reported that it was aborting the scan, but then pretty much froze. Netstat was showing 115 connections to the remote website, all in a CLOSE_WAIT state, and they remained in that state for over two hours.

I suspect that the remote site wasn't handling the load, and failed to properly close several TCP connections. Arachni appears not to handle this, causing sockets to get stuck in a CLOSE_WAIT state forever.

To attempt to recover, I attached gdb and forcibly closed one of the stuck sockets' handle. This triggered a whole bunch of stack traces pointing to Arachni reactor's select_connections function (error: bad file descriptor). It didn't help though, and I had to kill Arachni at that point.

I think there's an issue somewhere in Arachni (or ruby?) where it doesn't handle sockets that get stuck in a CLOSE_WAIT state properly. Can you please investigate?

(Sorry, but I can't provide a reproduction environment; this was a client website.)