Implement distributed crawling

Created by: Zapotek

The re-written crawler can easily be fed new paths at any time, it's only a matter of adding the RPC server hooks and implementing the path distribution/routing algorithm.