[Feature] Job dependencies

Created by: ccollie

Let's say you have one job that depends on another, but the task definitions are fundamentally different. Consider the following example:

const debitAccount = await orderQueue.add('debitAccount', 
      {id : accountId, orderId: orderId, amount: costOfWidget});

const preparePackingSlip = await orderQueue.add('prepareShipment', 
      {id : accountId, orderId: orderId}, {depends : [debitAccount.id]);

// debitAccount.id not necessary here, but used to demonstrate that 
// we can have multiple dependencies
notificationQueue.add('notifyWidgetShipped', {orderId : orderId}, 
     {depends : [debitAccount.id, preparePackingSlip.id]});

Notifications are sent only after the packing slip job is completed.

We're considering using bull instead of our in-house solution, and this is one of the major missing features. I'd be willing to take the lead on this.

Implementation

Internally this would require a 'dependent' list in addition to 'delayed'. As a jobs are completed, their dependent lists are scanned and the id of the completed job is removed. Once the count of a child job's dependencies reaches 0, it's moved from the 'dependent' list to 'active'. If any of the parent jobs fail, the dependents are discarded recursively.