Skip to content
GitLab
Closed
Issue created by Administrator@rootContributor

Share redis client across queues created by bull() calls?

Created by: redroot

Hi, I'm not sure if this issue has already been addressed so I'm tentatively opening a new issue, please advise me if I've missed an already open issue.

I noticed that bull tends to create a new redis client everytime you call bull(), rather than sharing a pre-existing client (memoizing it maybe). Background: We're using bull on Heroku, using Heroku's Redis addon, meaning we have a limited number of connections so the more worker we want to add, the more expensive the plan we have to pay for.

Is this a design decision or something that I can work on improving in a PR?

You can recreate it locally by running the following node code which is lifted from my codebase, and then running CLIENT LIST in redis-cli

var bull = require('bull');
var _ = require('underscore');

var createQueue = function(name) {
  return bull(name, 6379, '127.0.0.1');
};


var queueProcessingMap = [
  {
    queueName: 'url-download',
    queueProcessor: function() { },
    concurrency: 5
  },
  {
    queueName: 'file-creation',
    queueProcessor: function() { },
    concurrency: 3
  },
  {
    queueName: 'pdf-process',
    queueProcessor: function() { },
    concurrency: 5
  },
];

// use to shutdown all queues on SIGTERM later
var queueList = [];

_.each(queueProcessingMap, function(item) {
  _.times(item.concurrency, function(i) {
    console.log("Starting " + item.queueName + " worker " + (i + 1) + " of " + item.concurrency);
    var queue = createQueue(item.queueName);
    queue.process(item.queueProcessor);
    // queue.on() listeners ...
    queueList.push(queue);
  });
});

// call close of all queue workers then exit process
process.once('SIGTERM', function(sig) {
  logger.info('Closing all queues before killing worker ...');
  var closePromises = [];
  _.each(queueList, function(queue) {
    closePromises.push(queue.close());
  });
  Promise
    .all(closePromises)
    .then(function() {
      logger.info("All queues closed, killing worker");
      process.exit(0);
    }).catch(function(err) {
      logger.info("Error closing queues, killing worker");
      logger.error(err);
      process.exit(0);
    });
});

Output from the above code:

$ node test.js 
Starting url-download worker 1 of 5
Starting url-download worker 2 of 5
Starting url-download worker 3 of 5
Starting url-download worker 4 of 5
Starting url-download worker 5 of 5
Starting file-creation worker 1 of 3
Starting file-creation worker 2 of 3
Starting file-creation worker 3 of 3
Starting pdf-process worker 1 of 5
Starting pdf-process worker 2 of 5
Starting pdf-process worker 3 of 5
Starting pdf-process worker 4 of 5
Starting pdf-process worker 5 of 5

Output from CLIENT LIST in redis-cli:

127.0.0.1:6379> client list
id=447 addr=127.0.0.1:58140 fd=44 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=448 addr=127.0.0.1:58141 fd=45 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=409 addr=127.0.0.1:55512 fd=6 name= age=6907 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
id=410 addr=127.0.0.1:58103 fd=7 name= age=4 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=411 addr=127.0.0.1:58104 fd=8 name= age=4 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=412 addr=127.0.0.1:58105 fd=9 name= age=4 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=413 addr=127.0.0.1:58106 fd=10 name= age=4 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=404 addr=127.0.0.1:54771 fd=5 name= age=91304 idle=91255 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=keys
id=414 addr=127.0.0.1:58107 fd=11 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=415 addr=127.0.0.1:58108 fd=12 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=416 addr=127.0.0.1:58109 fd=13 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=417 addr=127.0.0.1:58110 fd=14 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=418 addr=127.0.0.1:58111 fd=15 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=419 addr=127.0.0.1:58112 fd=16 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=420 addr=127.0.0.1:58113 fd=17 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=421 addr=127.0.0.1:58114 fd=18 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=422 addr=127.0.0.1:58115 fd=19 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=423 addr=127.0.0.1:58116 fd=20 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=424 addr=127.0.0.1:58117 fd=21 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=425 addr=127.0.0.1:58118 fd=22 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=426 addr=127.0.0.1:58119 fd=23 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=427 addr=127.0.0.1:58120 fd=24 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=428 addr=127.0.0.1:58121 fd=25 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=429 addr=127.0.0.1:58122 fd=26 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=430 addr=127.0.0.1:58123 fd=27 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=431 addr=127.0.0.1:58124 fd=28 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=432 addr=127.0.0.1:58125 fd=29 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=433 addr=127.0.0.1:58126 fd=30 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=434 addr=127.0.0.1:58127 fd=31 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=435 addr=127.0.0.1:58128 fd=32 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=436 addr=127.0.0.1:58129 fd=33 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=437 addr=127.0.0.1:58130 fd=34 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=438 addr=127.0.0.1:58131 fd=35 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=439 addr=127.0.0.1:58132 fd=36 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=440 addr=127.0.0.1:58133 fd=37 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=441 addr=127.0.0.1:58134 fd=38 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=442 addr=127.0.0.1:58135 fd=39 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=443 addr=127.0.0.1:58136 fd=40 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=444 addr=127.0.0.1:58137 fd=41 name= age=3 idle=3 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=445 addr=127.0.0.1:58138 fd=42 name= age=3 idle=3 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=446 addr=127.0.0.1:58139 fd=43 name= age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange