Rate Limiter per queue wrong duration
Created by: EmperiorEric
Description
It is quite possible I have misconfigured something, but at the moment it would appear that the queues are not respecting their rate limits and are instead performing significantly slower than the durations we've set. I have several queues with different rate limits. For one, they appear to work at the same rate of speed. And worse that rate of speed seems to be significantly higher than the duration we've set on the limiter.
Minimal, Working Test code to reproduce the issue.
Here are a couple queues we setup. In total we have 6 queues, each with a different rate limit specific to the service.
let queueOptions = {
createClient: function (type) {
switch (type) {
case 'client':
return client
case 'subscriber':
return subscriber
default:
return defaultRedis
}
}
}
const lowPriorityQueue = new Bull('lowPriority', {
createClient: queueOptions.createClient,
limiter: {
// process max 1 request
max: 1,
// per 1.5 seconds for a total of 2400 jobs per hour.
// DNSimple's rate limit is 2400, so its a good base.
duration: 1500
},
})
const dnsimpleQueue = new Bull('dnsimpleQueue', {
createClient: queueOptions.createClient,
limiter: {
// process max 1 request
max: 1,
duration: 3600
},
})
In this case the job is a simple database row delete. It takes little to no time. But looking at the completed events from the queue, there is nearly 75 seconds in between each job.
Nov 07 15:40:33 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #507508 with priority 9007199254740991
Nov 07 15:40:33 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #507508 scrub subdomain successfully completed with result: undefined
Nov 07 15:41:49 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #379951 with priority 9007199254740991
Nov 07 15:41:49 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #379951 scrub subdomain successfully completed with result: undefined
Nov 07 15:43:05 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #379963 with priority 9007199254740991
Nov 07 15:43:05 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #379963 scrub subdomain successfully completed with result: undefined
Nov 07 15:44:21 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #514633 with priority 9007199254740991
Nov 07 15:44:21 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #514633 scrub subdomain successfully completed with result: undefined
Nov 07 15:45:37 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #178134 with priority 9007199254740991
Nov 07 15:45:37 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #178134 scrub subdomain successfully completed with result: undefined
Bull version
version: 3.4.8