Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • B bull
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 175
    • Issues 175
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 9
    • Merge requests 9
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • OptimalBits
  • bull
  • Issues
  • #1116
Closed
Open
Issue created Nov 07, 2018 by Administrator@rootContributor

Rate Limiter per queue wrong duration

Created by: EmperiorEric

Description

It is quite possible I have misconfigured something, but at the moment it would appear that the queues are not respecting their rate limits and are instead performing significantly slower than the durations we've set. I have several queues with different rate limits. For one, they appear to work at the same rate of speed. And worse that rate of speed seems to be significantly higher than the duration we've set on the limiter.

Minimal, Working Test code to reproduce the issue.

Here are a couple queues we setup. In total we have 6 queues, each with a different rate limit specific to the service.

let queueOptions = {
    createClient: function (type) {
        switch (type) {
            case 'client':
                return client
            case 'subscriber':
                return subscriber
            default:
                return defaultRedis
        }
    }
}

const lowPriorityQueue = new Bull('lowPriority', {
    createClient: queueOptions.createClient,
    limiter: {
        // process max 1 request
        max: 1,
        // per 1.5 seconds for a total of 2400 jobs per hour.
        // DNSimple's rate limit is 2400, so its a good base.
        duration: 1500
    },
})

const dnsimpleQueue = new Bull('dnsimpleQueue', {
    createClient: queueOptions.createClient,
    limiter: {
        // process max 1 request
        max: 1,
        duration: 3600
    },
})

In this case the job is a simple database row delete. It takes little to no time. But looking at the completed events from the queue, there is nearly 75 seconds in between each job.

Nov 07 15:40:33 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #507508 with priority 9007199254740991 
Nov 07 15:40:33 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #507508 scrub subdomain successfully completed with result: undefined 
Nov 07 15:41:49 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #379951 with priority 9007199254740991 
Nov 07 15:41:49 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #379951 scrub subdomain successfully completed with result: undefined 
Nov 07 15:43:05 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #379963 with priority 9007199254740991 
Nov 07 15:43:05 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #379963 scrub subdomain successfully completed with result: undefined 
Nov 07 15:44:21 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #514633 with priority 9007199254740991 
Nov 07 15:44:21 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #514633 scrub subdomain successfully completed with result: undefined 
Nov 07 15:45:37 app/worker.1: [Jobs: lowPriorityQueue] processing 'scrub subdomain' job #178134 with priority 9007199254740991 
Nov 07 15:45:37 app/worker.1: [Jobs: lowPriorityQueue] Low Priority Job #178134 scrub subdomain successfully completed with result: undefined 

Bull version

version: 3.4.8

Assignee
Assign to
Time tracking