Performant Priority Queues

Created by: zebulonj

I've been using Bull for more than a year in a large-scale, high-performance application (1000+ jobs per minute, 24/7)... and am very happy with it. On the horizon I have the need to add priority to our queues, but O(N) performance per insert is likely going to pose a problem (I expect to be dumping upwards of a million jobs at a time into queues on a regular basis). Opening this issue because I'm interested in learning whether there has been any discussion of more performant architectures for handling priority via Bull queues. Off the top of my head, I'd be interested in running three or four fixed priority queues (where each queue represented jobs of the same priority), with the set of queues feeding into workers that each pull from the highest priority queue with a pending job... thereby removing the need to sort jobs within a queue based on priority.

I'm happy to toss around ideas and ultimately contribute code. Just thought I'd raise this with the community before I invest in any particular path.