---
title: "Fair Background Processing for Uneven Batches with BullMQ"
description: "Why one big customer can hold up everyone else in a FIFO queue, and how to give each customer a fair turn with BullMQ."
publishedAt: "2026-11-13"
tags:
  - queues
  - bullmq
  - architecture
draft: false
art: dots
canonical_url: /blog/fair-background-processing-with-bullmq
markdown_url: /blog/fair-background-processing-with-bullmq.md
last_updated: 2026-11-13
---

# Fair Background Processing for Uneven Batches with BullMQ



Picture a product where accounting teams upload scanned receipts and invoices. Every file that lands needs work before anyone can use it. Something has to read the scan, pull out the date and the amount, and file it under the right month.

That work is slow, so it doesn’t happen while the person waits. Each file becomes a job in a queue, and a few workers in the background take jobs off the queue one at a time.

Most days, nobody notices any of this. Then the end of the month arrives.

At nine in the morning, Ana’s firm drops 2,000 receipts into the app. A minute later, Ben, a freelancer, uploads twelve. Ben’s twelve receipts should take a few seconds to process. He waits, refreshes the page, and waits some more. His files are sitting behind all 2,000 of Ana’s. These are made-up, round numbers to reason with.

Nothing is broken. The queue is doing exactly what it was built to do.

## A queue is a line

It’s the same as a coffee shop with three baristas, where Ana orders 24 coffees for her office and Ben, a minute later, orders one. The baristas make drinks in the order they were ordered, so Ben stands there while 24 coffees are made in front of him.

That’s first in, first out, or FIFO[^annotation-1]. It’s the order a BullMQ queue uses by default, and it’s fair in one specific sense. Nobody jumps the line. But it has no idea who is in the line, so a single large order fills it.

Here’s the same morning, shrunk down to something you can watch. Ana submits 24 jobs, and three smaller batches arrive just after from Ben, Cleo, and Dev. Three workers share the line, and every job takes the same time. Press play.

**One line for everyone (FIFO) (simulation)**

There are 3 workers. Every job takes 2 ticks.

1. **t = 0: Ana submits 24 jobs.** Every one of the 3 workers starts one of Ana’s jobs. The other 21 wait.
2. **t = 1: Ben submits 3 jobs.** They join the back of the one line, behind 21 of Ana’s. FIFO knows only arrival order, so a small batch waits behind everything that came first.
3. **t = 3: Cleo submits 5 jobs.** They join the back of the one line, behind 18 of Ana’s and 3 of Ben’s. FIFO knows only arrival order, so a small batch waits behind everything that came first.
4. **t = 6: Dev submits 2 jobs.** They join the back of the one line, behind 12 of Ana’s, 3 of Ben’s and 5 of Cleo’s. FIFO knows only arrival order, so a small batch waits behind everything that came first.
5. **t = 16: Ben’s first job starts.** 15 ticks after Ben submitted. Until now every worker went to Ana, whose batch never had to share.
6. **t = 24: All work done.** All 34 jobs finish at t = 24. Ana finished at 16. The small batches finished Ben at 18, Cleo at 22 and Dev at 24, each one after all of Ana’s.

*FIFO is the order a single BullMQ queue uses by default. Synthetic batches, and every job takes the same time.*

Ben submitted at t = 1, but his first job didn’t start until t = 16. Every worker spent those fifteen ticks on Ana’s jobs, because hers were first in line. Cleo and Dev waited behind Ana and Ben both, and Dev, who had only two jobs, was the last person to finish.

Ana, meanwhile, was done at t = 16. Her batch never had to share.

## The obvious fixes

The first instinct is to add workers. That makes the line move faster, but only while the workers are the bottleneck. Every worker takes CPU and memory, and every job it runs holds database connections and calls whatever service reads the scans. Add workers on the busiest morning of the month and the pressure moves downstream. The database runs out of connections, or the scanning service starts rejecting requests, and jobs fail and retry instead of finishing. More capacity helps when the machines underneath have room to spare. Without that headroom, it makes things worse.

And even when it works, it doesn’t change the order. Ben is still behind every one of Ana’s jobs, just for a shorter time. The next time Ana drops 20,000 receipts, he’s right back where he started.

The second instinct is to put small batches first. That helps Ben today, but now a big customer can be stuck forever behind a stream of small ones, and anyone can split one big upload into lots of small ones to skip ahead.

Either way, the line still doesn’t know whose jobs it holds.

## Take turns

Back to the coffee shop, with one rule changed. Each barista now makes the next drink for whoever has waited longest since their last one. Ana still gets all 24 coffees. But after one of hers, Ben’s gets made, and then one of Ana’s, and so on.

That’s round-robin by customer. Each customer’s jobs stay in their own order, and every free worker serves the customer who was served least recently. Here’s the same morning again, with that one rule changed.

**One turn each (round-robin) (simulation)**

There are 3 workers. Every job takes 2 ticks.

1. **t = 0: Ana submits 24 jobs.** Every one of the 3 workers starts one of Ana’s jobs. The other 21 wait.
2. **t = 1: Ben submits 3 jobs.** Ben’s jobs wait in their own lane. When a worker frees up at t = 2, Ben has been served least recently, so Ben goes first.
3. **t = 3: Cleo submits 5 jobs.** Cleo’s jobs wait in their own lane. When a worker frees up at t = 4, Cleo has been served least recently, so Cleo goes first.
4. **t = 6: Dev submits 2 jobs.** A worker is free right now, and Dev has never been served, so Dev’s first job starts at once.
5. **t = 12: Every small batch is done.** By t = 12, Ben, Cleo and Dev have finished. Ana gets every worker again, because round-robin only takes the turns somebody else is waiting for.
6. **t = 24: All work done.** All 34 jobs still finish at t = 24. The same work, in a different order. The small batches finished Ben at 6, Cleo at 12 and Dev at 10. Ana finished at 24, the price of sharing.

*Each free worker takes the next job of the user served least recently. BullMQ Pro’s Groups do this for you. With open-source BullMQ, your application builds it.*

This time Ben’s first job started at t = 2, the moment a worker came free. Cleo and Dev each got a turn almost as soon as they arrived. By t = 12 every small batch was done, and Ana had all three workers to herself again.

Now put the two runs side by side, and look at how long each batch took, from the moment it was submitted to its last job finishing.

**How long until each batch was done (simulation)**

| Person | Jobs | One line (FIFO) | Taking turns (round-robin) |
| --- | --- | --- | --- |
| Ana | 24 | 16 | 24 |
| Ben | 3 | 17 | 5 |
| Cleo | 5 | 19 | 9 |
| Dev | 2 | 18 | 4 |

*From submitting a batch to its last job finishing, in ticks. Same work and the same workers, and both finish everything at t = 24.*

Ben’s batch was done 5 ticks after he submitted it, instead of 17. Cleo’s and Dev’s took less than half as long as before. Neither policy did the work any faster, since both finished everything at t = 24. What changed is who waited. Ana’s batch took 24 ticks instead of 16, because she now shares the workers, though she still gets every turn nobody else is waiting for.

## Where the turns get decided

Here’s the catch. BullMQ’s ordinary queue doesn’t take turns by customer. It gives you jobs that survive a crashed worker, retries, priorities, and ways to limit how much runs at once. None of them knows that a job belongs to somebody.

Priorities come closest. But a priority is a rank you give a job when you add it, and a turn depends on who else is waiting at that moment. You could rank each customer’s first job above everyone’s second, and so on down. That’s the small-batches-first idea again, with the same hole in it. A queue per customer doesn’t get you there either, because each queue needs its own workers and nothing shares them out.

**BullMQ can do it, for a price.** BullMQ Pro has a feature called [Groups](https://docs.bullmq.io/bullmq-pro/groups/) that processes jobs round-robin across groups, such as one group per customer. It’s a commercial product, though. At the time of writing, its standard license is listed at $1,395 a year per deployment, or $139 a month, for organizations with fewer than 100 employees, and larger companies get a custom quote ([BullMQ Pro pricing](https://bullmq.io/#bullmq-pro)). If that fits your budget, it’s the simplest route. For many small teams, it’s a lot to pay for one scheduling rule.

You may also find older examples that pass a `groupKey` to BullMQ’s rate limiter. That option was removed in BullMQ 3.0, and it only ever slowed customers down, rather than giving them turns.

Without Pro, you can build the turn-taking yourself. It lives in a small piece of your own code that sits between your database and the queue.

**Where the fairness decision happens**

1. **Backlog** (Your database): Every user’s jobs, each in its own order. Durable, and allowed to be large.
2. **Dispatcher** (Your application): Admits one job at a time, from the user served least recently. This is the fairness decision.
3. **BullMQ queue** (BullMQ): Holds only a short window of admitted jobs, in FIFO order.
4. **Workers** (BullMQ): Run jobs under concurrency caps that protect capacity, not order.

*The queue spreads admitted work over time, and the dispatcher decides whose work is admitted next. BullMQ Pro’s Groups move that decision into the queue itself.*

## Building the turn-taker

The dispatcher decides which customer gets the next turn, and that’s all it does. Instead of putting all 2,000 of Ana’s jobs straight into BullMQ, you keep every customer’s waiting jobs in your database, in their own order. The dispatcher then hands BullMQ one job at a time, always from the customer who was served least recently.

**Keep the queue short.** This is the part that’s easy to get wrong. If the dispatcher pushes everything into BullMQ at once, you’re back to one long line, and Ben is behind Ana again. So it keeps only as many jobs in BullMQ as the workers can run, plus a few so that no worker sits idle waiting for the next one. Those few are a trade. Every spare job in there is standing in a plain line again, so the more you allow, the longer a new customer waits for a first turn. The long wait happens in your database, where the turns are decided.

**Believe the database.** Every job now lives in two places, your database and BullMQ, and the two can disagree. The dispatcher can crash after choosing a job and before queueing it. A worker can die without reporting back. Redis can lose a job your database still thinks is queued. You can’t make the two agree at every moment, so decide which one is right. The database is the record, and BullMQ is only where jobs go to run.

Most of the design follows from that. The dispatcher saves its choice in the database before it queues anything, inside a transaction that also locks out any other copy of the dispatcher. If it crashes right after, the saved choice is still there, and it queues the job when it comes back. A job takes up a place in the short queue from the moment it’s chosen until the database hears that it finished, whatever BullMQ believes in between.

Then, every so often, the dispatcher compares its list against BullMQ’s. A job BullMQ has finished gives its place back. A job BullMQ has never heard of goes back into its customer’s line. It’s tempting to skip this check, because nothing goes wrong for weeks. But every lost job keeps its place forever, and one day the dispatcher has none left to give out while every worker sits idle.

Queueing a job again means the same job can arrive twice. BullMQ ignores a job whose [ID it already holds](https://docs.bullmq.io/guide/jobs/job-ids), so give each attempt at a job its own ID. A repeat of the same attempt gets dropped, and a real retry gets through. The ID belongs to the attempt, not the job, because BullMQ keeps failed jobs around unless you tell it otherwise, and a retry that reused the ID would be thrown away as a copy. The guard only lasts while BullMQ remembers the first copy, though, so it catches most duplicates and not all of them (more on that below).

**Own the retries.** BullMQ can [retry a failed job](https://docs.bullmq.io/guide/retrying-failing-jobs) by itself if you configure it to. But those retries stay inside BullMQ, and a job waiting for its next try holds a place that no other customer can use. A customer with a batch of broken files would end up holding most of the short queue. So in this design, your own code handles failures. A failed job goes to the back of its own customer’s line, with a wait before it can be picked again, and after a few failures it’s set aside where someone can look at it. Don’t leave out the wait. Without it, an outage in the scanning service fails every job in seconds, and the whole backlog is set aside before anyone has noticed.

That’s real extra machinery, with state to keep, a check to run, and numbers to watch. It’s worth it when a slow morning for small customers costs you more than the code does.

## Don’t forget the database

Turn-taking decides the order. How much runs at once is a separate question, and the short queue already answers it, because nothing runs that the dispatcher hasn’t let in. So size it by what your database and the scanning service can take.

The usual mistake is to count per worker. [Worker concurrency](https://docs.bullmq.io/guide/workers/concurrency) is set per worker, so it multiplies. Three workers that each run 8 jobs at a time can run 24 jobs at once. If each of those jobs keeps two database connections busy, that’s 48 connections, before the rest of your product asks for any. Set [global concurrency](https://docs.bullmq.io/guide/queues/global-concurrency) to the same number as the short queue, so BullMQ holds the line even if the dispatcher miscounts, and keep each job’s database work short. If an outside service limits how fast you can call it, cap how many jobs start per second with [rate limiting](https://docs.bullmq.io/guide/rate-limiting). Jobs that mostly crunch numbers rather than wait on the network are better off in BullMQ’s [sandboxed processors](https://docs.bullmq.io/guide/workers/sandboxed-processors) than with more concurrency.

## Every job might run twice

A queue like this one can run the same job twice. The dispatcher queues a job again when it can’t tell whether the first try landed. A retry is a second run by design. And a worker can do the work and crash before reporting it, in which case BullMQ treats the job as [stalled](https://docs.bullmq.io/guide/workers/stalled-jobs) and gives it to another worker. Queues accept that on purpose, because the alternative is losing jobs.

So the job itself has to make a second run harmless. Here that’s easy. Processing a receipt means writing its date and amount to that receipt’s own record, so a second run overwrites the first instead of adding to it. BullMQ’s docs call this an [idempotent job](https://docs.bullmq.io/patterns/idempotent-jobs).

One cheap check goes a long way. Before it starts, the worker asks the database whether this job is still waiting to be done, and drops it if not. That catches the copy that turns up late, after the job was already finished or set aside.

The hard cases are actions you can’t take back, like emailing a customer or charging a card. Those need a way to recognise a repeat. Usually that’s a unique key the other service accepts, so it ignores the second request, or a record that says the action was already done, checked before acting.

## When jobs aren’t equal

Taking turns by job is easy to reason about when every job costs about the same. Real receipts don’t. If one customer’s files take ten times longer to process, an equal number of turns gives them ten times as much of the workers’ time as anyone else. When that matters, you can give turns by estimated cost instead of by count, at the price of having to estimate costs well.

Whatever rule you pick, measure what it’s for. Track how long each customer waits before their first job starts, and how long their batch takes to finish. Those numbers tell you whether the next Ben will be waiting for seconds or for an hour.

Back to that morning. With turns, Ben’s twelve receipts are done before he thinks to refresh. Ana’s 2,000 take a little longer than they used to. And the workers spent the whole morning just as busy as before. Only the order changed.

[^annotation-1]: First in, first out. Jobs are handled in the order they were added, whoever added them.
