Laravel queues at scale: sizing workers and surviving retries

Why "just add more workers" fails Laravel queues at scale#

Reach for a Laravel queue and the first instinct is to add workers when the backlog grows. However, a worker count is a derived number, not a dial you turn until the graph looks better. Add too few and the queue never drains. Add too many and they fight over the same rows and the same connection pool.

Laravel queues at scale behave like any other distributed system. Jobs arrive at some rate. Each takes some time to service. Furthermore the delivery guarantee is at-least-once, so a job can and will run more than once. Two questions decide whether the system holds. First, how many workers do you actually need? Second, what happens when a job runs twice? This page answers both, and everything else hangs off those two pillars.

Pillar 1: how many workers do you actually need?#

The first wedge is a real capacity model. Instead of copying a supervisor config off a blog post, you derive the pool size from three numbers you can measure. Therefore the answer changes with your traffic, not with someone else's.

Little's Law applied to Laravel#

Queueing theory gives the shape directly. In a stable system the work in progress equals the arrival rate times the service time. For a worker pool that means a simple rule. The number of busy workers you need is the arrival rate times the mean job runtime.

Because you never want a pool running flat out, you divide by a target utilization below 1.0. So the formula is required_workers = arrival_rate x mean_job_seconds / target_utilization. Arrival rate is dispatches per second at your peak. Mean job seconds is the average wall-clock runtime. Target utilization is your headroom for spikes, and 0.7 is a sane starting point.

The one term teams guess is mean_job_seconds. Instead of guessing, measure it. A queue middleware can time every job and push the sample to Horizon or your own histogram.

// A queue middleware that records each job's wall-clock runtime.
// Feed the samples to Horizon metrics or your own histogram, then read
// mean_job_seconds off real traffic instead of guessing it.
public function handle(object $job, Closure $next): void
{
    $startedAt = hrtime(true);

    $next($job);

    $seconds = (hrtime(true) - $startedAt) / 1_000_000_000;
    Metrics::observe('job_runtime_seconds', $seconds, [
        'job'   => $job::class,
        'queue' => $job->queue ?? 'default',
    ]);
}

Now you have a real number for service time. Consequently the worker count stops being a hunch and becomes arithmetic.

The worked example: size a real pool with real numbers#

This is the promise made operable. Drag the arrival rate, the mean job seconds, and the target utilization, and watch required_workers recompute. Then set retry_after against the job timeout and see the safety verdict flip. The default values are the worked example from the summary above.

Notice what the number is not. It is not a permanent setting. Because arrival rate moves through the day, the pool ceiling is a peak figure, and auto-balancing fills in below it. The calculator gives you that ceiling, and the next sections turn it into config.

The retry_after must exceed timeout footgun#

Here is the single most common way Laravel queues at scale start running jobs twice. The queue driver hides a reserved job for retry_after seconds. Meanwhile the worker runs that job under its own timeout. If retry_after is smaller than the timeout, the driver releases the job back to the queue while a worker is still running it. The Laravel queue documentation spells this out, yet the default config makes the mistake easy.

Set the two values deliberately in config. Below is the broken pairing next to the correct one, so you can pattern-match your own queue.php against it.

// config/queue.php, WRONG. retry_after (60) is not greater than the
// worker timeout (90, set with --timeout=90). A job that runs 75 seconds
// is released back to the queue while it is still running, and a second
// worker grabs it. Every slow job runs twice.
'redis' => [
    'driver'      => 'redis',
    'connection'  => 'default',
    'queue'       => 'orders',
    'retry_after' => 60,   // BUG: shorter than the 90s job timeout
    'block_for'   => 5,
],

// config/queue.php, RIGHT. retry_after (120) comfortably exceeds the
// 90s worker timeout. A slow job is killed by the timeout long before the
// driver would ever re-release it, so nothing double-runs.
'redis' => [
    'driver'      => 'redis',
    'connection'  => 'default',
    'queue'       => 'orders',
    'retry_after' => 120,  // > the 90s job timeout, with headroom
    'block_for'   => 5,
],

The diagram below traces both branches. Follow the "No" branch and you land on the second execution that nobody intended.

From the derived number to Horizon supervisor tuning#

The capacity model gave you a ceiling. Now map it onto a supervisor in Laravel Horizon. maxProcesses is that ceiling. minProcesses is a warm floor so a sudden spike is not cold-started. Meanwhile balance set to auto spreads processes across queues by backlog.

// config/horizon.php. The supervisor turns the derived worker count
// into real processes. maxProcesses is the pool ceiling the model gave you.
'supervisor-orders' => [
    'connection'    => 'redis',
    'queue'         => ['orders'],
    'balance'       => 'auto',  // spread processes by backlog across queues
    'minProcesses'  => 8,       // a warm floor so a spike is not cold-started
    'maxProcesses'  => 69,      // the number the capacity model derived
    'balanceMaxShift' => 4,     // how fast auto-balance may add processes
    'balanceCooldown' => 3,
    'tries'         => 3,
    'timeout'       => 60,      // MUST stay below the queue retry_after
],

The annotated view pins each field to what the model dictates. Read the legend below the figure for what each number is doing.

Pillar 2: will it run twice? The at-least-once reliability playbook#

Every ranking page teaches the dispatch and stops. Meanwhile the guarantee underneath goes unmentioned. Laravel queues at scale are at-least-once, so retries are normal and duplicate delivery is a feature of the model, not an incident. Therefore the job body has to be safe to run more than once.

At-least-once is a promise, not a bug#

A worker reserves a job, runs it, then deletes it on success. However, a crash between "ran it" and "deleted it" leaves the job in the queue. Consequently it runs again. This is the queue keeping its promise to never silently drop your work. In exchange, you owe it idempotency.

Idempotency keys: make replay a no-op#

The cheapest defense is an idempotency key. Derive a stable key from the work, then record it the first time the job runs. On a replay, the record already exists, so the job returns early.

public function handle(): void
{
    // At-least-once delivery means this method WILL run twice for some jobs.
    // An idempotency key makes the second run a safe no-op.
    $key = "charge:{$this->order->id}";

    $firstTime = Cache::add($key, true, now()->addDay());
    if (! $firstTime) {
        return; // already charged this order; the retry is a no-op
    }

    $this->gateway->charge($this->order->total, $this->order->id);
}

Because Cache::add is atomic, two concurrent runs cannot both win the key. So exactly one run does the charge, and the other returns. For a payment or an email, that one line is the difference between a healthy system and an angry inbox.

WithoutOverlapping and unique jobs#

Some jobs must not run alongside a copy of themselves. Inventory sync is the classic case. Two workers adjusting one product's stock at once will corrupt the count. Laravel ships two tools for this, and they solve different halves.

use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Queue\Middleware\WithoutOverlapping;

class SyncInventory implements ShouldQueue, ShouldBeUnique
{
    // ShouldBeUnique keeps a second copy of this job off the queue while
    // one is already pending, keyed by the product id.
    public function uniqueId(): string
    {
        return (string) $this->product->id;
    }

    public function middleware(): array
    {
        // WithoutOverlapping serialises runs for the same key, so two
        // workers never mutate one product's stock at the same moment.
        return [(new WithoutOverlapping($this->product->id))->expireAfter(180)];
    }
}

ShouldBeUnique keeps a duplicate off the queue while one is pending. Meanwhile WithoutOverlapping serialises the runs that do happen, keyed by id. Use both together for anything that mutates one shared row.

The transactional outbox: never double-charge#

Idempotency protects the job body. However, one gap remains. If you save an order and then dispatch a job, a crash between the two leaves an order with no job, or a job with no order. The transactional outbox closes that gap. You write the business row and an outbox row in one database transaction. Either both commit or neither does.

// The write and the intent to enqueue commit together, or not at all.
// No lost job if the queue is down, no ghost job if the write rolls back.
DB::transaction(function () use ($order) {
    $order->save();

    Outbox::create([
        'id'      => (string) Str::uuid(),   // this row id IS the idempotency key
        'type'    => 'order.placed',
        'payload' => ['order_id' => $order->id],
    ]);
});

// A separate relay polls unsent rows, dispatches each one, and marks it sent.
// If the relay crashes mid-flight the row stays unsent and is retried, so the
// job is delivered at least once and applied exactly once downstream.
Outbox::query()->whereNull('sent_at')->each(function (Outbox $row) {
    ProcessOutbox::dispatch($row->id);
    $row->update(['sent_at' => now()]);
});

A separate relay reads unsent outbox rows and dispatches them. Because the outbox row id doubles as the idempotency key, a redelivery is caught downstream. The sequence below shows the whole path, and where the "exactly once" guarantee actually lives.

Choosing a queue driver: the throughput ceiling#

Both pillars assume a driver that can keep up. Meanwhile each driver has a throughput ceiling, and a busy app will find it. So pick the driver for the load you actually have, not the one in the quick-start. The matrix below sets the ceilings side by side.

For most teams the answer is Redis with Horizon. However, if load is spiky and you would rather not run the queue yourself, SQS trades metrics for a managed ceiling. The database driver is fine to start, yet it is the first thing that buckles as you grow.

Backpressure, pool isolation, and the SLIs that page you#

Two pillars carry the design. Meanwhile three habits keep it upright once it is live. First, isolate pools by workload. A slow report job and a fast payment job on one queue means the slow one starves the fast one. So give them separate queues and separate supervisors.

Second, respect backpressure. When arrival rate exceeds pool capacity, latency grows without bound. Therefore shed load or scale before the wait time crosses your target, rather than after. The capacity model tells you exactly where that line sits.

Third, watch the right signals. Raw queue depth is noisy. Instead, track wait time, the oldest pending job age, and the failed-job rate. Because those three map to user pain, they are the ones worth a page at 3am. Everything else is a dashboard you glance at, not an alert.

When not to reach for a queue#

A queue is not free, so it is not always the right tool. Understanding the failure surface is the point of this guide, and part of that is knowing when to stay synchronous.

Where to go next#

The reliability habits here are not unique to PHP. If you run a large PHP codebase, our guide to structuring WordPress at scale with Composer and PSR-4 covers the same discipline applied to dependencies and autoloading. Meanwhile the wider question of shipping systems that hold up in production sits inside enterprise software development practices, which is the organisational half of what this post argues technically.

Because a queue is one piece of a fast platform, keeping the rest quick is part of the same job, and that is our performance work. If you want a second set of hands on a queue that has to survive its busiest day, you can hire Laravel developers or hire PHP developers from our team. No pressure and no lock-in: everything above is standard, documented Laravel you own outright.