Engineering
Designing Large Scale Background Processing Systems
Designing Large Scale Background Processing Systems
When it comes to building processing systems, whether it’s trade processing or order processing, things become a lot more interesting when scale is involved.
You have to think about the system as a whole, account for increasing traffic, and design it in a way that is not only reliable but also scalable and capable of handling concurrent requests. It’s no longer just about processing a request successfully. You also need to consider what happens when something fails halfway through.
In this article, I want to cover a few common architectural patterns, the thought process behind them, and the trade-offs worth considering when designing such systems. I’ll also share some insights into what could go wrong and how we can prepare for those situations.
We’ll use a concert ticket-booking system as an example throughout the article.
1. First Things First: Know Your Requirements, Know Your Numbers
Before jumping into the architecture, it’s important to understand what we’re actually building.
Let’s start with some basic requirements for our concert-booking platform:
- Users should be able to view and select available seats.
- Users should be able to temporarily reserve seats while completing payment.
- Once payment is successful, their booking should be confirmed.
- Users should receive their tickets and invoices via email.
- The system should prevent two users from booking the same seat.
- Failed operations should be recoverable without losing important information.
Now let’s establish a few assumptions.
Our platform currently serves around 2 million registered users, and we expect traffic to increase significantly whenever tickets for a popular event go on sale.
We’ll start with PostgreSQL as our primary database and assume our application runs across multiple Kubernetes pods.
PostgreSQL should be a reasonable starting point, but there’s something worth pointing out here.
Having 2 million registered users tells us very little about the actual load on our system.
What matters more is how many of them are active simultaneously, how many requests they generate, and how much contention those requests create.
For example, if 200,000 users attempt to book tickets over five minutes, that’s roughly 667 booking attempts per second on average. Actual peak traffic could be significantly higher, and each user may generate multiple API requests.
Before finalizing our architecture, we’d ideally want to know:
- Expected peak requests per second.
- Number of concurrent bookings.
- Acceptable latency for seat reservation and payment confirmation.
- How many background jobs each booking generates.
- How much delay is acceptable for sending invoices.
- How the system should behave when demand exceeds capacity.
These numbers will influence whether our existing infrastructure is sufficient or whether we need additional components.
The important thing is to avoid introducing complexity without understanding why we need it.
2. Reliability: What Happens When a Pod Crashes?
Let’s start with a simple scenario.
A user selects seat A12 and proceeds to reserve it. The request reaches one of our application pods, which successfully saves the reservation in PostgreSQL.
Immediately afterward, that pod crashes.
What happens to the reservation?
If we’ve committed the reservation to PostgreSQL before returning success, the reservation survives the crash. Another pod can retrieve the same information and continue processing.
But suppose we’re storing pending reservations only in an in-memory collection or relying on a local asynchronous task.
When that pod goes down, we might lose the work entirely.
This is where durability comes into the picture.
A durable job is one whose pending state is persisted somewhere that survives application crashes. Even if the worker responsible for processing it goes down, another worker should be able to recover the job.
For our system, PostgreSQL can store the reservation and its processing state.
A simple reservation lifecycle could be:
AVAILABLE → HELD → CONFIRMED
A seat temporarily enters the HELD state while the user completes payment. If payment succeeds and the reservation remains valid, it transitions to CONFIRMED.
If the user abandons the payment, the hold can expire and the seat becomes available again.
The important part is that these transitions are persisted and handled safely.
We should never depend on a particular pod staying alive for a booking to complete.
3. Concurrency: What If Two Users Select the Same Seat?
Now consider another scenario.
Two users, A and B, attempt to reserve seat A12 at almost exactly the same time.
The requests land on two different application pods.
Both pods query PostgreSQL and find the seat available.
If our implementation simply reads the seat status and then updates it, both requests might proceed based on the same outdated information.
We could end up with two reservations for one seat.
This is a classic race condition.
How do we prevent it?
One approach is to use an atomic conditional update at the database level.
Instead of separately checking whether a seat is available and then updating it, we perform the update only if the seat is still available.
Conceptually, our SQL would look like:
UPDATE seats
SET status = 'HELD',
reservation_id = :reservationId,
expires_at = :expiry
WHERE seat_id = :seatId
AND status = 'AVAILABLE'
RETURNING seat_id;
We then check whether the update affected a row.
If one request successfully updates the seat, a competing request cannot also claim that same available seat.
For expired reservations, we’d extend the condition to permit reclaiming an expired hold, provided the seat hasn’t already been confirmed.
PostgreSQL also provides row-level locking through SELECT FOR UPDATE, which can be useful for more complex reservation operations.
One thing I would avoid is keeping a database transaction open while the user completes payment.
A payment could take several minutes. Holding database locks for that duration would create unnecessary contention.
Instead, we create a time-limited reservation, commit it, and let the payment process happen separately.
For bookings involving multiple seats, we should also make sure the entire group of seats is reserved atomically. We shouldn’t accidentally confirm only two seats when the customer requested four.
Concurrency control should happen at the shared database level, not through synchronization local to one application pod.
4. Event-Driven Processing: Not Everything Needs to Happen Immediately
Let’s say the customer successfully completes payment.
Our application now needs to:
- Confirm the booking.
- Generate the ticket.
- Generate an invoice.
- Send an email containing the confirmation and invoice.
Do we really need to perform all these operations before responding to the user?
Probably not.
Confirming the booking is critical. Generating a PDF and sending an email can usually happen in the background.
This is where event-driven processing becomes useful.
Once the payment has been verified and the booking confirmed, the system can generate an event such as:
BOOKING_CONFIRMED
That event can be consumed by background workers responsible for generating tickets, creating invoices, or sending emails.
This provides a few advantages.
First, our booking API doesn’t have to wait for the email provider or PDF-generation service.
Second, failures in non-critical background processing don’t directly affect the booking API.
And finally, different parts of the system can scale independently.
For example, during a popular concert sale, we may need additional booking-service instances and invoice-processing workers. Those two components don’t necessarily need to scale at the same rate.
We can use a durable message broker such as RabbitMQ, Amazon SQS, or Azure Service Bus for distributing these jobs.
Kafka could also be useful if we need replayable event streams or multiple independent consumers.
However, a dedicated message broker isn’t always necessary. For smaller workloads, a database-backed job queue may be sufficient.
The decision should depend on requirements such as throughput, ordering, durability, and operational complexity.
5. Transactional Outbox: What If We Commit but Fail to Publish?
There’s an interesting failure scenario in the approach we just discussed.
Suppose the payment has succeeded, and our application performs two operations:
- Update the booking status to
CONFIRMEDin PostgreSQL. - Publish a
BOOKING_CONFIRMEDevent to the message broker.
Now imagine the database transaction commits successfully, but the application crashes before publishing the event.
From the database’s perspective, the booking is confirmed.
But our background workers never receive the event, so the ticket and invoice might never be generated.
This is known as the dual-write problem.
The issue is that we’re trying to update two independent systems: PostgreSQL and the message broker.
A database transaction cannot automatically guarantee that both operations succeed together.
Using the transactional outbox pattern
One way to solve this is by introducing an outbox_events table.
Instead of publishing directly to the broker, we perform two database operations in the same transaction:
- Update the booking to
CONFIRMED. - Insert a
BOOKING_CONFIRMEDevent into the outbox table.
Because both operations belong to the same transaction, either both commit or both roll back.
A separate component, often called an outbox relay, reads unpublished events from the database and sends them to the message broker.
If the application crashes after committing the booking, the event still exists in the outbox table.
The relay can publish it when processing resumes.
But there’s a catch.
What if the relay successfully publishes the event and crashes before marking it as published?
It may publish the same event again after restarting.
So although the outbox protects against losing committed events, it doesn’t guarantee that the event will be published only once.
This leads us to another important concept.
6. Idempotency: Handling Duplicate Messages
Distributed systems often have to deal with duplicate messages.
Suppose our invoice worker receives an event for booking 101.
The worker generates the invoice and stores it successfully. Before acknowledging the message, however, the worker crashes.
The broker doesn’t know that processing completed, so it may deliver the same event again.
Now another worker starts processing booking 101.
Without any protection, we might generate duplicate invoices or send duplicate emails.
This is why idempotency is important.
An idempotent operation should produce the same intended business outcome even if it’s executed multiple times.
For our booking platform, we can assign unique identifiers to events and maintain records of processed events.
For example, before generating an invoice, the worker can check whether an invoice already exists for that booking.
But this check must be concurrency-safe.
If two workers both check for an invoice and find nothing, both could try creating one.
A unique database constraint on the booking’s invoice identifier, combined with appropriate transactional handling, can help prevent this.
For external operations such as sending emails, things become more complicated. A provider might successfully send an email just before our worker crashes.
A local database flag alone cannot completely eliminate the possibility of duplicate external delivery.
Where supported, we can use provider-level idempotency keys. Otherwise, we need to define how duplicate attempts are handled.
The main point is simple: if we’re designing a system that retries failed operations, we must also think about what happens when the same operation executes twice.
7. Retries and Dead-Letter Queues
Let’s assume our email provider becomes unavailable for 30 minutes.
We certainly don’t want to lose all the confirmation emails generated during that period.
But continuously retrying the requests without any delay isn’t a good solution either.
We’d simply keep sending requests to an already failing service.
A better approach is exponential backoff.
For example, retries might happen after approximately:
- 1 second
- 2 seconds
- 4 seconds
- 8 seconds
We can also introduce jitter, which adds a little randomness to the delay.
This helps prevent thousands of workers from retrying at exactly the same moment.
However, not every failure is temporary.
A timeout might resolve itself.
An invalid payload, on the other hand, may continue failing regardless of how many times we retry.
Dead-letter queues
After a configured number of failed attempts, we can move the message to a dead-letter queue (DLQ).
For example, if the invoice worker repeatedly fails to process booking 101 because of malformed data, the event can be moved to the DLQ.
An engineer can investigate the issue, fix the problem, and replay the message if appropriate.
But simply introducing a DLQ isn’t enough.
We need to know when messages start accumulating there.
Otherwise, we could have hundreds of failed invoice-generation tasks sitting in the DLQ while the rest of the system appears healthy.
A DLQ should always be accompanied by monitoring, alerting, and a recovery process.
8. Handling Concurrent Workers Across Multiple Pods
Suppose our invoice-processing service has ten replicas.
Each pod is looking for pending jobs.
If we’re using a database-backed job queue, a naive implementation might:
- Query the database for jobs with status
PENDING. - Pick the first available job.
- Update the job status to
IN_PROGRESS. - Begin processing.
The problem is that two workers might select the same pending job before either updates its status.
We can avoid this using atomic job claiming.
PostgreSQL provides SELECT FOR UPDATE SKIP LOCKED, which is useful for this situation.
When one worker locks a pending job, another worker can skip that locked row and claim a different job instead.
The important thing is to keep the claim operation inside a short database transaction.
We don’t want to hold row locks throughout the entire invoice-generation process.
What if a worker crashes after claiming a job?
We can maintain a lease for each claimed job.
For example, when worker A claims a job, it receives ownership for five minutes.
If the job takes longer, the worker can periodically send a heartbeat to extend that lease.
If the worker crashes, the heartbeats stop. Once the lease expires, another worker can recover the job.
We should also use claim tokens or similar ownership checks to prevent a worker whose lease has expired from modifying the job after another worker takes over.
And because a job might be executed again during recovery, idempotency remains necessary.
9. Scalability and Backpressure
Now imagine thousands of customers successfully booking tickets within a few minutes.
Every confirmed booking generates an event for invoice processing.
Suppose our system receives 1,000 events per second, but our workers can process only 400 events per second.
That leaves 600 additional messages waiting every second.
A message queue helps us handle temporary spikes by buffering pending work.
But queues have finite storage, and workers have finite processing capacity.
If we continue accepting work faster than we can process it, the backlog will keep growing.
This is where backpressure comes in.
Backpressure is a way of controlling incoming work when the system is approaching its processing limits.
There are several approaches:
- Limit the number of jobs a worker can process concurrently.
- Use bounded queues or admission limits.
- Apply rate limiting.
- Scale worker instances based on backlog and processing capacity.
- Temporarily reject or delay new work when the system cannot handle it safely.
For example, if our email provider supports only 500 requests per second, scaling the worker deployment to 1,000 instances won’t necessarily help.
We’d just end up overwhelming the email provider.
A better design would control concurrency and scale workers only within the limits of downstream dependencies.
There is also an important distinction between our booking API and background workers.
We cannot simply queue seat-reservation requests indefinitely. By the time an old request is processed, the seat may no longer be available.
For extremely popular events, we might need rate limiting or a virtual waiting room to control how many users reach the booking system simultaneously.
For background jobs, on the other hand, a durable queue can absorb short spikes without making customers wait for non-critical processing.
So scalability isn’t just about adding more pods.
It’s about understanding the bottleneck and controlling how work flows through the system.
10. Scheduled Jobs and Reservation Expiry
Let’s return to our temporary seat reservations.
A user reserves seat A12 for five minutes but never completes payment.
Eventually, we need to release that seat.
One straightforward solution is a scheduled job that checks for expired reservations.
In Spring Boot, we could implement this using @Scheduled.
But there’s something to consider.
If our booking application runs on five pods, all five could execute the scheduled task at the same time.
They might query the same expired reservations and attempt to process them simultaneously.
One solution is to coordinate the scheduler using a distributed lock.
Another is to run a dedicated scheduler, such as a Kubernetes CronJob, instead of scheduling the task inside every API replica.
For larger workloads, we could have one scheduler discover pending work and distribute it to workers through a queue.
If multiple workers process jobs directly from PostgreSQL, atomic claiming with SKIP LOCKED can help them pick different jobs.
There’s another important detail
We shouldn’t rely entirely on the scheduled cleanup job to determine whether a reservation has expired.
Suppose the cleanup scheduler is down for ten minutes.
A reservation that expired at 10:05 should still be considered expired at 10:06, even if its database status hasn’t been cleaned up yet.
The reservation’s expiry timestamp should be checked whenever we attempt to claim or confirm a seat.
The scheduler is then responsible for cleaning up expired records, rather than being the only mechanism enforcing expiry.
What if payment succeeds after the reservation expires?
This is a case we need to handle explicitly.
Suppose:
- User A reserves seat A12.
- Their payment takes longer than expected.
- The reservation expires.
- User B successfully reserves the same seat.
- User A’s payment is reported as successful.
We obviously cannot confirm the same seat for both users.
The booking service must verify that the reservation is still valid before finalizing it.
If payment has succeeded but the reservation can no longer be fulfilled, we need a compensation process, such as initiating a refund and notifying the customer.
This is a good example of why distributed workflows need careful state management.
A successful database transaction cannot automatically undo an external payment.
11. Observability: Knowing What the System Is Actually Doing
Observability is something I consider critical when designing these systems.
It’s not enough for the application to be running. We need to know whether it’s actually processing work correctly.
For our concert-booking system, I’d want a dashboard showing:
- Incoming requests per second.
- Number of active booking requests.
- Successful and failed bookings.
- Seat-reservation latency.
- Number of pending background jobs.
- Worker processing throughput.
- Age of the oldest pending message.
- Number of retries.
- Messages waiting in the DLQ.
- Payment-success events without a completed booking.
- Database connection usage and lock contention.
These metrics tell us much more than whether our Kubernetes pods are healthy.
For example, all our pods might be running, but invoice-generation jobs could be piling up because the workers cannot reach the email provider.
Without proper monitoring, we might only discover the problem when customers start complaining.
Alerting is equally important
We should configure alerts for situations such as:
- Messages remaining in the DLQ beyond an acceptable duration.
- Queue backlog growing continuously.
- Worker throughput dropping unexpectedly.
- Unusually high retry rates.
- Payment confirmations remaining unresolved.
- Increased booking failures or database lock contention.
Distributed tracing can also help us follow a booking across services.
If someone reports that their payment succeeded but they never received a ticket, we should be able to trace that booking from the payment event through confirmation, outbox publication, ticket generation, and email delivery.
Having correlation IDs across these operations makes debugging much easier.
12. Testing Beyond the Happy Path
Most systems work reasonably well when everything behaves as expected.
The interesting problems usually appear when something fails at an inconvenient moment.
For our booking platform, I’d specifically test scenarios such as:
Concurrent reservations: Two users attempt to book the same seat. Only one should succeed.
Application crash: A pod crashes after committing a booking but before publishing an event. The outbox should allow the event to be recovered.
Duplicate message delivery: The same booking event is processed twice. We shouldn’t create duplicate tickets or invoices.
Worker failure: A worker crashes after claiming a job. Another worker should be able to recover the unfinished task.
Database rollback: If the booking transaction fails, neither the booking update nor its outbox event should be committed.
Delayed payment: A payment succeeds after the seat reservation expires. We should not double-book the seat, and the compensation process should work correctly.
High traffic: A popular concert causes a sudden increase in requests. The system should either handle the load within its limits or reject excess requests safely.
These scenarios should be covered through a combination of unit tests, integration tests, concurrency tests, and load tests.
I would also add regression tests whenever we discover and fix a production bug, so that the same issue doesn’t reappear during future changes.
13. Understanding the Trade-offs
After going through these patterns, it’s tempting to think that a reliable system must use PostgreSQL, Kafka, Redis, distributed locks, autoscaling workers, and several independent services.
But every additional component comes with its own cost and complexity.
A database-backed job queue might work perfectly well for a moderate workload.
A dedicated message broker becomes more useful when we need to buffer large volumes of work, isolate consumers, or scale different processing pipelines independently.
Similarly, strict ordering guarantees may simplify some business workflows but can limit parallel processing.
A transactional outbox improves reliability, but we now need to operate and monitor an outbox relay.
And autoscaling isn’t always the solution if our real bottleneck is a database or third-party API.
For our concert-booking platform, I would start with PostgreSQL as the source of truth for reservations and bookings, enforce concurrency control at the database level, and use durable background processing for ticket generation and email delivery.
As traffic grows, I’d measure where the actual bottlenecks are before introducing additional infrastructure.
There is no single architecture that is ideal for every workload.
The right solution depends on what we’re trying to achieve and which trade-offs we’re comfortable making.
Final Thoughts
When designing a processing system, one habit I find useful is to walk through the flow and imagine the application crashing at different points.
What if the database commits but the message isn’t published?
What if two workers pick the same job?
What if a worker completes an operation but crashes before acknowledging it?
What if the message is delivered twice?
What if a downstream service stays unavailable for several hours?
What if traffic suddenly increases tenfold?
Thinking through these scenarios helps uncover weaknesses that aren’t obvious when we only look at the normal execution flow.
The patterns we discussed—concurrency control, transactional outbox, idempotency, retries, dead-letter queues, backpressure, and observability—help address different parts of these problems.
But they aren’t necessarily required in every system.
Ultimately, good system design is about understanding the requirements, anticipating failures, and choosing the simplest architecture that can meet those requirements reliably.
And as the system grows, being prepared to revisit those decisions.