Partner with CodeWalnut to drive your digital growth!

Tell us about yourself and we will show you how technology can help drive business growth.

Thank you for your interest in CodeWalnut.Our digital expert will reach you within 24-48 hours.
Oops! Something went wrong while submitting the form.
Insights

Common Backend Production Issues That Can Wake You at Midnight -Part 1

September 16, 2026

6 min. read

Double charges, stuck searches, exposed bookings, skipped checks. Four backend problems that pass review and reach production, and the practices that prevent them.

The release goes live: code reviewed, checks passed, everything you tested worked as expected.

Then a customer reports two charges for one purchase. A slow payment service leaves other screens waiting. A logged-in customer changes a booking reference and accesses someone else’s details. And a new feature turns out to skip a check the old one always ran.

You don’t need to work in airlines to recognise these surprises. They show up across domains such as retail, e-commerce, and banking, financial services and insurance (BFSI)—from a shopping checkout to a money transfer or a regular online payment. These are real-life issues for end users.

Why these can surface after release

A feature can work in a quiet demo yet fail when replies slow down or requests overlap.

Imagine a busy sale with hundreds of customers paying at once. Payment replies slow down, but new requests keep arriving. Waiting payments pile up and can tie up the workers that also handle searches—so even customers just browsing can get stuck.

Problems 3 and 4 can happen even with just one user: a customer can access someone else’s booking, or the booking feature can skip a payment check. Neither requires heavy traffic.

Are you in the same boat?

At CodeWalnut, our work on an airline system helped shape the backend practices we follow. Here are four problems and how we address them, using simplified examples—not specific client incidents. The practices apply across languages and frameworks.

Let’s dive into the problems.

1. Booked once. Charged twice?

Ravi pays for his booking. The payment succeeds, but the confirmation never reaches him. The screen offers “Try again”, so he taps it. The backend treats that as a new purchase and charges the same booking a second time.

A missing reply does not mean a missing payment.

The practice we follow

Give the purchase a reference that stays the same across retries. Save the payment attempt before calling the provider, and use that reference with the provider’s duplicate protection. Repeat requests must find the same attempt—even when they arrive together.

If the result is unknown, check that payment instead of starting another charge. This is idempotency: trying the same purchase again must not mean paying again.

Removing the button does not solve it. The same second request can come from a client library retrying a timeout, or from a proxy re-sending to another instance — retries you did not write and cannot see. The backend has to be the thing that recognises the repeat.

2. Slow payments. Search stuck?

The payment provider is slow. Flight search does not call it, yet searches start waiting too. Why? Both use the same pool of workers. Workers waiting for payment replies cannot pick up a search.

Shared capacity

No worker free for search.

Payment requestsReplies are slow
Flight searchWaiting for a free worker
Worker 1PaymentWaiting…
Worker 2PaymentWaiting…
Worker 3PaymentWaiting…
Worker 4PaymentWaiting…

All four workers are waiting on payments.

Leave a worker free for search.

Payment requestsReplies are slow
Flight searchA worker can handle it
Worker 1PaymentWaiting…
Worker 2PaymentWaiting…
Worker 3PaymentWaiting…

Payment calls capped at three; one worker can handle search.

Illustrative numbers. Switch views to see the difference.

The practice we follow

Set a time limit for every remote call. Limit how many payment calls can run together, leaving room for other work. When that limit is reached, allow only a bounded queue or ask new requests to try later. Don’t let waiting work grow forever.

A timeout alone is not enough if new payment calls keep filling every available slot. More retries can make the queue worse.

The stakes are real. GAO identified 34 IT outages affecting 11 of 12 selected US airlines between 2015 and 2017. About 85% led to some flight delays or cancellations. These are industry-wide examples of disruption—not evidence that the four patterns here caused those outages. Read the GAO findings.

3. Your login. Someone else’s data?

Meera opens her booking, B100. Then she changes the request to B200, Ravi’s booking, and receives his details. The backend checked who she was, but not what she was allowed to read.

That does not take an attacker. The reference sits in the URL bar, in the confirmation email and in every support ticket. Someone curious tries the next one, and the backend answers.

Being signed in does not make every booking yours to read.

The practice we follow

Use the identity established at sign-in, not a passenger ID supplied in the request. Check permission for the specific booking: it belongs to that person, or an explicit rule grants access, such as an authorised support role.

Without permission, return no booking data. A hidden button or a hard-to-guess reference does not replace that check.

4. Booking added. Payment check skipped?

Seat selection already checks payment before saving a reservation. Then you add booking, which saves reservations directly through the database code. The new feature never runs the existing payment check.

How does that reach production? The failed-payment case was written once, against seat selection. When booking arrived it was tested the way new features usually are — a successful payment, a saved reservation, the expected result. Nobody ran an unsuccessful payment against the new path. The test exists. It was never pointed at the second entry point.

Reservation paths

One check. Two write paths.

Seat selection
Check payment
Booking
No payment check
Seat recordsBoth paths save a reservation here

Booking bypasses the payment check.

Both paths run the same check.

Seat selection
Booking
↓ both use the same entry point ↓
Reserve a seatRun the payment check here
Seat recordsNo direct write from booking

Both callers use the reservation entry point; no direct write from booking.

Code paths, not necessarily separate services.

The practice we follow

Keep each feature’s files together so developers can find them. More importantly, give seat selection one reservation entry point that runs its checks. Make booking use it instead of writing seat records directly.

Move existing callers to that entry point, then enforce the boundary so new code cannot bypass it. Folder names help you find the feature. They do not enforce its rules.

Pick one flow. Ask four questions.

Can it run twice safely? Can it get stuck waiting? Can the wrong person access it? Can another path skip its rules?

Start there—not with a rewrite.

Next: the confirmation screen shows a flight and seat. What else did the backend send to the browser? We’ll look at internal data that accidentally becomes public, and how to keep it out of a response.

Author
Author
Atul Kumar Prajapati
Atul Kumar Prajapati
Software Engineer