Thirty seconds was the symptom
A notification service taking half a minute to respond, failing a third of the time. Both numbers had the same cause.
Context
A multinational insurer. Enterprise notification services — the layer other systems depend on to tell customers that something has happened.
Notification infrastructure has an unusual failure profile: when it breaks, it rarely breaks loudly. The systems calling it get a slow response or an error and move on. The customer simply never hears anything, or hears it far too late to matter. The failure lands on someone who has no idea which system failed them.
The problem
Three numbers, and for a long time they were treated as three problems:
- Responses took around 30 seconds
- The error rate sat at 38%
- Production deadlocks were a recurring operational event
More than one in three notification attempts was failing. Anything that did succeed took long enough that the calling system had often given up waiting.
They were not three problems. They were one, observed from three angles. Transactions were being held open longer than they needed to be, against tables that weren’t indexed for the way they were actually queried. That produced lock contention. Contention produced deadlocks. Deadlocks produced timeouts and failed transactions — which is what the 38% was. And the waiting that preceded each failure is what the 30 seconds was.
The latency was the symptom everybody could see. The contention underneath it was the disease.
The constraints
- A live production system in a regulated industry. No appetite for experimentation in place, and changes carry an audit trail.
- A dependency, not a destination. Other systems called this one. Changing its behaviour meant changing the conditions every caller ran under.
- The failure was load-dependent. It was at its worst exactly when it mattered most — under peak transaction volume — which is also the hardest state to reproduce or safely test against.
What I did
Investigated it in production, with production’s own evidence. Observability tooling for where time was actually going, query-level analysis for what the database was doing under contention, and a review of the system design to understand why those queries were shaped the way they were. The deadlock reports were the thread worth pulling: they pointed at contention, and contention pointed at both the access patterns and the transaction boundaries.
Fixed it in three places, because one wouldn’t have held.
- Indexing. Identified which columns were actually being filtered and joined on under real load, and indexed for those access patterns rather than for how the schema had been imagined.
- Transaction handling. Corrected how database transactions were scoped and managed in code, so work was committed in appropriate units instead of holding locks across operations that had no reason to be inside the same transaction. This is what removed the contention at its source.
- Resource management and allocation. Tuned how the service acquired, used, and released its resources, so that pressure at peak no longer cascaded into the queueing that turned a slow query into a thirty-second wait.
Indexing alone would have made the queries faster while leaving locks held too long. Fixing the transaction scope alone would have reduced contention against queries that were still doing more work than they needed to. The three changes addressed the same failure at three different depths.
Verified it before it went anywhere near customers. Validated in a pre-production environment by quality engineers and software engineers together, with load and failure scenarios simulated against production-like conditions before deployment. The point was to reproduce the state the system actually failed in — peak load — rather than to confirm it worked when nothing was happening.
Outcome
| Before | After | |
|---|---|---|
| Response time | ~30s | ~318ms |
| Error rate | 38% | 0% |
| Production deadlocks | recurring | down over 99.5% |
Roughly a 94x improvement in response time, with errors eliminated and deadlocks all but gone.
The zero was measured where it counts: across normal peak transaction periods in production, which is the load profile that produced the original failures.
What I’d do differently
The capability that verified this fix should have been pointed at the system before it broke.
The load and scenario simulation built to validate the change was the right tool, used at the wrong end of the problem. Every one of these failures was load-dependent and therefore reproducible on demand — contention under concurrency is not a mystery, it is something you can provoke deliberately. Running that simulation as a routine exercise rather than as a release gate would have surfaced the deadlocks, and the access patterns underneath them, well before they became a 38% error rate in production.
The broader lesson I took: a system that fails only under load will look healthy in every environment where nobody is loading it, and will keep looking healthy right up until the moment it matters.
Working on the same thing?
Happy to go deeper on this one — get in touch.