Preventing Inventory Race Conditions in High-Volume E-Commerce: Distributed Locks, Optimistic Concurrency, and Queue Throttling
An engineering deep-dive into preventing inventory overselling during high-concurrency flash sales: row-level database locking, atomic Redis Lua scripts, distributed locks, and two-phase reservation architectures.
Robin Singh · Published · 10 min read
During a promotional product drop for an apparel brand, 42,000 concurrent shoppers attempted to check out 350 limited-edition units over an 80-second window. The storefront sat behind a globally distributed CDN, and the landing page performed flawlessly. Underneath, however, the checkout API collapsed: the relational database spiked to 100% CPU utilization within four seconds, database connection pools were exhausted, and row lock acquisition timed out across all application worker nodes.
When the incident team terminated stalled connections and reconciled the ledger forty minutes later, the result was catastrophic:
- The database recorded -84 units in stock (350 physical units sold 434 times).
- Payment gateways had captured 434 credit card authorizations, incurring over $2,400 in non-refundable merchant interchange fees.
- Customer support received 1,200 angry tickets within three hours.
- Automated fulfillment webhooks had already dispatched shipping orders to the third-party logistics (3PL) warehouse for stock that did not exist.
Overselling during high-volume spikes is not a warehouse bookkeeping failure. It is a fundamental concurrency defect: multiple application threads simultaneously read identical inventory numbers, independently verify availability, and execute concurrent writes without serialized coordination.
When we engineer high-throughput retail platforms at Palmate Solutions, our e-commerce development, API integration, and custom software engineering practices treat inventory control as a mission-critical distributed systems discipline.
This guide details the architectural patterns, database lock mechanics, atomic Redis Lua routines, and traffic throttling strategies required to eliminate inventory race conditions at scale.
Anatomy of the Lost Update: Why Naive ORM Code Collapses
The most common source of overselling is the classic "Read-Modify-Write" anti-pattern implemented in application-level ORMs:
// ANTI-PATTERN: Prone to catastrophic race conditions under concurrent load
async function checkoutItemNaive(productId: string, quantity: number, customerId: string) {
const product = await db.product.findUnique({ where: { id: productId } });
if (!product || product.stock < quantity) {
throw new Error("Insufficient inventory");
}
// Latency window: 15ms–80ms during external payment authorization
await processPayment(customerId, product.price * quantity);
await db.product.update({
where: { id: productId },
data: { stock: product.stock - quantity },
});
}
When two or more requests execute this routine concurrently, they read the identical initial stock value before either transaction writes its decrement:
Time Thread 1 (Shopper A) Thread 2 (Shopper B) Database State
│ Stock = 1
├────► Read stock: 1 │
│ (Stock >= 1: TRUE) │
│ ├────► Read stock: 1
│ │ (Stock >= 1: TRUE)
├────► Payment authorized ($150) │
│ ├────► Payment authorized ($150)
├────► Write: stock = 1 - 1 = 0 │ Stock = 0
│ ├────► Write: stock = 1 - 1 = 0 Stock = 0
▼ ▼
Result: 2 payments captured, 2 orders created, 1 physical item sold twice.
Even if the payment step is removed from the database transaction, executing separate SELECT and UPDATE statements without atomic synchronization allows interleaved execution under any concurrency level greater than one.
Relational Concurrency Control: Pessimistic vs. Optimistic Locking
To guarantee that an item is never sold below zero in a relational database (PostgreSQL, MySQL, SQL Server), updates must either hold exclusive write locks or enforce atomic predicate conditions.
Strategy 1: Pessimistic Row-Level Locking (SELECT FOR UPDATE)
Pessimistic locking explicitly blocks concurrent transactions from reading or modifying a targeted database row until the holding transaction issues a COMMIT or ROLLBACK:
-- PostgreSQL Transaction
BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;
-- Acquire exclusive row lock on SKU row
SELECT id, available_stock
FROM inventory_items
WHERE sku = 'PALM-PRO-BLK'
FOR UPDATE;
-- Validate stock availability inside application or SQL logic
UPDATE inventory_items
SET available_stock = available_stock - 1,
reserved_stock = reserved_stock + 1,
updated_at = NOW()
WHERE sku = 'PALM-PRO-BLK'
AND available_stock >= 1;
COMMIT;
The Bottleneck of Pessimistic Locking
While pessimistic locking guarantees absolute consistency, it introduces severe operational constraints under extreme load:
- Thread Serialization: If 2,000 customers contest the same SKU, 1,999 requests block in memory, holding open database connections while waiting for the mutex.
- Connection Pool Exhaustion: Backed-up queries saturate application connection pools (e.g., PgBouncer, HikariCP), causing unrelated queries across the application to time out.
- Deadlocks on Multi-Item Carts: If Transaction A locks
SKU-1then requestsSKU-2, while Transaction B locksSKU-2and requestsSKU-1, the database engine triggers a deadlock abort (40P01in PostgreSQL).
[!IMPORTANT] To prevent multi-item cart deadlocks when using pessimistic row locks, always sort SKU IDs lexicographically within the application layer before acquiring database locks:
TYPESCRIPTconst sortedSkus = cartItems.map(item => item.sku).sort();
Strategy 2: Optimistic Concurrency Control (Atomic Decrement with Versioning)
Optimistic concurrency assumes collisions are rare or accepts failed retries instead of holding blocking locks. Rather than locking the row on read, the update executes conditionally against an internal version counter or a non-negative threshold predicate:
-- Single-statement atomic conditional update
UPDATE inventory_items
SET available_stock = available_stock - :requested_qty,
version = version + 1,
updated_at = NOW()
WHERE sku = :sku
AND available_stock >= :requested_qty;
In your application layer, check the database driver's rows_affected (or pg_affected_rows):
- If
rows_affected === 1, the reservation succeeded atomically. - If
rows_affected === 0, the stock was insufficient or modified concurrently; the transaction is rejected immediately without holding locks.
// Production TypeScript implementation of atomic SQL reservation
import { PoolClient } from "pg";
interface ReservationResult {
success: boolean;
reservationId?: string;
failureReason?: "OUT_OF_STOCK" | "DATABASE_ERROR";
}
export async function reserveInventoryAtomicSql(
client: PoolClient,
sku: string,
quantity: number,
orderId: string
): Promise<ReservationResult> {
try {
await client.query("BEGIN;");
// Single atomic conditional update - no row lock held prior to execution
const updateResult = await client.query(
`UPDATE inventory_items
SET available_stock = available_stock - $1,
version = version + 1,
updated_at = CLOCK_TIMESTAMP()
WHERE sku = $2
AND available_stock >= $1
RETURNING id, available_stock;`,
[quantity, sku]
);
if (updateResult.rowCount === 0) {
await client.query("ROLLBACK;");
return { success: false, failureReason: "OUT_OF_STOCK" };
}
const reservationId = `res_${orderId}_${Date.now()}`;
await client.query(
`INSERT INTO inventory_reservations (id, sku, order_id, quantity, status, expires_at)
VALUES ($1, $2, $3, $4, 'PENDING_PAYMENT', NOW() + INTERVAL '10 minutes');`,
[reservationId, sku, orderId, quantity]
);
await client.query("COMMIT;");
return { success: true, reservationId };
} catch (error) {
await client.query("ROLLBACK;");
throw error;
}
}
Locking Strategy Comparison Matrix
| Concurrency Pattern | Throughput Capacity | Contention Behavior | Deadlock Risk | Implementation Complexity | Primary Failure Mode |
|---|---|---|---|---|---|
| Application-level ORM | Low (< 500 req/sec) | Non-blocking, invalid | Zero | Very Low | Catastrophic overselling (silent corruption) |
Pessimistic Row Lock (FOR UPDATE) | Moderate (~1,200 req/sec) | Blocks caller; thread queuing | High (on multi-SKU carts) | Low | Connection pool exhaustion, timeouts |
Pessimistic Non-Blocking (SKIP LOCKED) | High (Queue workers) | Skips locked rows | Zero | Moderate | Skips items in queue worker pipelines |
Optimistic Predicate (stock >= qty) | High (~4,500 req/sec) | Instant failure on 0 rows | Zero | Moderate | High retry churn on extreme spikes |
| Distributed Redis Lua Script | Extreme (> 35,000 req/sec) | Single-threaded in-memory lock | Zero | High | Cache-to-DB reconciliation divergence |
High-Throughput In-Memory Reservations with Redis and Lua
When flash sales exceed 5,000 requests per second on a single SKU, hitting the primary relational database—even with atomic optimistic queries—creates unmanageable I/O pressure and replication lag.
The industry-standard solution decouples the ephemeral reservation hot-path from the durable relational ledger using Redis. Because Redis processes commands on a single-threaded event loop, a custom Lua script executes with guaranteed atomicity across multiple keys.
┌───────────────────────────────────────────────────────────────────────────────────┐
│ FLASH SALE ARCHITECTURE: TWO-PHASE INVENTORY RESERVATION │
│ │
│ [ Incoming Flash Checkout Traffic (50,000 req/min) ] │
│ │ │
│ ▼ │
│ ┌───────────────────────┐ │
│ │ API Gateway / Ingress │ │
│ │ (Token Bucket Limiter)│ │
│ └───────────┬───────────┘ │
│ │ │
│ ▼ │
│ ┌───────────────────────────┐ │
│ │ Phase 1: Soft Reservation │ │
│ │ (Atomic Redis Lua Script) │ │
│ └─────────────┬─────────────┘ │
│ ▲ │ │
│ Stock Exhausted? │ │ Reservation Granted (10-Minute Lease) │
│ HTTP 409 Rejected│ ▼ │
│ ┌───────────────────────────┐ │
│ │ Customer Payment Gateway │ │
│ │ (Stripe, Adyen, Apple Pay)│ │
│ └─────────────┬─────────────┘ │
│ │ │
│ ┌──────────────┴──────────────┐ │
│ │ │ │
│ Payment Succeeded Payment Failed / Expired │
│ ▼ ▼ │
│ ┌───────────────────────────┐ ┌───────────────────────────┐ │
│ │ Phase 2: Hard Allocation │ │ Background Janitor Worker │ │
│ │ - Commit to Postgres │ │ - Redis INCRBY stock back │ │
│ │ - Kafka/Event Dispatch │ │ - Del Reservation Key │ │
│ │ - Fulfillment Execution │ │ - Release lease │ │
│ └───────────────────────────┘ └───────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────────────┘
Production Redis Lua Reservation Script
This script validates that stock is available, decrements the active balance, stores an expiring reservation hash, and returns an atomic decision—all in a single round-trip:
-- File: reserve_stock.lua
-- KEYS[1]: stock key (e.g. "inventory:stock:SKU-9988")
-- KEYS[2]: reservation key (e.g. "inventory:res:SKU-9988:ORDER-1234")
-- ARGV[1]: requested quantity (integer string)
-- ARGV[2]: reservation TTL in seconds (e.g. 600 for 10 minutes)
-- ARGV[3]: customer/order metadata string
local stock_key = KEYS[1]
local res_key = KEYS[2]
local req_qty = tonumber(ARGV[1])
local ttl_seconds = tonumber(ARGV[2])
local metadata = ARGV[3]
-- 1. Check if reservation already exists (Idempotency check)
if redis.call("EXISTS", res_key) == 1 then
return {1, "ALREADY_RESERVED"}
end
-- 2. Fetch current stock
local current_stock = tonumber(redis.call("GET", stock_key))
-- If key does not exist or stock is below required quantity, reject
if not current_stock or current_stock < req_qty then
return {0, "OUT_OF_STOCK"}
end
-- 3. Atomic Decrement and Lease Creation
redis.call("DECRBY", stock_key, req_qty)
-- Create reservation record with expiration TTL
redis.call("HSET", res_key,
"quantity", req_qty,
"metadata", metadata,
"created_at", redis.call("TIME")[1]
)
redis.call("EXPIRE", res_key, ttl_seconds)
-- Return success with remaining stock
local remaining_stock = current_stock - req_qty
return {1, tostring(remaining_stock)}
Node.js / TypeScript Redis Reservation Client
Using ioredis, we define the command once and execute it with strict TypeScript interfaces:
import Redis from "ioredis";
import fs from "node:fs";
import path from "node:path";
const redis = new Redis(process.env.REDIS_URL || "redis://localhost:6379");
// Load Lua script
const reserveLuaScript = fs.readFileSync(
path.join(__dirname, "reserve_stock.lua"),
"utf8"
);
redis.defineCommand("reserveStockLua", {
numberOfKeys: 2,
lua: reserveLuaScript,
});
declare module "ioredis" {
interface RedisCommander<Context> {
reserveStockLua(
stockKey: string,
reservationKey: string,
quantity: number,
ttlSeconds: number,
metadata: string
): Promise<[number, string]>;
}
}
export interface InventoryReservationParams {
sku: string;
orderId: string;
quantity: number;
ttlSeconds?: number;
}
export async function attemptInventoryHold(params: InventoryReservationParams) {
const { sku, orderId, quantity, ttlSeconds = 600 } = params;
const stockKey = `inventory:stock:${sku}`;
const reservationKey = `inventory:res:${sku}:${orderId}`;
try {
const [statusCode, responseMessage] = await redis.reserveStockLua(
stockKey,
reservationKey,
quantity,
ttlSeconds,
JSON.stringify({ orderId, requestedAt: new Date().toISOString() })
);
if (statusCode === 1) {
return {
success: true,
reservationKey,
detail: responseMessage,
};
}
return {
success: false,
reason: responseMessage, // "OUT_OF_STOCK"
};
} catch (err) {
// Fail closed to prevent overselling on infrastructure error
console.error(`[Inventory Error] Redis reservation failed for SKU: ${sku}`, err);
throw new Error("Inventory service temporarily unavailable");
}
}
Two-Phase Checkout: Soft Reservation vs. Hard Allocation
Treating checkout as an all-or-nothing database write creates payment vulnerabilities. If your database writes the inventory decrement after the customer's credit card is charged, and the database subsequently errors out, the customer has been billed for an item that is out of stock. If you decrement before payment without a lease, abandoned carts permanently deplete available stock.
A resilient e-commerce architecture divides checkout into two deterministic phases:
Phase 1: Soft Reservation (Hold with TTL)
- Cart / Checkout Initiation: When the buyer clicks "Proceed to Payment", the API calls the atomic reservation routine.
- Stock Decrement: Available stock decrements in Redis immediately.
- Lease Expiry: A reservation hash is stored with a strict Time-to-Live (e.g., 10 minutes).
- Idempotency Key: The reservation key is tied to the unique payment intent or checkout session ID.
Phase 2: Hard Allocation (Commitment on Payment Webhook)
- Webhook Processing: The payment processor sends an asynchronous
payment_intent.succeededwebhook. - Idempotent Verification: The backend verifies the reservation key exists in Redis and has not expired.
- Ledger Commit: An atomic SQL transaction records the final order in PostgreSQL, logs the inventory audit line, and changes the reservation status to
FULFILLED. - Redis Cleanup: The reservation key is deleted, or marked completed so a background janitor worker will not release the inventory.
Abandonment and Expiration (The Janitor Routine)
What happens if the buyer closes their browser tab or their card declines?
Do not rely exclusively on Redis EXPIRE keyspace notifications (notify-keyspace-events), as Redis pub/sub delivers notifications via fire-and-forget; if your worker restarts or lags, expiration messages are dropped permanently.
Instead, employ a reliable periodic reconciliation sweep:
// Background reconciliation worker running every 60 seconds
export async function reconcileExpiredReservations(sku: string) {
const stream = redis.scanStream({
match: `inventory:res:${sku}:*`,
count: 100,
});
stream.on("data", async (keys: string[]) => {
for (const key of keys) {
const ttl = await redis.ttl(key);
// If key is expired or has negative TTL without being fulfilled
if (ttl <= 0) {
const reservationData = await redis.hgetall(key);
if (reservationData && reservationData.quantity) {
const qty = parseInt(reservationData.quantity, 10);
// Re-increment stock atomically
await redis.incrby(`inventory:stock:${sku}`, qty);
await redis.del(key);
console.log(`[Janitor] Released ${qty} units for expired reservation ${key}`);
}
}
}
});
}
Traffic Shaping and Queue Throttling: The Virtual Waiting Room
Even with optimized in-memory Redis scripts, an unchecked influx of 100,000 HTTP requests hitting application server pods simultaneously will saturate ingress network bandwidth, trigger TCP socket exhaustion, and drive up memory consumption.
To protect backend services, high-demand sales must decouple arrival rate from execution rate.
HIGH-VELOCITY TRAFFIC (50,000 req/sec)
│
▼
┌───────────────────────────────────────┐
│ Cloudflare / Edge API Gateway │
│ (Virtual Waiting Room / Token Bucket) │
└───────────────────┬───────────────────┘
│ Regulated Flow (1,500 req/sec)
▼
┌───────────────────────────────────────┐
│ Ingestion Queue (BullMQ / SQS / Kafka)│
│ Partitioned by SKU hash key │
└───────────────────┬───────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Serialized Inventory Workers │
│ (Atomic Reservation Execution) │
└───────────────────────────────────────┘
1. Partitioned Message Queues
When orders pass through an asynchronous queue (such as Apache Kafka, AWS SQS FIFO, or Redis BullMQ), message ordering is guaranteed within a partition.
By assigning the SKU ID as the message partition key, all checkout requests for SKU-001 arrive sequentially at a single designated worker thread. This eliminates concurrency contention entirely: orders for the same item are evaluated serially in memory at microseconds per transaction, completely sidestepping database lock waits.
2. Upstream Token-Bucket Admission Control
For immediate, synchronous REST APIs, implement a distributed token bucket limiter at your edge proxy or gateway. Only requests that secure an admission token proceed to the checkout reservation endpoint; excess requests receive standard HTTP 429 Too Many Requests responses with a Retry-After header:
// Edge Token Bucket Middleware (Express/NestJS)
export async function flashSaleRateLimiter(req: Request, res: Response, next: NextFunction) {
const sku = req.body.sku;
const bucketKey = `ratelimit:flash:${sku}`;
// Rate: 200 admissions per second with a burst capacity of 500
const allowed = await redis.eval(
tokenBucketLuaScript,
1,
bucketKey,
200, // replenish rate per sec
500, // capacity
Date.now()
);
if (!allowed) {
res.setHeader("Retry-After", "2");
return res.status(429).json({
error: "HIGH_TRAFFIC_WAIT",
message: "The checkout room is at capacity. Please retry momentarily.",
});
}
next();
}
Common Edge Cases and Production Failure Modes
Building production-resilient e-commerce systems requires anticipating real-world infrastructure failures:
1. The Delayed Webhook Dilemma
The Failure: A customer begins checkout at 12:00:00. The reservation lease is set to 10 minutes (expires at 12:10:00). At 12:09:50, the customer completes 3D-Secure card verification, and their bank captures the funds. Due to network congestion between the payment gateway and your servers, the webhook arrives at 12:11:15—one minute after the janitor has released the reservation and sold the unit to someone else.
The Fix: In your payment webhook handler, verify stock allocation before creating the order. If the reservation has expired and available stock is zero, do not fail silently:
- Immediately execute an automated, full payment reversal via API:
TYPESCRIPT
await stripe.refunds.create({ payment_intent: paymentIntentId, reason: "duplicate", metadata: { reason: "STOCK_EXPIRED_PRIOR_TO_CAPTURE" }, }); - Send an automated email notification explaining that the reservation window elapsed before payment authorization completed, and funds were immediately returned.
2. Multi-Item Cart Deadlocks across Distributed Redis Locks
If a customer attempts to purchase SKU-A and SKU-B simultaneously, and you use distributed locks (such as Redlock), acquiring lock A on Node 1 and lock B on Node 2 can cause deadlocks if another transaction attempts to acquire lock B then lock A.
The Fix: Never acquire distributed locks in arbitrary order. Always sort resource identifiers canonically before lock acquisition:
const sortedResourceKeys = items
.map(i => `lock:inventory:${i.sku}`)
.sort();
// Acquire locks strictly in ascending order
for (const lockKey of sortedResourceKeys) {
await acquireLock(lockKey);
}
3. Redis Persistence Lag (RDB vs. AOF)
By default, Redis saves snapshots to disk using RDB at periodic intervals (e.g., every 60 seconds). If your Redis primary instance crashes during a flash sale, up to 60 seconds of atomic inventory decrements could be lost upon failover, resurrecting sold-out inventory.
The Fix:
For nodes managing financial or inventory state, configure Append-Only File (AOF) persistence with everysec synchronization:
appendonly yes
appendfsync everysec
On replica failover, perform an automated audit comparing the Redis stock balance against the sum of active PostgreSQL reservation ledgers before routing customer traffic back to the cluster.
Production Implementation Checklist
Before launching high-concurrency promotions or product drops, verify your architecture against this engineering checklist:
- No Naked Read-Modify-Write Cycles: Verified that zero application endpoints read stock into an ORM model and subsequently write
stock - 1without conditional predicates or locks. - Database Constraints Enforced: The relational database schema enforces an unsigned constraint or a
CHECK (available_stock >= 0)constraint on all inventory tables. - Canonical Sorting on Multi-SKU Carts: All multi-item checkout operations sort SKU identifiers in memory before initiating database transactions or lock acquisitions.
- Atomic Hot-Path Scripts: High-velocity product drops route inventory decrements through an atomic Redis Lua script rather than raw relational table updates.
- Idempotent Webhook Handlers: Payment authorization webhooks store a processed event identifier in the database to prevent duplicate decrements on network retries.
- Reconciliation Janitor Deployed: A reliable background daemon monitors expired soft reservations and returns unpurchased stock to the pool.
- Edge Rate Limiting: Traffic shaping (HTTP 429 backoff or virtual waiting rooms) throttles upstream surges before they saturate backend compute or connection pools.
- Automated Payment Reversals: Webhook consumers are programmed to issue an automatic refund if an asynchronous payment clears after an inventory reservation lease has expired.
To estimate technical timelines, infrastructure footprints, and integration budgets for enterprise retail systems, utilize Palmate's API project estimator or evaluate infrastructure reliability targets with our uptime calculator.
Authoritative References & Standards
- ISO/IEC 9075-2: Information Technology — Database Languages — SQL: Foundation specification defining ACID transaction isolation levels (
READ COMMITTED,REPEATABLE READ,SERIALIZABLE) and serializability anomaly phenomena. - IETF RFC 6585: Additional HTTP Status Codes: Section 4 defines the specification and header requirements for
429 Too Many Requestsrate limiting. https://www.rfc-editor.org/rfc/rfc6585 - Redis Official Documentation: Distributed Locks with Redis and Lua Scripting: Detailed implementation guide for atomic multi-key scripting and Redlock distributed consensus mechanics. https://redis.io/docs/manual/patterns/distributed-locks/
- Martin Kleppmann: "How to do distributed locking": Rigorous distributed systems critique on lock safety, clock drift assumptions, and fencing tokens. https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
