Skip to content
Palmate Solutions

E-commerce Technology

Preventing Inventory Race Conditions in High-Volume E-Commerce: Distributed Locks, Optimistic Concurrency, and Queue Throttling

An engineering deep-dive into preventing inventory overselling during high-concurrency flash sales: row-level database locking, atomic Redis Lua scripts, distributed locks, and two-phase reservation architectures.

Robin Singh · Published · 10 min read

Share Architecture Note
PostLinkedIn

During a promotional product drop for an apparel brand, 42,000 concurrent shoppers attempted to check out 350 limited-edition units over an 80-second window. The storefront sat behind a globally distributed CDN, and the landing page performed flawlessly. Underneath, however, the checkout API collapsed: the relational database spiked to 100% CPU utilization within four seconds, database connection pools were exhausted, and row lock acquisition timed out across all application worker nodes.

When the incident team terminated stalled connections and reconciled the ledger forty minutes later, the result was catastrophic:

  • The database recorded -84 units in stock (350 physical units sold 434 times).
  • Payment gateways had captured 434 credit card authorizations, incurring over $2,400 in non-refundable merchant interchange fees.
  • Customer support received 1,200 angry tickets within three hours.
  • Automated fulfillment webhooks had already dispatched shipping orders to the third-party logistics (3PL) warehouse for stock that did not exist.

Overselling during high-volume spikes is not a warehouse bookkeeping failure. It is a fundamental concurrency defect: multiple application threads simultaneously read identical inventory numbers, independently verify availability, and execute concurrent writes without serialized coordination.

When we engineer high-throughput retail platforms at Palmate Solutions, our e-commerce development, API integration, and custom software engineering practices treat inventory control as a mission-critical distributed systems discipline.

This guide details the architectural patterns, database lock mechanics, atomic Redis Lua routines, and traffic throttling strategies required to eliminate inventory race conditions at scale.


Anatomy of the Lost Update: Why Naive ORM Code Collapses

The most common source of overselling is the classic "Read-Modify-Write" anti-pattern implemented in application-level ORMs:

TYPESCRIPT
// ANTI-PATTERN: Prone to catastrophic race conditions under concurrent load async function checkoutItemNaive(productId: string, quantity: number, customerId: string) { const product = await db.product.findUnique({ where: { id: productId } }); if (!product || product.stock < quantity) { throw new Error("Insufficient inventory"); } // Latency window: 15ms–80ms during external payment authorization await processPayment(customerId, product.price * quantity); await db.product.update({ where: { id: productId }, data: { stock: product.stock - quantity }, }); }

When two or more requests execute this routine concurrently, they read the identical initial stock value before either transaction writes its decrement:

TEXT
Time Thread 1 (Shopper A) Thread 2 (Shopper B) Database State │ Stock = 1 ├────► Read stock: 1 │ │ (Stock >= 1: TRUE) │ │ ├────► Read stock: 1 │ │ (Stock >= 1: TRUE) ├────► Payment authorized ($150) │ │ ├────► Payment authorized ($150) ├────► Write: stock = 1 - 1 = 0 │ Stock = 0 │ ├────► Write: stock = 1 - 1 = 0 Stock = 0 ▼ ▼ Result: 2 payments captured, 2 orders created, 1 physical item sold twice.

Even if the payment step is removed from the database transaction, executing separate SELECT and UPDATE statements without atomic synchronization allows interleaved execution under any concurrency level greater than one.


Relational Concurrency Control: Pessimistic vs. Optimistic Locking

To guarantee that an item is never sold below zero in a relational database (PostgreSQL, MySQL, SQL Server), updates must either hold exclusive write locks or enforce atomic predicate conditions.

Strategy 1: Pessimistic Row-Level Locking (SELECT FOR UPDATE)

Pessimistic locking explicitly blocks concurrent transactions from reading or modifying a targeted database row until the holding transaction issues a COMMIT or ROLLBACK:

SQL
-- PostgreSQL Transaction BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED; -- Acquire exclusive row lock on SKU row SELECT id, available_stock FROM inventory_items WHERE sku = 'PALM-PRO-BLK' FOR UPDATE; -- Validate stock availability inside application or SQL logic UPDATE inventory_items SET available_stock = available_stock - 1, reserved_stock = reserved_stock + 1, updated_at = NOW() WHERE sku = 'PALM-PRO-BLK' AND available_stock >= 1; COMMIT;

The Bottleneck of Pessimistic Locking

While pessimistic locking guarantees absolute consistency, it introduces severe operational constraints under extreme load:

  1. Thread Serialization: If 2,000 customers contest the same SKU, 1,999 requests block in memory, holding open database connections while waiting for the mutex.
  2. Connection Pool Exhaustion: Backed-up queries saturate application connection pools (e.g., PgBouncer, HikariCP), causing unrelated queries across the application to time out.
  3. Deadlocks on Multi-Item Carts: If Transaction A locks SKU-1 then requests SKU-2, while Transaction B locks SKU-2 and requests SKU-1, the database engine triggers a deadlock abort (40P01 in PostgreSQL).

[!IMPORTANT] To prevent multi-item cart deadlocks when using pessimistic row locks, always sort SKU IDs lexicographically within the application layer before acquiring database locks:

TYPESCRIPT
const sortedSkus = cartItems.map(item => item.sku).sort();

Strategy 2: Optimistic Concurrency Control (Atomic Decrement with Versioning)

Optimistic concurrency assumes collisions are rare or accepts failed retries instead of holding blocking locks. Rather than locking the row on read, the update executes conditionally against an internal version counter or a non-negative threshold predicate:

SQL
-- Single-statement atomic conditional update UPDATE inventory_items SET available_stock = available_stock - :requested_qty, version = version + 1, updated_at = NOW() WHERE sku = :sku AND available_stock >= :requested_qty;

In your application layer, check the database driver's rows_affected (or pg_affected_rows):

  • If rows_affected === 1, the reservation succeeded atomically.
  • If rows_affected === 0, the stock was insufficient or modified concurrently; the transaction is rejected immediately without holding locks.
TYPESCRIPT
// Production TypeScript implementation of atomic SQL reservation import { PoolClient } from "pg"; interface ReservationResult { success: boolean; reservationId?: string; failureReason?: "OUT_OF_STOCK" | "DATABASE_ERROR"; } export async function reserveInventoryAtomicSql( client: PoolClient, sku: string, quantity: number, orderId: string ): Promise<ReservationResult> { try { await client.query("BEGIN;"); // Single atomic conditional update - no row lock held prior to execution const updateResult = await client.query( `UPDATE inventory_items SET available_stock = available_stock - $1, version = version + 1, updated_at = CLOCK_TIMESTAMP() WHERE sku = $2 AND available_stock >= $1 RETURNING id, available_stock;`, [quantity, sku] ); if (updateResult.rowCount === 0) { await client.query("ROLLBACK;"); return { success: false, failureReason: "OUT_OF_STOCK" }; } const reservationId = `res_${orderId}_${Date.now()}`; await client.query( `INSERT INTO inventory_reservations (id, sku, order_id, quantity, status, expires_at) VALUES ($1, $2, $3, $4, 'PENDING_PAYMENT', NOW() + INTERVAL '10 minutes');`, [reservationId, sku, orderId, quantity] ); await client.query("COMMIT;"); return { success: true, reservationId }; } catch (error) { await client.query("ROLLBACK;"); throw error; } }

Locking Strategy Comparison Matrix

Concurrency PatternThroughput CapacityContention BehaviorDeadlock RiskImplementation ComplexityPrimary Failure Mode
Application-level ORMLow (< 500 req/sec)Non-blocking, invalidZeroVery LowCatastrophic overselling (silent corruption)
Pessimistic Row Lock (FOR UPDATE)Moderate (~1,200 req/sec)Blocks caller; thread queuingHigh (on multi-SKU carts)LowConnection pool exhaustion, timeouts
Pessimistic Non-Blocking (SKIP LOCKED)High (Queue workers)Skips locked rowsZeroModerateSkips items in queue worker pipelines
Optimistic Predicate (stock >= qty)High (~4,500 req/sec)Instant failure on 0 rowsZeroModerateHigh retry churn on extreme spikes
Distributed Redis Lua ScriptExtreme (> 35,000 req/sec)Single-threaded in-memory lockZeroHighCache-to-DB reconciliation divergence

High-Throughput In-Memory Reservations with Redis and Lua

When flash sales exceed 5,000 requests per second on a single SKU, hitting the primary relational database—even with atomic optimistic queries—creates unmanageable I/O pressure and replication lag.

The industry-standard solution decouples the ephemeral reservation hot-path from the durable relational ledger using Redis. Because Redis processes commands on a single-threaded event loop, a custom Lua script executes with guaranteed atomicity across multiple keys.

TEXT
┌───────────────────────────────────────────────────────────────────────────────────┐ │ FLASH SALE ARCHITECTURE: TWO-PHASE INVENTORY RESERVATION │ │ │ │ [ Incoming Flash Checkout Traffic (50,000 req/min) ] │ │ │ │ │ ▼ │ │ ┌───────────────────────┐ │ │ │ API Gateway / Ingress │ │ │ │ (Token Bucket Limiter)│ │ │ └───────────┬───────────┘ │ │ │ │ │ ▼ │ │ ┌───────────────────────────┐ │ │ │ Phase 1: Soft Reservation │ │ │ │ (Atomic Redis Lua Script) │ │ │ └─────────────┬─────────────┘ │ │ ▲ │ │ │ Stock Exhausted? │ │ Reservation Granted (10-Minute Lease) │ │ HTTP 409 Rejected│ ▼ │ │ ┌───────────────────────────┐ │ │ │ Customer Payment Gateway │ │ │ │ (Stripe, Adyen, Apple Pay)│ │ │ └─────────────┬─────────────┘ │ │ │ │ │ ┌──────────────┴──────────────┐ │ │ │ │ │ │ Payment Succeeded Payment Failed / Expired │ │ ▼ ▼ │ │ ┌───────────────────────────┐ ┌───────────────────────────┐ │ │ │ Phase 2: Hard Allocation │ │ Background Janitor Worker │ │ │ │ - Commit to Postgres │ │ - Redis INCRBY stock back │ │ │ │ - Kafka/Event Dispatch │ │ - Del Reservation Key │ │ │ │ - Fulfillment Execution │ │ - Release lease │ │ │ └───────────────────────────┘ └───────────────────────────┘ │ └───────────────────────────────────────────────────────────────────────────────────┘

Production Redis Lua Reservation Script

This script validates that stock is available, decrements the active balance, stores an expiring reservation hash, and returns an atomic decision—all in a single round-trip:

LUA
-- File: reserve_stock.lua -- KEYS[1]: stock key (e.g. "inventory:stock:SKU-9988") -- KEYS[2]: reservation key (e.g. "inventory:res:SKU-9988:ORDER-1234") -- ARGV[1]: requested quantity (integer string) -- ARGV[2]: reservation TTL in seconds (e.g. 600 for 10 minutes) -- ARGV[3]: customer/order metadata string local stock_key = KEYS[1] local res_key = KEYS[2] local req_qty = tonumber(ARGV[1]) local ttl_seconds = tonumber(ARGV[2]) local metadata = ARGV[3] -- 1. Check if reservation already exists (Idempotency check) if redis.call("EXISTS", res_key) == 1 then return {1, "ALREADY_RESERVED"} end -- 2. Fetch current stock local current_stock = tonumber(redis.call("GET", stock_key)) -- If key does not exist or stock is below required quantity, reject if not current_stock or current_stock < req_qty then return {0, "OUT_OF_STOCK"} end -- 3. Atomic Decrement and Lease Creation redis.call("DECRBY", stock_key, req_qty) -- Create reservation record with expiration TTL redis.call("HSET", res_key, "quantity", req_qty, "metadata", metadata, "created_at", redis.call("TIME")[1] ) redis.call("EXPIRE", res_key, ttl_seconds) -- Return success with remaining stock local remaining_stock = current_stock - req_qty return {1, tostring(remaining_stock)}

Node.js / TypeScript Redis Reservation Client

Using ioredis, we define the command once and execute it with strict TypeScript interfaces:

TYPESCRIPT
import Redis from "ioredis"; import fs from "node:fs"; import path from "node:path"; const redis = new Redis(process.env.REDIS_URL || "redis://localhost:6379"); // Load Lua script const reserveLuaScript = fs.readFileSync( path.join(__dirname, "reserve_stock.lua"), "utf8" ); redis.defineCommand("reserveStockLua", { numberOfKeys: 2, lua: reserveLuaScript, }); declare module "ioredis" { interface RedisCommander<Context> { reserveStockLua( stockKey: string, reservationKey: string, quantity: number, ttlSeconds: number, metadata: string ): Promise<[number, string]>; } } export interface InventoryReservationParams { sku: string; orderId: string; quantity: number; ttlSeconds?: number; } export async function attemptInventoryHold(params: InventoryReservationParams) { const { sku, orderId, quantity, ttlSeconds = 600 } = params; const stockKey = `inventory:stock:${sku}`; const reservationKey = `inventory:res:${sku}:${orderId}`; try { const [statusCode, responseMessage] = await redis.reserveStockLua( stockKey, reservationKey, quantity, ttlSeconds, JSON.stringify({ orderId, requestedAt: new Date().toISOString() }) ); if (statusCode === 1) { return { success: true, reservationKey, detail: responseMessage, }; } return { success: false, reason: responseMessage, // "OUT_OF_STOCK" }; } catch (err) { // Fail closed to prevent overselling on infrastructure error console.error(`[Inventory Error] Redis reservation failed for SKU: ${sku}`, err); throw new Error("Inventory service temporarily unavailable"); } }

Two-Phase Checkout: Soft Reservation vs. Hard Allocation

Treating checkout as an all-or-nothing database write creates payment vulnerabilities. If your database writes the inventory decrement after the customer's credit card is charged, and the database subsequently errors out, the customer has been billed for an item that is out of stock. If you decrement before payment without a lease, abandoned carts permanently deplete available stock.

A resilient e-commerce architecture divides checkout into two deterministic phases:

Phase 1: Soft Reservation (Hold with TTL)

  1. Cart / Checkout Initiation: When the buyer clicks "Proceed to Payment", the API calls the atomic reservation routine.
  2. Stock Decrement: Available stock decrements in Redis immediately.
  3. Lease Expiry: A reservation hash is stored with a strict Time-to-Live (e.g., 10 minutes).
  4. Idempotency Key: The reservation key is tied to the unique payment intent or checkout session ID.

Phase 2: Hard Allocation (Commitment on Payment Webhook)

  1. Webhook Processing: The payment processor sends an asynchronous payment_intent.succeeded webhook.
  2. Idempotent Verification: The backend verifies the reservation key exists in Redis and has not expired.
  3. Ledger Commit: An atomic SQL transaction records the final order in PostgreSQL, logs the inventory audit line, and changes the reservation status to FULFILLED.
  4. Redis Cleanup: The reservation key is deleted, or marked completed so a background janitor worker will not release the inventory.

Abandonment and Expiration (The Janitor Routine)

What happens if the buyer closes their browser tab or their card declines?

Do not rely exclusively on Redis EXPIRE keyspace notifications (notify-keyspace-events), as Redis pub/sub delivers notifications via fire-and-forget; if your worker restarts or lags, expiration messages are dropped permanently.

Instead, employ a reliable periodic reconciliation sweep:

TYPESCRIPT
// Background reconciliation worker running every 60 seconds export async function reconcileExpiredReservations(sku: string) { const stream = redis.scanStream({ match: `inventory:res:${sku}:*`, count: 100, }); stream.on("data", async (keys: string[]) => { for (const key of keys) { const ttl = await redis.ttl(key); // If key is expired or has negative TTL without being fulfilled if (ttl <= 0) { const reservationData = await redis.hgetall(key); if (reservationData && reservationData.quantity) { const qty = parseInt(reservationData.quantity, 10); // Re-increment stock atomically await redis.incrby(`inventory:stock:${sku}`, qty); await redis.del(key); console.log(`[Janitor] Released ${qty} units for expired reservation ${key}`); } } } }); }

Traffic Shaping and Queue Throttling: The Virtual Waiting Room

Even with optimized in-memory Redis scripts, an unchecked influx of 100,000 HTTP requests hitting application server pods simultaneously will saturate ingress network bandwidth, trigger TCP socket exhaustion, and drive up memory consumption.

To protect backend services, high-demand sales must decouple arrival rate from execution rate.

TEXT
HIGH-VELOCITY TRAFFIC (50,000 req/sec) │ ▼ ┌───────────────────────────────────────┐ │ Cloudflare / Edge API Gateway │ │ (Virtual Waiting Room / Token Bucket) │ └───────────────────┬───────────────────┘ │ Regulated Flow (1,500 req/sec) ▼ ┌───────────────────────────────────────┐ │ Ingestion Queue (BullMQ / SQS / Kafka)│ │ Partitioned by SKU hash key │ └───────────────────┬───────────────────┘ │ ▼ ┌───────────────────────────────────────┐ │ Serialized Inventory Workers │ │ (Atomic Reservation Execution) │ └───────────────────────────────────────┘

1. Partitioned Message Queues

When orders pass through an asynchronous queue (such as Apache Kafka, AWS SQS FIFO, or Redis BullMQ), message ordering is guaranteed within a partition.

By assigning the SKU ID as the message partition key, all checkout requests for SKU-001 arrive sequentially at a single designated worker thread. This eliminates concurrency contention entirely: orders for the same item are evaluated serially in memory at microseconds per transaction, completely sidestepping database lock waits.

2. Upstream Token-Bucket Admission Control

For immediate, synchronous REST APIs, implement a distributed token bucket limiter at your edge proxy or gateway. Only requests that secure an admission token proceed to the checkout reservation endpoint; excess requests receive standard HTTP 429 Too Many Requests responses with a Retry-After header:

TYPESCRIPT
// Edge Token Bucket Middleware (Express/NestJS) export async function flashSaleRateLimiter(req: Request, res: Response, next: NextFunction) { const sku = req.body.sku; const bucketKey = `ratelimit:flash:${sku}`; // Rate: 200 admissions per second with a burst capacity of 500 const allowed = await redis.eval( tokenBucketLuaScript, 1, bucketKey, 200, // replenish rate per sec 500, // capacity Date.now() ); if (!allowed) { res.setHeader("Retry-After", "2"); return res.status(429).json({ error: "HIGH_TRAFFIC_WAIT", message: "The checkout room is at capacity. Please retry momentarily.", }); } next(); }

Common Edge Cases and Production Failure Modes

Building production-resilient e-commerce systems requires anticipating real-world infrastructure failures:

1. The Delayed Webhook Dilemma

The Failure: A customer begins checkout at 12:00:00. The reservation lease is set to 10 minutes (expires at 12:10:00). At 12:09:50, the customer completes 3D-Secure card verification, and their bank captures the funds. Due to network congestion between the payment gateway and your servers, the webhook arrives at 12:11:15—one minute after the janitor has released the reservation and sold the unit to someone else.

The Fix: In your payment webhook handler, verify stock allocation before creating the order. If the reservation has expired and available stock is zero, do not fail silently:

  1. Immediately execute an automated, full payment reversal via API:
    TYPESCRIPT
    await stripe.refunds.create({ payment_intent: paymentIntentId, reason: "duplicate", metadata: { reason: "STOCK_EXPIRED_PRIOR_TO_CAPTURE" }, });
  2. Send an automated email notification explaining that the reservation window elapsed before payment authorization completed, and funds were immediately returned.

2. Multi-Item Cart Deadlocks across Distributed Redis Locks

If a customer attempts to purchase SKU-A and SKU-B simultaneously, and you use distributed locks (such as Redlock), acquiring lock A on Node 1 and lock B on Node 2 can cause deadlocks if another transaction attempts to acquire lock B then lock A.

The Fix: Never acquire distributed locks in arbitrary order. Always sort resource identifiers canonically before lock acquisition:

TYPESCRIPT
const sortedResourceKeys = items .map(i => `lock:inventory:${i.sku}`) .sort(); // Acquire locks strictly in ascending order for (const lockKey of sortedResourceKeys) { await acquireLock(lockKey); }

3. Redis Persistence Lag (RDB vs. AOF)

By default, Redis saves snapshots to disk using RDB at periodic intervals (e.g., every 60 seconds). If your Redis primary instance crashes during a flash sale, up to 60 seconds of atomic inventory decrements could be lost upon failover, resurrecting sold-out inventory.

The Fix: For nodes managing financial or inventory state, configure Append-Only File (AOF) persistence with everysec synchronization:

CONF
appendonly yes appendfsync everysec

On replica failover, perform an automated audit comparing the Redis stock balance against the sum of active PostgreSQL reservation ledgers before routing customer traffic back to the cluster.


Production Implementation Checklist

Before launching high-concurrency promotions or product drops, verify your architecture against this engineering checklist:

  • No Naked Read-Modify-Write Cycles: Verified that zero application endpoints read stock into an ORM model and subsequently write stock - 1 without conditional predicates or locks.
  • Database Constraints Enforced: The relational database schema enforces an unsigned constraint or a CHECK (available_stock >= 0) constraint on all inventory tables.
  • Canonical Sorting on Multi-SKU Carts: All multi-item checkout operations sort SKU identifiers in memory before initiating database transactions or lock acquisitions.
  • Atomic Hot-Path Scripts: High-velocity product drops route inventory decrements through an atomic Redis Lua script rather than raw relational table updates.
  • Idempotent Webhook Handlers: Payment authorization webhooks store a processed event identifier in the database to prevent duplicate decrements on network retries.
  • Reconciliation Janitor Deployed: A reliable background daemon monitors expired soft reservations and returns unpurchased stock to the pool.
  • Edge Rate Limiting: Traffic shaping (HTTP 429 backoff or virtual waiting rooms) throttles upstream surges before they saturate backend compute or connection pools.
  • Automated Payment Reversals: Webhook consumers are programmed to issue an automatic refund if an asynchronous payment clears after an inventory reservation lease has expired.

To estimate technical timelines, infrastructure footprints, and integration budgets for enterprise retail systems, utilize Palmate's API project estimator or evaluate infrastructure reliability targets with our uptime calculator.


Authoritative References & Standards

  1. ISO/IEC 9075-2: Information Technology — Database Languages — SQL: Foundation specification defining ACID transaction isolation levels (READ COMMITTED, REPEATABLE READ, SERIALIZABLE) and serializability anomaly phenomena.
  2. IETF RFC 6585: Additional HTTP Status Codes: Section 4 defines the specification and header requirements for 429 Too Many Requests rate limiting. https://www.rfc-editor.org/rfc/rfc6585
  3. Redis Official Documentation: Distributed Locks with Redis and Lua Scripting: Detailed implementation guide for atomic multi-key scripting and Redlock distributed consensus mechanics. https://redis.io/docs/manual/patterns/distributed-locks/
  4. Martin Kleppmann: "How to do distributed locking": Rigorous distributed systems critique on lock safety, clock drift assumptions, and fencing tokens. https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
Share Architecture Note
PostLinkedIn