Distributed locks and fencing tokens: why TTL alone is not enough

You deploy a nightly job that rebuilds a search index. Two pods wake up at the same second; both acquire what they believe is an exclusive lock and start writing to the same object storage prefix. The index is corrupted, but the pipeline reports success because each task finished without throwing. In freelance and consulting engagements, this class of bug often arrives after the team “solved” concurrency with a Redis SET key NX: mutual exclusion looks correct in happy-path tests and breaks only under GC pauses, clock skew, or partial outages.

This article explains what distributed locks actually guarantee, why time-to-live (TTL) leases are necessary but insufficient, and how fencing tokens—monotonic counters issued by a strongly consistent coordinator—let downstream storage reject work from a client that no longer holds the lock. The goal is not to recommend Redis or etcd by brand, but to give you a decision checklist for production systems where “only one writer” must remain true even when networks and processes lie.

What problem a distributed lock is trying to solve

In a single OS process, a mutex protects a critical section: at most one thread runs the guarded code at a time. Across machines, you no longer have shared memory; you simulate exclusivity by agreeing on a shared flag in an external store (or via a consensus protocol). Callers typically:

1. Acquire the lock (create or flip a key with rules that make concurrent acquires contend).
2. Perform work that must not overlap with other holders.
3. Release the lock (delete the key or compare-and-swap a value).

If step 3 never runs—crash, SIGKILL, zone failure—the flag would block everyone forever. Production systems therefore almost always use leases: the lock expires automatically after a TTL unless renewed. That shifts the failure mode from “stuck forever” to “eventually someone else may acquire”—which is healthier but introduces a subtler bug: two processes can still believe they own the critical section, one legitimately and one stale.

Leases, clocks, and the stale lock holder

A lease is a contract: “I may act as the owner until time T.” Implementations usually approximate T with:

• Wall-clock expiry stored in the lock service (e.g. Redis PEXPIRE), or
• Versioned records where the coordinator compares fence values or terms on renew.

A process can become stale—still running, still convinced it is the leader—when:

• It paused (long GC, debugger attachment, container freeze) and missed renewal while TTL elapsed; another worker acquired the lock.
• Clock skew between client and server made expiry appear later to the client than it was to the coordinator (less common on managed Redis with server-side TTL, but still relevant when clients compare timestamps locally).
• A network partition isolated the old holder from the lock store; it could not renew, lost the lease, and did not observe the loss until later.

If the critical section touches only the lock store (e.g. “only one counter increment per minute”), detecting overlap is easier. The painful cases are external side effects: writing to object storage, sending payments, or mutating a database where the second writer is not comparing against the same key Redis sees.

That gap—lock service says “new owner” while old owner’s code is still running—is where fencing enters.

Fencing tokens: a monotonic witness of legitimacy

A fencing token is a number (or comparable token) issued by the same coordination layer that grants the lock, with a simple rule: every successful lock acquisition gets a strictly greater token than the previous one, cluster-wide for that logical lock.

Typical sources:

• A Redis INCR on a dedicated key (single shard) or Lua script that sets the lock and increments a counter atomically.
• etcd or ZooKeeper: revision or version metadata on a key created with compare-and-swap semantics.
• DynamoDB conditional writes with an attribute that must increase.

The critical property is linearizable ordering: any storage that accepts writes must be able to compare the token against the last committed value and reject lower tokens as “late” or “stale.”

How this differs from “just use a UUID per lock”

A random request id proves idempotency of a single retried operation; it does not order different lock holders across time. Fencing tokens impose a total order on epochs of ownership. The database does not need to know about Redis key names—it only stores last_fencing_token for the protected resource and updates it only when the incoming token is greater.

Wiring fencing into a relational database

Suppose multiple workers append rows or update a singleton configuration row. Without fencing, the stale worker can overwrite the new owner’s write after reconnecting.

Pattern:

1. Acquire lease in Redis (or another coordinator); receive fencing token F (monotonic).
2. Start a transaction; SELECT last_token FROM resource_locks WHERE id = ? FOR UPDATE.
3. If F > last_token, apply mutation and UPDATE resource_locks SET last_token = F.
4. If F <= last_token, abort: you are stale; do not touch business data (optionally log and exit).

-- Table holds the latest committed fence for each logical resource.
CREATE TABLE resource_locks (
  resource_id   text PRIMARY KEY,
  last_token    bigint NOT NULL DEFAULT 0
);

-- Inside a transaction, after acquiring lock with token :f in application code:
UPDATE resource_locks
SET last_token = :f
WHERE resource_id = :id AND :f > last_token;

-- If rowcount = 0, either the resource is new (handle insert) or the token lost.

For insert-only workloads (audit logs), you might skip the update and instead insert with token F and enforce uniqueness or ordering in queries; the principle remains: downstream state must observe the token.

In Node.js services using pg, keep the fence check and the business write in one transaction so no other connection can interleave.

import type { PoolClient } from "pg";

export async function writeWithFence(
  client: PoolClient,
  resourceId: string,
  fencingToken: bigint,
  payload: unknown,
): Promise<"ok" | "stale"> {
  await client.query("BEGIN");
  try {
    const u = await client.query<{ last_token: string }>(
      `UPDATE resource_locks
       SET last_token = $2
       WHERE resource_id = $1 AND $2 > last_token
       RETURNING last_token`,
      [resourceId, fencingToken.toString()],
    );
    if (u.rowCount === 0) {
      const ins = await client.query(
        `INSERT INTO resource_locks (resource_id, last_token)
         VALUES ($1, $2)
         ON CONFLICT (resource_id) DO NOTHING`,
        [resourceId, fencingToken.toString()],
      );
      if (ins.rowCount === 0) {
        await client.query("ROLLBACK");
        return "stale";
      }
    }
    await client.query(
      `INSERT INTO audit_log (resource_id, fencing_token, payload)
       VALUES ($1, $2, $3::jsonb)`,
      [resourceId, fencingToken.toString(), JSON.stringify(payload)],
    );
    await client.query("COMMIT");
    return "ok";
  } catch (e) {
    await client.query("ROLLBACK");
    throw e;
  }
}

This sketch elides production details (migrations, backoff, metrics), but it shows the invariant: the database will not accept writes from a lower token once a higher one committed.

Redis, Redlock, and what not to hand-wave

Redis single-instance locks with SET key value NX PX ttl are widely used because they are fast and simple. Redlock (multiple independent Redis nodes, quorum-based acquisition) tries to survive node failures. Whether Redlock is appropriate is debated; the important engineering questions are:

• Can you tolerate failure modes where two clients briefly believe they hold the lock? If that is unacceptable for external side effects, fencing downstream is mandatory regardless of Redlock vs single-node.
• Is your Redis deployment linearizable enough for your fencing counter? A single primary with synchronous replication to a standby is a common compromise; async replication between primaries can reintroduce split-brain semantics at the storage layer.

For etcd / Consul / ZooKeeper, lock recipes often expose session semantics and monotonic indexes—use those indexes as fencing tokens.

Trade-offs and operational costs

Approach	Pros	Cons
TTL lease only	Simple; self-heals after crashes	Stale workers can corrupt external state
Lease + fencing to DB	Strong safety for writes	Extra round trip; schema must carry last_token
Consensus leader election	Clear single writer	Heavier ops; still fence side effects if lease can lapse

Renewal storms: if work duration routinely approaches TTL, you will churn renewals; instrument time held and renew count. Long TTL reduces false staleness but delays failover when the active process truly died.

Practical example: lock, fence, and guarded blob write

Imagine exclusive writers to reports/{tenant}/{id}.json in object storage. Versioning buckets help, but you still want to abort stale processes before they PUT.

1. Acquire Redis lock lock:report:{tenant}:{id} with random lockVal, TTL 30s.
2. Increment global fence fence:report:{tenant}:{id} via INCR (or include INCR inside Lua with SET NX).
3. Write to Postgres row for that report: UPDATE ... WHERE fencing_token < $1 (pattern above).
4. If DB accepts, upload blob; if DB rejects, skip upload—another epoch owns the resource.
5. Release lock with Lua compare-and-delete using lockVal to avoid deleting a new owner’s key.

The object store never sees a write from a losing fence value because the application stopped at the database gate—exactly the “defense in depth” that shows up when hardening pipelines for clients who need predictable, production-ready behavior.

Common mistakes and pitfalls

• Assuming the lock is a mutex like pthread: it is a probabilistic lease; design for expiry and duplication.
• Auto-renewing forever: a wedged process can hold liveness indefinitely; cap renewals or bound total runtime.
• Fencing token checked only in application memory: the stale process still has an old token in RAM—authoritative check must be on the durable store receiving the side effect.
• Using wall-clock “started at” instead of a coordinator counter: NTP does not give you global ordering of events across nodes.
• Same lock key for unrelated resources: token namespaces must not collide; scope keys per resource id or shard.
• Ignoring exactly-once vs at-least-once: fencing does not replace idempotent writes; it prevents stale epochs from winning, not duplicate delivery of the same epoch.

Conclusion

Distributed locks solve liveness and mostly exclusive access; leases fix crash safety at the cost of stale holders. Fencing tokens bridge the coordinator and your durable systems so a late writer cannot clobber a legitimate one—provided every protected resource persists and compares the token in the same transaction as the mutation.

When reviewing architectures—whether internal platforms or consulting deliverables—the question is not “Redis or etcd?” but “Where is the single authoritative ordering for writes that must never overlap?” Answer that with leases plus monotonic witnesses at the storage boundary, and double-execution becomes an observable stale outcome instead of silent data loss.