Distributed scheduled jobs: leases, idempotency, and why clocks lie

You enable horizontal scaling for your API and notice duplicate nightly emails, or conversely, jobs that never run after a deploy because the old pod still held a vague “lock.” Neither outcome shows up in unit tests: your handler is correct in isolation; the environment is not. Scheduled work in distributed systems fails for mundane reasons—two processes think they are the leader, one process dies mid-run, or wall-clock schedules diverge across machines—long before you need exotic consensus algorithms.

This article is about the control plane for recurring tasks: how to ensure at-most-once side effects where that matters, at-least-once delivery with idempotent outcomes where it does not, and how to reason about time when cron expressions meet virtual machines and containers. The patterns below show up in production APIs and data pipelines; they are the same ones teams adopt when moving from “a script on a server” to scalable, production-ready job execution.

Why “run this every hour on one instance” is underspecified

A classic anti-pattern is an in-process timer inside a web server: setInterval (or framework hooks) fires on every replica. If the job sends notifications or mutates shared state, you multiply effects by replica count. The opposite mistake is assuming only one process exists because traffic is low today—Kubernetes, autoscaling groups, and blue/green deploys all introduce temporary duplication even when steady-state size is one.

You need an explicit contract:

1. Exactly one active executor per schedule slice (leader / tenant / shard), or a defined rule for sharding work so parallelism is safe.
2. Survivability: if the leader crashes, another instance may take over without waiting for human intervention.
3. Semantics for overlap: if the previous run is still executing when the next tick arrives, do you skip, queue, or allow concurrent runs (and under what limits)?

Without that contract, “it worked on my machine” is literally true—your laptop had one process.

Leader election via leases: short TTL, explicit renewal

Leases are time-bounded grants stored in a coordination store: Redis (SET key value NX PX ttl), relational rows (UPDATE ... WHERE lease_owner = $self AND expires_at < now()), DynamoDB conditional writes, etc. The holder renews the lease while healthy; if it stops renewing, the key expires and another worker can acquire it.

Why leases beat “check a flag in Postgres once”:

• Failure detection is asynchronous in distributed systems. A process can be stuck or partitioned without releasing a fat binary lock. TTL converts “I forgot to unlock” into “someone else may proceed after bounded delay.”
• Renewal ties correctness to liveness: if you cannot renew, you should stop assuming leadership—finish gracefully or abort, depending on the task.

Fencing: stale leaders must not write

The uncomfortable detail is lease expiration versus wall time. Your worker believes it is leader until 12:00:00; the store believes the lease ended at 11:59:45 because renewal RPCs failed during a network blip. For side-effecting jobs (charges, emails, external APIs), combine the lease with a fencing token: a monotonic generation stored with the lease. Every downstream write carries the current token; storage rejects writes from older tokens even if the old leader still runs—this is the same family of ideas used in more formal distributed locking discussions.

For read-only or idempotent-replay jobs, strict fencing is sometimes relaxed—but you must prove replay cannot corrupt state.

Idempotency: make “at-least-once” safe

Leases reduce duplicate execution; they do not eliminate it. Renewals race with expiry; deploys overlap runs; manual retries happen. The robust baseline is idempotent handlers: executing the same logical job twice produces the same observable outcome as once (or fails safely without corrupting data).

Practical patterns:

• Deterministic keys in external systems (payment idempotency keys, idempotent REST PUTs).
• Natural keys in your DB (UNIQUE constraint on (job_name, period_start)).
• Claim rows in a job_runs table: INSERT ... ON CONFLICT DO NOTHING RETURNING id—only the insert winner performs work.

Store enough metadata to detect partial completion: a row in processing state may require a compensating check against the external system before retry.

In consulting-style reviews of backend job runners, the dominant failure mode is not cryptographic locking—it is handlers that are only accidentally idempotent (e.g., “delete old rows” is safe; “increment counter” is not without a guard).

Scheduling semantics: cron, calendars, and skew

cron uses the scheduler’s local timezone and clock. Three subtleties bite teams:

1. Skipped ticks — If the machine sleeps or the process pauses longer than one interval, classic cron does not “catch up” every missed invocation; many distributed schedulers behave similarly unless configured otherwise.
2. Clock skew — NTP keeps clocks close but not identical. Sub-second ordering across machines is unreliable; design keys and locks using coarse windows (minute or hour buckets), not “exactly at this millisecond everywhere.”
3. DST and time zones — Daily jobs around 01:30 local time may run twice or not at all on DST transitions. Prefer UTC for server-side schedules and explicit tenant time zones only at presentation boundaries.

For wall-clock–sensitive jobs (billing periods, regulatory reporting), anchor on business calendars stored in the database rather than implicit OS timezone rules.

Where to put the scheduler

You have three common architectures:

Approach	Pros	Cons
External orchestrator (cloud scheduler, Airflow, Nomad periodic)	Central visibility, mature retries	Another system to operate; network path to your workers
DB-backed queue + lease consumer	Transactional claims; great with Postgres	DB load; careful indexing and pruning
Leader-elected in-app timer	Low moving parts	Easy to get wrong under rolling deploys

Hybrid stacks often use an external scheduler to enqueue a message (“run nightly aggregation for tenant batch A”) and workers that compete idempotently—avoiding the need for global leader election for every cron line while keeping execution horizontally scalable.

Practical example: Postgres lease + idempotent job claim

The following pattern fits many Node.js services that already use PostgreSQL. It is illustrative, not a full library: production code would add metrics, structured logging, jittered renewal, and backoff.

Schema:

CREATE TABLE job_schedules (
  name TEXT PRIMARY KEY,
  lease_owner TEXT NOT NULL,
  lease_expires_at TIMESTAMPTZ NOT NULL
);

CREATE TABLE job_invocations (
  job_name TEXT NOT NULL,
  period_start TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (job_name, period_start)
);

Acquire or renew lease (once per process loop):

import pg from "pg";

type LeaseResult = { leader: true } | { leader: false };

export async function tryAcquireOrRenewLease(
  pool: pg.Pool,
  jobName: string,
  ownerId: string,
  ttlMs: number,
): Promise<LeaseResult> {
  const client = await pool.connect();
  try {
    await client.query("BEGIN");

    const updated = await client.query(
      `UPDATE job_schedules
         SET lease_owner = $2,
             lease_expires_at = NOW() + ($3::bigint * INTERVAL '1 millisecond')
       WHERE name = $1
         AND (lease_expires_at < NOW() OR lease_owner = $2)`,
      [jobName, ownerId, ttlMs],
    );

    if (updated.rowCount === 0) {
      const inserted = await client.query(
        `INSERT INTO job_schedules (name, lease_owner, lease_expires_at)
         VALUES ($1, $2, NOW() + ($3::bigint * INTERVAL '1 millisecond'))
         ON CONFLICT (name) DO NOTHING`,
        [jobName, ownerId, ttlMs],
      );

      if (inserted.rowCount === 0) {
        await client.query("ROLLBACK");
        return { leader: false };
      }
    }

    await client.query("COMMIT");
    return { leader: true };
  } catch (e) {
    await client.query("ROLLBACK");
    throw e;
  } finally {
    client.release();
  }
}

Idempotent period execution:

export async function runIfClaimed(
  pool: pg.Pool,
  jobName: string,
  periodStart: Date,
  work: () => Promise<void>,
): Promise<boolean> {
  const res = await pool.query(
    `INSERT INTO job_invocations (job_name, period_start)
     VALUES ($1, $2)
     ON CONFLICT DO NOTHING`,
    [jobName, periodStart.toISOString()],
  );

  if (res.rowCount === 0) return false;

  await work();
  return true;
}

Wire these together in a loop: renew lease frequently; only the leader calls runIfClaimed for each schedule bucket. For side-effecting calls where stale leaders could still complete after lease loss, add a monotonic fencing token (see above) and validate it at the downstream store—this snippet omits that detail to keep the transaction shape readable.

Common mistakes and pitfalls

• Long TTL without renewal — Leadership stalls until expiry after a crash; short TTL with aggressive renewal reduces recovery time but increases store load. Tune intentionally.
• Non-idempotent work behind a lease — Leases expire; duplicates still happen. Treat leases as optimization, idempotency as correctness.
• Using the OS clock as source of truth for financial periods — Prefer domain-defined periods persisted in your database.
• Ignoring overlap — If work can exceed the interval, define whether you allow concurrency, skip, or use a worker pool with max parallelism per job name.
• Silent failure in renewal loops — Monitor renewal errors; losing leadership without noticing turns distributed scheduling into undefined behavior.

Conclusion

Distributed scheduled jobs are less about picking Redis versus Postgres and more about clear semantics: who may run, for which time bucket, what happens on failure, and how you prove work was not double-applied. Leases bound recovery time; fencing protects shared resources from stale leaders; idempotency makes “at-least-once” execution honest under retries and deploys. Teams building scalable, production-ready backends benefit from writing these rules down before the second replica ships—future you, and anyone on call when cron misbehaves at 03:00, will have something concrete to debug against.

If you are evaluating job runners or reviewing architecture for a growing platform, the About page summarizes relevant experience; for a direct conversation, use Contact.