RAG freshness pipelines: incremental indexing, document versions, and stale retrieval detection

Your RAG assistant answers a compliance question with a paragraph from a policy document that was superseded last Tuesday. Retrieval scored the chunk highly; the model cited it confidently. Support escalations follow, and the postmortem reveals the obvious failure: the vector index still contained embeddings from the old version. Ingestion ran once at launch; updates went to Postgres or a CMS, but nobody wired the index to change events. The retrieval layer did exactly what it was built to do—return the nearest vectors—not what the business needed: return evidence that is still true.

Freshness is not a footnote in RAG design. It is the difference between a knowledge base that tracks reality and one that fossilizes the day you shipped. This article covers how to build incremental indexing pipelines, model document versions explicitly, and add runtime guards so stale chunks are detected before they reach the generator. The patterns show up constantly when hardening assistants for teams that cannot afford "mostly up to date" answers in regulated or fast-moving domains.

Why batch re-indexing stops scaling

The first production RAG system many teams ship looks like this:

1. Export all documents.
2. Chunk, embed, upsert into a vector store.
3. Schedule a nightly or weekly full rebuild "just in case."

That works until it does not:

• Latency to truth — A pricing change at 10:00 AM should not wait until Sunday's batch job. Users and auditors measure freshness in minutes, not cron intervals.
• Cost at scale — Re-embedding a million unchanged chunks because one paragraph changed burns API budget and extends maintenance windows.
• Blast radius — Full rebuilds often require blue/green index swaps. A bad deploy can poison the entire corpus at once.
• Deletion gaps — Documents removed from the source of truth often linger in the index unless you explicitly tombstone them. Vector search has no concept of "this file was deleted."

Incremental pipelines treat each document as a versioned entity with a deterministic identity. Changes propagate as small, idempotent jobs—not monolithic rewrites.

Document identity: stable keys and content versions

Before incremental indexing, define what "a document" means in your system.

Stable doc_id

Use an identifier that survives renames, CMS slug changes, and file moves:

• Primary key from your database (documents.id).
• A content-addressable hash only if immutability is guaranteed (rare for editable docs).

Never key retrieval metadata on file paths alone unless paths are immutable contracts.

Content version (content_version)

Every indexable document needs a monotonic or hash-based version stamp:

Strategy	When to use	Example
Row updated_at + content_hash	OLTP source of truth in Postgres	updated_at for ordering; SHA-256 of normalized body for equality
CMS revision id	Headless CMS with explicit revisions	revision: 4821
Git commit SHA	Docs-as-code in a repository	abc123f on main

Store both on every chunk:

export type ChunkMetadata = {
  docId: string;
  contentVersion: string; // hash or revision id
  corpusVersion: string;  // global index generation (see below)
  locale: string;
  title: string;
  sourceUrl?: string;
  effectiveFrom?: string; // ISO date for policy docs
};

Why contentVersion per chunk: When a document changes, old chunks must be removed or superseded atomically relative to new chunks. Comparing versions at query time lets you reject stale hits even if deletion lagged.

Corpus version (corpus_version)

A global counter or timestamp incremented on any corpus-affecting change—embedding model swap, chunking policy change, or bulk purge. Attach it to:

• Chunk metadata in the vector store.
• Session context in your LLM backend (see context engineering).

If a user's session started under corpus_version: 17 and you deploy 18 with a new chunking strategy, you can force re-retrieval instead of mixing evidence from incompatible indexing runs.

Incremental indexing: the change-detection loop

The core loop is simple; the edge cases are not.

source change event
  → load document snapshot
  → compute content_hash
  → if hash unchanged: ACK and exit
  → else: delete old chunks for doc_id
  → chunk → embed → upsert new chunks
  → record index_state(doc_id, content_version, indexed_at)

Content hashing: normalize before you hash

Hash the canonical normalized body, not raw bytes:

• Normalize line endings (\r\n → \n).
• Collapse repeated whitespace where it does not affect meaning.
• Strip boilerplate blocks you never want indexed (navigation chrome, "last updated" footers that change without semantic change—optional and product-specific).

import { createHash } from "node:crypto";

export function normalizeForIndexing(body: string): string {
  return body.replace(/\r\n/g, "\n").replace(/\n{3,}/g, "\n\n").trim();
}

export function contentHash(body: string): string {
  return createHash("sha256").update(normalizeForIndexing(body), "utf8").digest("hex");
}

Identical hashes short-circuit work—critical when webhooks fire on trivial metadata updates.

Tombstone deletes: vectors must disappear

Vector stores rarely offer transactional "replace document." You implement delete-then-insert or soft-delete flags:

1. Hard delete — Remove all chunk ids matching doc_id before upserting replacements. Prefer idempotent delete APIs (delete by filter: doc_id = X).
2. Tombstone metadata — Mark deleted: true on chunks; filter at query time. Useful when deletes are eventually consistent but risky if filters are forgotten.

In consulting work on multi-tenant knowledge bases, the most common freshness bug is orphan chunks after partial deletes. Automate delete verification: after indexing, assert count(chunks where doc_id = X) == expected.

Idempotent workers

Ingestion jobs retry. Keys must be stable:

• Chunk id: ${docId}#${contentVersion}#${part} or ${docId}#${part} with version only in metadata (choose one scheme and never mix).
• Job idempotency: index_job:${docId}:${contentVersion} stored in Postgres with ON CONFLICT DO NOTHING.

Driving ingestion from change events

How you learn about changes determines freshness SLAs.

Webhooks and outbox (application-level)

CMS webhooks (document.published) or your API emitting events after commit work when you control the write path. Pair with a transactional outbox (see transactional outbox) so index jobs are not lost if the worker crashes after the DB commit.

CDC (database-level)

When documents live in Postgres and many services write to them, change data capture is more reliable than hoping every code path emits a webhook. Consume INSERT/UPDATE/DELETE on documents and enqueue index jobs. This pairs naturally with CDC-driven cache invalidation patterns—the same change stream can invalidate caches and refresh search indexes.

Polling fallback

Poll updated_at > last_cursor for sources without events. Simpler, higher lag, but a acceptable safety net when webhooks or CDC miss edge cases.

Mechanism	Typical lag	Complexity	Best for
Webhook + outbox	Seconds	Medium	CMS, controlled APIs
CDC	Sub-minute	Higher	Postgres as source of truth
Scheduled poll	Minutes–hours	Low	Legacy stores, backfill

Teams I work with on production assistants usually run CDC or outbox as primary and polling as reconciliation (nightly diff: source ids vs index_state table).

Query-time freshness gates

Indexing lag is inevitable—deploys, queue backlog, embedding rate limits. Query-time checks are your last line of defense.

Metadata filters

Constrain vector search with filters the store supports:

export type RetrievalFilter = {
  tenantId: string;
  locale: string;
  corpusVersion: string; // exact match or gte minimum
  deleted?: false;
  effectiveBefore?: string; // optional: policy effective dates
};

If your store cannot filter efficiently, maintain a sidecar table of valid chunk ids per corpus version and post-filter results—a latency trade-off, but correct.

Version reconciliation after retrieval

For each candidate chunk, verify against authoritative state:

type IndexState = {
  docId: string;
  contentVersion: string;
  indexedAt: string;
};

export async function filterStaleChunks(
  hits: Array<{ chunkId: string; metadata: ChunkMetadata; score: number }>,
  authoritative: Map<string, IndexState>,
): Promise<typeof hits> {
  return hits.filter((hit) => {
    const current = authoritative.get(hit.metadata.docId);
    if (!current) return false; // doc deleted or unknown
    return current.contentVersion === hit.metadata.contentVersion;
  });
}

Load authoritative versions from a fast cache (Redis) keyed by doc_id, warmed from Postgres. A single batch MGET per query is cheap compared to a wrong LLM completion.

Stale retrieval metrics

Instrument what you filter out:

• rag_stale_chunks_dropped_total{reason="version_mismatch"}
• rag_index_lag_seconds — now() - indexed_at for documents in the retrieval set
• rag_corpus_version_skew — sessions running older corpus versions

Alert on lag percentiles, not just averages. A p99 lag of four hours with a p50 of thirty seconds still produces memorable incidents.

Embedding and chunking migrations

Changing the embedding model or chunking policy is a new corpus, not an in-place patch.

1. Bump corpus_version.
2. Run a backfill job that re-indexes all documents under the new policy (can be throttled).
3. Dual-read during cutover: query both indexes or prefer new with fallback—measure quality before dropping old.
4. Delete the old index namespace when eval passes.

Skipping this ceremony produces subtle bugs: half the chunks embedded with text-embedding-3-small and half with a legacy model will rank inconsistently even if document text is current.

Practical example: Postgres-backed index state with incremental worker

The following sketches a minimal but production-shaped pipeline: source rows in Postgres, index state tracking, content-hash short circuit, and delete-before-upsert.

import { createHash } from "node:crypto";
import type { Pool } from "pg";

// --- schema (SQL) ---
// CREATE TABLE documents (
//   id          UUID PRIMARY KEY,
//   tenant_id   UUID NOT NULL,
//   locale      TEXT NOT NULL DEFAULT 'en',
//   title       TEXT NOT NULL,
//   body        TEXT NOT NULL,
//   updated_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
//   deleted_at  TIMESTAMPTZ
// );
//
// CREATE TABLE rag_index_state (
//   doc_id           UUID PRIMARY KEY REFERENCES documents(id),
//   content_version  TEXT NOT NULL,
//   chunk_count      INT NOT NULL,
//   corpus_version   TEXT NOT NULL,
//   indexed_at       TIMESTAMPTZ NOT NULL DEFAULT now()
// );

export type VectorStore = {
  deleteByDocId: (docId: string) => Promise<void>;
  upsertChunks: (chunks: Array<{
    id: string;
    embedding: number[];
    metadata: Record<string, string | number | boolean>;
  }>) => Promise<void>;
};

export type EmbedFn = (texts: string[]) => Promise<number[][]>;

const CORPUS_VERSION = "2026-08-02-v1";

function normalize(body: string): string {
  return body.replace(/\r\n/g, "\n").trim();
}

function hash(body: string): string {
  return createHash("sha256").update(normalize(body), "utf8").digest("hex");
}

function chunkText(docId: string, body: string, maxChars = 1200, overlap = 200) {
  const chunks: Array<{ id: string; text: string; part: number }> = [];
  let i = 0;
  let part = 0;
  while (i < body.length) {
    const end = Math.min(i + maxChars, body.length);
    const text = body.slice(i, end).trim();
    if (text) chunks.push({ id: `${docId}#${part}`, text, part });
    if (end >= body.length) break;
    i = end - overlap;
    part += 1;
  }
  return chunks;
}

export async function indexDocument(
  pool: Pool,
  vectorStore: VectorStore,
  embed: EmbedFn,
  docId: string,
): Promise<"skipped" | "indexed" | "deleted"> {
  const { rows } = await pool.query<{
    id: string;
    tenant_id: string;
    locale: string;
    title: string;
    body: string;
    deleted_at: Date | null;
  }>(
    `SELECT id, tenant_id, locale, title, body, deleted_at
     FROM documents WHERE id = $1`,
    [docId],
  );
  const doc = rows[0];
  if (!doc) return "deleted";

  if (doc.deleted_at) {
    await vectorStore.deleteByDocId(docId);
    await pool.query(`DELETE FROM rag_index_state WHERE doc_id = $1`, [docId]);
    return "deleted";
  }

  const contentVersion = hash(doc.body);

  const { rows: stateRows } = await pool.query<{ content_version: string }>(
    `SELECT content_version FROM rag_index_state WHERE doc_id = $1`,
    [docId],
  );
  if (stateRows[0]?.content_version === contentVersion) {
    return "skipped";
  }

  const parts = chunkText(docId, doc.body);
  const embeddings = await embed(parts.map((p) => p.text));

  // Delete old vectors before upsert — prevents orphan chunks on shrink/rechunk
  await vectorStore.deleteByDocId(docId);

  await vectorStore.upsertChunks(
    parts.map((part, idx) => ({
      id: part.id,
      embedding: embeddings[idx]!,
      metadata: {
        docId,
        contentVersion,
        corpusVersion: CORPUS_VERSION,
        tenantId: doc.tenant_id,
        locale: doc.locale,
        title: doc.title,
        part: part.part,
      },
    })),
  );

  await pool.query(
    `INSERT INTO rag_index_state (doc_id, content_version, chunk_count, corpus_version)
     VALUES ($1, $2, $3, $4)
     ON CONFLICT (doc_id) DO UPDATE SET
       content_version = EXCLUDED.content_version,
       chunk_count = EXCLUDED.chunk_count,
       corpus_version = EXCLUDED.corpus_version,
       indexed_at = now()`,
    [docId, contentVersion, parts.length, CORPUS_VERSION],
  );

  return "indexed";
}

// Worker entry: called from queue consumer or CDC handler
export async function handleDocumentChange(
  pool: Pool,
  vectorStore: VectorStore,
  embed: EmbedFn,
  docId: string,
): Promise<void> {
  const outcome = await indexDocument(pool, vectorStore, embed, docId);
  console.info({ docId, outcome, corpusVersion: CORPUS_VERSION });
}

Hook handleDocumentChange to your queue (SQS, BullMQ, etc.) or CDC consumer. A nightly reconciliation job compares documents.id to rag_index_state.doc_id and enqueues missing or outdated rows.

Retrieval with version check

export async function retrieveWithFreshness(
  pool: Pool,
  vectorStore: { search: (q: number[], filter: Record<string, string>, k: number) => Promise<Array<{ metadata: ChunkMetadata; score: number }>> },
  embedQuery: (q: string) => Promise<number[]>,
  query: string,
  tenantId: string,
  locale: string,
  k = 8,
) {
  const queryVec = await embedQuery(query);
  const hits = await vectorStore.search(queryVec, {
    tenantId,
    locale,
    corpusVersion: CORPUS_VERSION,
  }, k * 2); // over-fetch; filter stale

  const docIds = [...new Set(hits.map((h) => h.metadata.docId))];
  const { rows } = await pool.query<{ doc_id: string; content_version: string }>(
    `SELECT doc_id, content_version FROM rag_index_state WHERE doc_id = ANY($1::uuid[])`,
    [docIds],
  );
  const authoritative = new Map(rows.map((r) => [r.doc_id, r.content_version]));

  const fresh = hits.filter((h) => authoritative.get(h.metadata.docId) === h.metadata.contentVersion);
  return fresh.slice(0, k);
}

This pattern costs one small SQL round-trip per query and eliminates an entire class of "ghost document" answers.

Common mistakes and pitfalls

Assuming the vector store is the source of truth

The index is a derived projection. Authoritative content lives in your database, CMS, or object store. If they disagree, the source wins—always.

Indexing on updated_at without content hash

Metadata-only updates (title tweak, tag change) should not trigger re-embedding. Conversely, some systems update updated_at on unrelated joins—hash the body you actually embed.

Forgetting deletes

Soft-deleted or archived documents must tombstone in the index. A compliance deletion in Postgres that does not remove vectors is a data retention incident.

No corpus version on breaking changes

Swapping embedding models without bumping corpus_version mixes incompatible vectors in one namespace. Similarity scores become meaningless.

Over-relying on TTL

TTL eviction in a cache layer is not a freshness strategy for knowledge bases. Documents do not "expire" uniformly; event-driven invalidation plus version gates are the durable approach.

Skipping reconciliation

Queues lose messages; CDC lag spikes; webhooks duplicate. A periodic diff job (source vs rag_index_state) catches drift before users do. Treat it like database backup verification—not optional hygiene.

Conclusion

RAG freshness is an engineering pipeline problem, not a prompt tweak. Stable document identity, content hashing, incremental workers, explicit deletes, and query-time version checks form a defense in depth: fast propagation when things work, correct behavior when they lag.

The teams that ship trustworthy assistants treat the index like a materialized view—versioned, observable, and reconciled against source data. If you are designing retrieval for a product where wrong answers have real cost, investing in freshness mechanics early is cheaper than explaining to customers why last week's policy still governs today's chat.

When I help teams move from demo RAG to production systems, we usually start with chunking and eval—and then spend the next sprint wiring change propagation because that is where silent failures live. Get indexing tied to truth, measure lag, and filter stale hits at query time. The model can only be as current as the evidence you allow it to see.