Cruxly

Cruxly › AI engineering › Retrieval and RAG

Ingestion Pipelines: Sync, Freshness, Permissions and Deletion

hashes instead of re-embeds, a leak suite instead of hope, and a delete that reaches five stores

Open the interactive lesson

Prompt: "Design the ingestion for 20 million documents from Confluence, a file share and a ticket system, with per-user permissions. A document was deleted in Confluence an hour ago and still shows up in answers: how does it leave the index, and in what time? How do you keep the index fresh without re-embedding everything, and how do you test that permissions are enforced?"

40 minutes first. Price a daily incremental sync against a full re-embed, write the freshness budget as numbers, and list every store a deleted document lives in. The trap is describing the pipeline for the happy path of new documents and forgetting edits, permission changes and deletes, which is where the incidents are.

Connectors and parsing: the part that decides whether anything works

Most of the engineering in a production RAG system sits before the first embedding call, and most incidents begin there. Sources are SaaS APIs (wikis, ticket systems, drives) and file shares, each with its own pagination, rate limits, auth expiry, and ways of failing halfway through.

Rate limits set your schedule. Backfilling 2 million documents at an allowed 10 requests per second takes 2,000,000 / 10 = 200,000 s, which is 55.6 hours, or 2.3 days of wall-clock time if nothing fails. Plan the first sync as a multi-day job with checkpoints, not a script you run on Friday. Throttle below the limit (the same API probably serves your users too), and respect Retry-After.

Design the pipeline in decoupled stages with a raw store.

@@ 1 | Fetch into a raw store The connector writes the original bytes (and source metadata: ID, URL, modified time, ACL) to object storage, keyed by source ID and version. Parsing bugs can then be fixed and replayed without hitting the source API again.

@@ 2 | Parse, then gate Layout-aware parsing and quality gates (section 6). Output is structured text with a title path; a failed parse goes to a dead-letter queue with a reason, not to the index.

@@ 3 | Chunk and embed Structure-aware chunking and embedding, as in Chunking and Document Parsing. Record chunker_version and model_id on every chunk, as Choosing, Evaluating and Fine-Tuning an Embedding Model requires.

@@ 4 | Upsert Write vectors, keyword postings, and chunk metadata idempotently (section 2), then mark the document's sync state as done.

Partial failure is the normal case. Across two million documents some will be corrupt, password-protected, too large, or return a 500 for a day. Track per-document state (fetched, parsed, embedded, indexed, failed:reason) so that you can answer "how many documents are not searchable and why?". Without that table, the answer is "I don't know", and silent partial indexes are how a bot confidently says a policy does not exist.

Know the cost shape. Embedding is cheap and parsing is not: 12 chunks of 400 tokens is 4,800 tokens per document, which at an assumed $0.02 per million tokens is about $0.0001. OCR, layout models and LLM-written chunk context can cost far more per document, so measure the whole pipeline, not just the embedding line.

Incremental sync: hashes, change feeds, tombstones and idempotence

You re-embed what changed, and you find out what changed with hashes and a change feed, not by guessing. The numbers: 2M documents, 12 chunks each, 400 tokens per chunk is 2,000,000 x 12 x 400 = 9.6 billion tokens. A full re-embed at an assumed $0.02 per million tokens is $192. If 3 percent of documents change per day, that is 60,000 documents and 60,000 x 12 x 400 = 288 million tokens, which is $5.76 a day, 33 times cheaper. (At 20M documents multiply both by ten.) The dollar figure is small either way; the real savings are time, load on the source system and index churn.

Detecting change.

Chunk boundaries decide how much a small edit costs. Insert one sentence at the start of a 6,000-token document chunked as fixed 512/64 windows and every window shifts: all 14 chunks get new hashes and are re-embedded. With structure-aware chunking, an edit inside one section changes that section's chunk only: 1 chunk. This is a quiet argument for the chunking choices in Chunking and Document Parsing: Structure-Aware Splits, Parent-Child, Late Chunking and Contextual Retrieval.

Idempotence gives you exactly-once effect. A queue delivers at least once, so a stage may run twice for the same document. Make that harmless: key every write by (doc_id, chunk_hash) or (doc_id, chunk_index, doc_version) and use upsert, never insert. Then retries, replays and duplicate webhooks produce the same final state, which is what "exactly-once effect" means: messages may repeat, but the result happens once. Order matters too: apply a document's events by version number, so a late "version 4" cannot overwrite "version 5".

Replacing a document is a transaction in spirit. Writing the new chunks, then deleting the old ones whose hashes are gone, in the wrong order or with a crash in between, leaves either duplicates (old and new both searchable) or a gap (neither). Write the new chunks with the new version number, flip a current_version pointer (or filter by it at query time), then garbage-collect old versions.

Freshness: a number you measure, not a word you promise

A freshness SLO is the time from "edited in the source" to "returned by a query", and the only honest way to know it is to measure it end to end. Write down the stages and their costs. An example pipeline with assumed timings: webhook arrival 5 s, queue wait 30 s, parse 20 s, embedding 2 s, upsert plus index refresh 1 s. Total: 58 s. That is the push path.

A pull path adds its polling interval. If you poll a source every 15 minutes, an edit waits on average 7.5 minutes for the next poll and up to 15 in the worst case, before the 58 seconds start. So "near real time" is false for polled sources; say "within 16 minutes" and be exact. Tier your sources: a ticket system that agents read live needs minutes; a wiki hours; an archive daily. A single SLO for everything either overspends on the archive or underserves the tickets.

Measure with a canary. Run a synthetic document in each source that a scheduler edits every 5 minutes, writing a unique token (CANARY-7f3a-0412) into it. A probe then queries for that token every 30 seconds and records the time it first appears. The delay between edit and first hit, aggregated as p50 and p95, is your real freshness, including queueing and index refresh. Alert when p95 exceeds the SLO. When it fails, the stage timings tell you where: queue depth rising points to the embedding stage, not the source.

Know the other latencies. A new document is searchable when its vectors are in the index and visible to queries. Many search engines refresh on an interval (one second is common, and bulk loads may raise it); HNSW indexes may batch writes. Answer caches add yet another delay: a cached answer built from the old document stays until it is invalidated. Link cache keys to the source document versions (Prompt Caching: Paying for the Same Tokens Once covers prefix caches; answer-level and semantic caches need the same discipline).

Backpressure is a freshness failure mode. A bulk import of 500,000 documents floods the queue, and the single edited policy page waits behind it. Use separate priority lanes: interactive edits and permission changes ahead of bulk backfills, with the backfill throttled so it never starves the live path.

Permissions: copy, fast-path, enforce in the database, then prove it

Permissions are data that must be ingested, kept fresh and enforced at query time, and each of those three steps has its own failure. The principle was set in Multi-Tenant, Multilingual RAG at Scale: enforce in the database, not in application code. Here is how the data gets there.

Copy the ACL into every chunk's metadata. At ingestion, read the source document's access list (users, groups, "anyone with the link") and store a normalised allowed_principals array on each chunk row, alongside doc_id and version. At query time the filter is "the user's principals overlap allowed_principals", executed by the database before results leave it. Decide whether to expand groups at index time (fast queries, but a membership change touches many chunks) or at query time (a small, always-fresh lookup of the user's groups). Query-time expansion of the user side is usually the better default, because one user's groups change far less often than a document's chunks.

A permission change is not a content change, so give it its own fast path. Revoking access to a salary document must not wait for a re-embed: it updates only the metadata of that document's existing chunks. Give it a tighter SLO than content freshness (seconds to minutes, not hours), a high-priority lane, and no embedding call, so cost is near zero. The failure to avoid: the content hash is unchanged, so the sync says "nothing to do", and the revoked permission is never copied. Hash the ACL separately and sync it separately.

Failure modes to name.

Prove it with a leak suite. Create planted canary documents, each readable by exactly one role and containing a unique token. Take 6 roles with 5 canaries each: 30 documents. Query each canary's token as each of the 6 roles: 6 x 30 = 180 queries. Exactly 30 should return the canary (the owning role), and 150 must return zero hits; any hit is a failure that blocks the release. Add the temporal tests: revoke a grant, then assert the canary disappears within the permission SLO; add a user to a group and assert it appears. Run the suite in CI and continuously in production against the live index, since the data is the thing that drifts. The Red-Teaming and Compliance as Code lesson treats this kind of check as a release gate.

Deletion: five stores, two speeds, one proof

A document is not deleted when it leaves the source; it is deleted when every store that held a copy of it, or something derived from it, has been purged and you can show that. For one deleted Confluence page, list the places it lives:

  1. Vector index rows (every chunk, including old versions and parent-child text).
  2. Keyword index postings (BM25), and any parent-text or chunk store.
  3. Raw and parse caches: the stored original bytes and extracted text in object storage.
  4. Derived caches: cached answers, semantic caches, contextual-retrieval text, reranker caches, and any summaries or graph entries built from it.
  5. Logs, traces, evaluation sets and the audit entry: prompts and retrieved chunks that were logged, and golden-set rows that copied its text.

(Backups are the sixth, and hard: state your policy, for example "backups expire on a 35-day cycle; restored data re-runs the deletion ledger".)

Two speeds. First a soft-delete that hides the document at query time within seconds (flag it deleted in the state table and metadata, and filter on it), so a deleted page cannot be answered from while the purge runs. Then a hard purge across all stores within a stated window, say 24 hours. Hard purge on a graph index is not instant: HNSW deletion usually leaves tombstones in the graph, as ANN Indexes in Depth: HNSW, IVF-PQ, DiskANN and the Recall/Latency/Memory Triangle explains, so the vector bytes can remain until compaction. If you promise erasure, schedule compaction inside the promise or use an index that removes data physically.

Detection is the hard part. A webhook may announce the delete; a polled source may say nothing. The nightly reconciliation (IDs at the source versus your state table) is what turns "page is gone" into "tombstone for doc 4821". Answer the interview question exactly: "It stops appearing in answers within seconds of detection. Detection takes seconds if the source sends a webhook and up to the next nightly reconciliation (24 hours) if not; and it is gone from all five stores within the purge window."

Prove it. For each store write a verification query ("select count from chunks where doc_id = 4821 = 0", "search BM25 for the planted token = 0 hits", "list the object prefix = empty") and run them after the purge, recording results in an append-only audit log: who or what requested the deletion, when, and the verification outcome. Erasure on request (a user asks to be forgotten) is the same flow keyed by person rather than by document, which adds the job of finding all documents that concern them. The legal rules differ by jurisdiction; Red-Teaming and Compliance as Code covers the Indian DPDP context, and the retention and deadline details here must come from your counsel's reading, not from this lesson (I could not retrieve the primary text for this pass).

:::warn Do not log full prompts and retrieved chunks forever "for debugging". Every log line containing document text is a copy that the delete flow must reach. Log document IDs and hashes, keep text logs on a short retention, and put them in the purge list. :::

Backfills, quality gates, and the cost of ingesting a document

Every ingestion pipeline needs gates that stop bad documents from becoming bad answers, and a backfill path that does not take the live system down.

Quality gates before indexing. Cheap checks that run on every document:

Count the rejects by reason and alert on shifts. If rejected-for-empty-parse jumps from 0.2 to 6 percent after a connector update, the parser, not the documents, changed.

Backfills and re-embedding without downtime. A model change or a chunker change means reprocessing the corpus. Use the blue/green approach from Choosing, Evaluating and Fine-Tuning an Embedding Model: build the new index alongside, replay from the raw store (so you do not hit source APIs again), dual-write new changes, shadow-compare, flip. Throttle the backfill below the embedding provider's rate limit and keep it on a low-priority lane. Estimate the duration before starting: 9.6B tokens at an assumed 2M tokens per minute of embedding throughput is 4,800 minutes, 80 hours.

Now design the whole thing for the prompt (20 million documents with permissions). Scale the numbers by ten: a full embed is about $1,920 in embedding tokens at the assumed rate, 600,000 documents change daily at 3 percent, and 240M chunks need the index tiering from ANN Indexes in Depth. The architecture is the one in this lesson: connectors writing to a raw store with per-source rate limits; parse gates and a dead-letter queue; hash-based incremental sync with nightly reconciliation; ACLs in chunk metadata with a fast path; database-enforced filtering; tombstone-and-purge with an audit trail; canary freshness; a leak suite in CI. Then say what you would measure first: freshness p95, rejected-document rate, and leak-suite pass rate.

How to say it in the interview

"I treat ingestion as a pipeline with a raw store, idempotent stages and a reconciliation job. Changes are detected with content hashes and change feeds, so we re-embed three percent of documents a day instead of everything, which at our sizes is about six dollars instead of two hundred. Permissions are copied into chunk metadata, changed on a fast metadata-only path, and enforced in the database at query time, and I prove it with a leak suite of planted documents. A delete is a soft-delete that hides the document within seconds and a hard purge across vectors, keyword postings, parent text, caches, logs and the audit trail within a stated window, each with a verification query. Freshness is measured end to end with a canary document, not assumed."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons