Cruxly

Cruxly › AI engineering › Retrieval and RAG

RAG From Scratch, No Framework

chunk, embed, retrieve, generate — with only the SDKs

Open the interactive lesson

Prompt: build a working retrieval-augmented generation pipeline over a corpus of business documents, using nothing but an embedding SDK, a vector store client, and an LLM SDK — no LangChain, no LlamaIndex.

Give yourself the full session before reading on. The point is not the framework you skip; it's being able to say what every line of a framework would otherwise have hidden from you.

Chunking: the decision everything else inherits

Chunk size is a trade-off between precision and context. Small chunks (a paragraph) retrieve precisely but may lack the surrounding context an answer needs. Large chunks (a full section) carry context but dilute the similarity signal — a chunk about five different subtopics scores mediocre-relevant to queries about any one of them, rather than strongly-relevant to any.

Never chunk on a fixed character count alone. A naive text[i:i+500] slices mid-sentence, mid-table-row, sometimes mid-word. Chunk on semantic boundaries — paragraph breaks, section headers, sentence boundaries as a fallback — and only then apply a size cap, splitting an oversized semantic unit rather than an arbitrary one.

Overlap between chunks (10-20% of chunk length) prevents boundary loss. A fact stated across a chunk boundary with zero overlap is invisible to retrieval no matter how good your embeddings are — neither chunk alone contains it. Overlap is cheap insurance against a failure mode that is otherwise undetectable until a real query hits it.

Store chunk metadata alongside the vector: source document, section, page number, chunk position. When faithfulness evaluation later finds a wrong answer, metadata is what lets you trace it back to which specific chunk misled the model — without it, debugging a bad answer means re-reading the entire corpus.

Embedding and storage: the swap nobody notices until it's too late

An embedding model produces vectors in its own coordinate space. Vectors from two different embedding models are not comparable — cosine similarity between a vector from model A and a vector from model B is a meaningless number that will still look like a valid similarity score.

Swapping embedding models requires re-embedding the entire corpus, every time, with no exceptions. The dangerous version of this bug: someone upgrades the embedding model for new documents going forward, and old documents keep their stale vectors. The system keeps running, keeps returning results, and every result touching an old document is now silently comparing incompatible vector spaces. Version the embedding model in your metadata schema and refuse to run a mixed-version index.

Store the raw text alongside the vector, never the vector alone. You can't operationally recover usable source text from a vector — if you ever need to re-embed with a new model, re-chunk with a new strategy, or simply show a human what was retrieved, you need the source text sitting right next to its vector, not a foreign key chase back to a separate document store that may have changed since indexing.

:::warn Embeddings are not anonymised data "You can't get the text back" is an operational statement, not a privacy one. Embedding inversion is demonstrated: vec2text (Morris et al., 2023) recovers most short inputs exactly from their embeddings. Treat vectors derived from personal data as personal data — same access controls, same erasure obligations as the text they came from. :::

Retrieval and reranking: top-K is a starting point, not an answer

Retrieve more candidates than you'll use, then rerank. Vector similarity search is fast but approximate; retrieve the top 20-50 by similarity, then apply a more expensive, more accurate reranking model (a cross-encoder, which scores query-document pairs jointly rather than comparing independent vectors) to the smaller candidate set, and keep only the true top-K after that. This two-stage shape — cheap broad retrieval, expensive narrow reranking — is the standard answer to "vector search alone isn't accurate enough" and it is dramatically cheaper than running the expensive model over the whole corpus.

Set a relevance threshold and be willing to return nothing. If the best-matching chunk still scores below a meaningful relevance threshold, the honest answer is "I don't have information about that." Put the threshold on the reranker's score, not the raw embedding cosine: cosine similarity isn't calibrated — a 0.78 can be a strong match for one query and noise for another, and the range shifts with every embedding model — while a cross-encoder's relevance score is far more comparable across queries. Return "I don't know" rather than the top-1 result regardless of how weak the match is. A RAG system that always answers confidently, even when nothing relevant was retrieved, is optimizing for the appearance of usefulness over actual correctness, and it is the single fastest way to lose a user's trust in the system.

Watch the context window budget explicitly. Retrieved chunks, the system prompt, the conversation history, and the user's query all compete for the same token budget. A pipeline that blindly concatenates the top-10 chunks without checking total token count will occasionally overflow the model's context window — or worse, silently truncate the end of the prompt, which is often the user's actual question.

Generation: grounding the answer, and catching it when it isn't

The prompt must instruct the model to answer only from the provided context, and to say so explicitly when it cannot. "Using only the context below, answer the question. If the context does not contain enough information, say so rather than guessing." This single instruction is the cheapest, highest-leverage lever against hallucination available, and skipping it is the most common first-pass mistake.

Cite which chunk supports which claim, even in a v1. Returning inline citations ([1], [2]) mapped back to source chunks does two things: it lets a user verify the answer without trusting it blindly, and it gives you a mechanical way to check faithfulness later — does every claim in the answer actually trace back to a cited chunk, or did the model add something uncited.

Log the full retrieval-to-generation trace for every request: the query, the chunks retrieved with their scores, the final prompt sent to the LLM, and the response. Without this trace, "the system gave a wrong answer yesterday" is undebuggable — you cannot tell whether retrieval failed (wrong chunks), the prompt failed (right chunks, bad instructions), or generation failed (right prompt, model still went off-script).

Past the basics: contextual chunks, small-to-big, query rewriting, and agentic search

A chunk loses its context the moment it's cut out of the document. "Revenue grew 3% over the previous quarter" retrieves badly because the chunk never says which company or which quarter. Four techniques now count as the standard toolkit beyond fixed chunking:

And ask whether you need retrieval at all. With 200K-1M-token context windows, a small, stable corpus (a few hundred pages) can go straight into the prompt, with prompt caching keeping repeat cost low. Two caveats make retrieval still the default for anything larger or changing:

How to say it in the interview

"I want to build this once with raw SDKs before reaching for a framework, because the failure modes here are all silent — a bad chunk boundary, a stale embedding, a context window overflow — and I want to have personally seen each one before I trust a library to handle it for me."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons