Cruxly › AI engineering › Portfolio Projects

Project: A RAG Assistant Over Real Documents With an Eval Gate in CI

hybrid retrieval, cited answers, abstention, a golden set with intervals and a CI gate, over the Python standard-library docs

Project: "Our engineers keep asking the same questions about the Python standard library in chat. Build an assistant that answers from the official documentation, shows which section each claim comes from, says 'I do not know' when the docs do not cover it, and cannot get worse without CI turning red."

Time-box: 20 hours minimum, about 30 typical, 45 with the stretch goals. You will have a tested package (rag, standard library plus llmkit, 16 passing tests) with structure-aware chunking and idempotent ingestion, SQLite holding both an FTS5 index and vectors, dense, BM25 and fused retrieval with a reranker stand-in, a citation verifier, an abstain path, a 50-question golden set with held-out slice, an eval script that prints an ablation table with intervals, a regression gate, a streaming WSGI endpoint with per-request traces, and a README with real numbers.

Why this project, the corpus and the licence, scope and non-goals

Every company asks for a RAG assistant, and a plain 'chat with a PDF' demo is the project hiring-side sources dismiss; what separates yours is what you measure. This one measures retrieval recall against answer quality separately, citation correctness, abstention on questions the documents do not answer, cost and latency, and a deploy gate. The RAG From Scratch, No Framework lesson builds the pipeline once; this project makes it measurable and defensible.

The corpus. A portfolio project should use a real, redistributable corpus. I used the Python standard-library documentation.

:::key Verified 24 October 2026 The docs.python.org licence page states: 'This page is licensed under the Python Software Foundation License Version 2. Examples, recipes, and other code in the documentation are additionally licensed under the Zero Clause BSD License.' It also says that starting with Python 3.8.6, examples and recipes are dual licensed under the PSF licence version 2 and Zero-Clause BSD. The page was the 3.15.1 documentation. The text build of the docs is downloadable as https://docs.python.org/3/archives/python-3.15-docs-text.zip (I downloaded it on that date; an earlier guess of a 3.14 filename returned an HTML 'Archive not found' page, so check the download page for the current name). Keep the licence notice and attribution in your README if you redistribute any of it. :::

Two corpora, on purpose. Tests must run anywhere with no download, so the repo ships a 12-file mini corpus I wrote myself (original short summaries of twelve modules: json, pathlib, sqlite3, collections, heapq, itertools, re, datetime, logging, argparse, subprocess, functools; 3 sections each, 36 chunks). Everything measured in this lesson is on that fixture. A second script, ingest_docs.py, ingests the real documentation, which I ran once to learn the scale (later section). Writing the fixture yourself also means there is no redistribution question for the part that sits in git.

Scope. Question in, cited answer or abstention out, streamed over HTTP. Non-goals: a vector database cluster, multi-tenant permissions, query-time web search, a polished UI. What a reviewer will probe: why hybrid retrieval and how you chose chunk size; how you know it is not hallucinating; how the eval gate works and when it caught something; how you would debug 'retrieval is fine but answers are wrong'; what changes with permissions and 10 million documents.

Success metric, set before building. On the golden set: recall@5 of at least 0.85 on answerable questions, abstain on most unanswerable ones without refusing answerable ones, tokens per answer under a budget, and a CI gate that fails if recall@5 drops more than 3 points, answer-correct drops more than 2 points, or tokens per answer rise more than 20 percent. You will see that the abstention goal is missed, and that is part of the story.

Builds on. Chunking and Document Parsing: Structure-Aware Splits, Parent-Child, Late Chunking and Contextual Retrieval, Hybrid Search, Rerankers and Late Interaction, Choosing, Evaluating and Fine-Tuning an Embedding Model, Evaluation Gates That Actually Block a Bad Deploy, Metrics That Mean Something: Task Metrics, Ranking Metrics, pass@k and Significance, Hallucination: Taxonomy, Detection and Mitigation in Production, Streaming Interfaces That Show Their Work and Case Study: Enterprise Search and Q&A Over 20 Million Documents With Permissions. Those lessons carry the reasoning; this one builds and measures.

Architecture and the eight milestones

Run it. From the project folder: python -m unittest discover -s tests for the 16 tests, python evals/make_golden.py to rewrite the golden file, python evals/run.py for the full report and gate, python demo_ask.py for three worked questions. Last test run:

----------------------------------------------------------------------
Ran 16 tests in 1.335s

OK

Milestone 1: structure-aware chunks and idempotent ingestion

Chunk by the document's own structure, carry the heading into the text, and make re-ingestion free. The chunker below splits a markdown file at ## headings and then packs whole sentences up to a word limit, so a chunk never crosses a heading and never ends mid-sentence. Each chunk's text starts with doc > heading., which gives both retrievers a topical signal that a bare paragraph lacks (the idea behind contextual chunk headers). Each chunk has a stable id (json#0) and a content hash.

# -*- coding: utf-8 -*-
"""rag.chunk: structure-aware chunks. Never cross a heading; carry the heading into the text."""
import hashlib
import re
from dataclasses import dataclass
from typing import List, Tuple


@dataclass(frozen=True)
class Chunk:
    id: str        # "<doc>#<n>", n counts chunks within the document
    doc: str
    heading: str
    text: str
    hash: str


def split_sections(md: str) -> List[Tuple[str, str]]:
    """[(heading, body)] for every '## ' section; the '# title' line is ignored."""
    out, head, buf = [], None, []
    for line in md.splitlines():
        if line.startswith("## "):
            if head is not None:
                out.append((head, " ".join(buf).strip()))
            head, buf = line[3:].strip(), []
        elif head is not None and line.strip():
            buf.append(line.strip())
    if head is not None:
        out.append((head, " ".join(buf).strip()))
    return out


def chunk_document(doc: str, md: str, max_words: int = 120, overlap: int = 0,
                   with_heading: bool = True) -> List[Chunk]:
    """Pack whole sentences into chunks of at most max_words (a single long sentence may exceed it).
    overlap = number of trailing sentences repeated at the start of the next chunk of the same section."""
    chunks, n = [], 0
    for heading, body in split_sections(md):
        sents = re.split(r"(?<=[.!?])\s+", body)
        cur: List[str] = []
        pieces = []
        for s in sents:
            if cur and sum(len(x.split()) for x in cur) + len(s.split()) > max_words:
                pieces.append(cur)
                cur = cur[-overlap:] if overlap else []
            cur.append(s)
        if cur:
            pieces.append(cur)
        for p in pieces:
            text = " ".join(p)
            if with_heading:
                text = "%s > %s. %s" % (doc, heading, text)
            chunks.append(Chunk("%s#%d" % (doc, n), doc, heading, text,
                                hashlib.sha1(text.encode("utf-8")).hexdigest()[:12]))
            n += 1
    return chunks

The store is a single SQLite file: a chunks table with the vector as JSON, and an FTS5 virtual table for keyword search. Index.add skips a chunk whose id and hash are unchanged, which is what makes ingestion idempotent: run it twice and the second run does zero writes. A changed paragraph rewrites only its own chunk. Dense search is a brute-force cosine scan over the vectors held in memory, which is honest and simple at this size and, as the scale section shows, the first thing to replace at thousands of chunks.

# -*- coding: utf-8 -*-
"""rag.index: one SQLite file holds chunks, an FTS5 table for BM25, and vectors for dense search."""
import json
import re
import sqlite3
from typing import Dict, List, Optional, Tuple

from llmkit import HashEmbedder, cosine

from .chunk import Chunk

STOP = set("a an the of to in is are was be do does how what which i it and or for with on as at by from that this "
           "can you my me get if when not no into out up all any use using only each so than then there their".split())


def terms(text: str) -> List[str]:
    return [t for t in re.findall(r"[a-z0-9]+", text.lower()) if t not in STOP]


class Index:
    def __init__(self, path: str = ":memory:", embedder: Optional[HashEmbedder] = None):
        self.db = sqlite3.connect(path)
        self.emb = embedder or HashEmbedder()
        self.db.executescript(
            "create table if not exists chunks(id text primary key, doc text, heading text, text text,"
            " hash text, vec text);"
            "create virtual table if not exists fts using fts5(id unindexed, text);")
        self.vecs: Dict[str, List[float]] = {}
        for cid, v in self.db.execute("select id, vec from chunks"):
            self.vecs[cid] = json.loads(v)

    def add(self, chunks: List[Chunk]) -> int:
        """Idempotent: a chunk whose id and hash are unchanged costs zero writes."""
        writes = 0
        for c in chunks:
            row = self.db.execute("select hash from chunks where id=?", (c.id,)).fetchone()
            if row and row[0] == c.hash:
                continue
            vec = self.emb.embed_one(c.text)
            self.db.execute("delete from fts where id=?", (c.id,))
            self.db.execute("insert or replace into chunks values (?,?,?,?,?,?)",
                            (c.id, c.doc, c.heading, c.text, c.hash, json.dumps(vec)))
            self.db.execute("insert into fts(id, text) values (?,?)", (c.id, c.text))
            self.vecs[c.id] = vec
            writes += 1
        self.db.commit()
        return writes

    def get(self, cid: str) -> Chunk:
        r = self.db.execute("select id, doc, heading, text, hash from chunks where id=?", (cid,)).fetchone()
        return Chunk(*r)

    def count(self) -> int:
        return len(self.vecs)

    def dense(self, query: str, k: int) -> List[Tuple[str, float]]:
        q = self.emb.embed_one(query)
        scored = sorted(((cosine(q, v), cid) for cid, v in self.vecs.items()), reverse=True)
        return [(cid, s) for s, cid in scored[:k]]

    def bm25(self, query: str, k: int) -> List[Tuple[str, float]]:
        ts = terms(query)
        if not ts:
            return []
        match = " OR ".join('"%s"' % t for t in ts)
        rows = self.db.execute("select id, bm25(fts) from fts where fts match ? order by bm25(fts) limit ?",
                               (match, k)).fetchall()
        return [(cid, -s) for cid, s in rows]     # FTS5 returns lower-is-better; flip so higher is better

What the tests prove. test_sizes_and_headings chunks a small document at 6 words and asserts the headings of the chunks are One, One, Two (never crossing) and that every chunk is within the limit; test_idempotent_and_changed_chunk_rewrites_only_itself asserts 2 writes, then 0, then 1 after editing one sentence; test_overlap_repeats_last_sentence checks the overlap option. FTS5 availability is a runtime fact, not a given: my Python 3.9.13 on Windows bundles SQLite 3.37.2 with FTS5 compiled in, and the create virtual table line in the index would raise OperationalError on a build without it. FTS5's bm25() returns lower-is-better values (I observed negative numbers), so bm25 flips the sign to keep 'higher is better' consistent across retrievers.

Milestones 2 to 4: the golden set, retrieval and the ablation

Build the golden set before you tune, label by section not by chunk, and write some questions you expect to fail. evals/golden/qa_v1.jsonl has 50 questions, generated by evals/make_golden.py from a readable table so you can edit it. Each row has a type, the question, the gold (document, heading) pairs, and facts the answer must contain. Gold is a section, not a chunk id, so you can change the chunker later and the set still means the same thing (the Choosing, Evaluating and Fine-Tuning an Embedding Model lesson makes the same point about span labels).

Every fifth question (10 of 50) is held out. test_golden_set_is_valid asserts every gold section exists in the corpus, that answerable means non-empty gold, the five types, at least 15 percent unanswerable and at least 20 percent held out.

Avoid lexical leakage. The easiest mistake is writing a question by reading the chunk and reusing its words, which flatters keyword search. The paraphrase rows are deliberately worded without the document's vocabulary ('Tally how often each element shows up in a sequence' for Counter), and the exact rows are bare identifiers, the case often claimed to favour keyword search. Here the hashed-bag-of-words dense leg also scored 1.000 on them, because identifiers are just tokens to it, so this fixture does not show dense retrieval failing on identifiers; a semantic embedder might.

The retrievers. Dense search uses llmkit's HashEmbedder; keyword search uses FTS5 with an OR query over stop-word-filtered terms; fusion is reciprocal rank fusion, score(d) = sum over lists of 1 / (60 + rank), which needs no score calibration because it uses ranks only; the reranker is a stand-in that scores each candidate by the fraction of the query's idf mass that appears in it. test_rrf_hand_worked checks the fusion order b, c, a, d for two small lists against a hand calculation.

# -*- coding: utf-8 -*-
"""rag.retrieve: dense, BM25, reciprocal rank fusion, and a stand-in reranker behind one function."""
import math
import re
from collections import defaultdict
from dataclasses import dataclass
from typing import Dict, List, Sequence, Tuple

from .index import Index, terms


@dataclass
class Hit:
    id: str
    score: float      # fused (or single-retriever) score
    rerank: float     # idf-weighted query-term coverage, 0..1


def rrf(rankings: Sequence[Sequence[str]], k: int = 60) -> List[Tuple[str, float]]:
    """Reciprocal rank fusion: score(d) = sum over lists of 1 / (k + rank), rank starting at 1."""
    s: Dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for r, cid in enumerate(ranking, 1):
            s[cid] += 1.0 / (k + r)
    return sorted(s.items(), key=lambda x: (-x[1], x[0]))


def coverage(index: Index, query: str, text: str) -> float:
    """Fraction of the query's idf mass that appears in the chunk. A cheap, local reranker stand-in."""
    n = max(1, index.count())
    qs = set(terms(query))
    if not qs:
        return 0.0
    have = set(terms(text))
    df = lambda t: index.db.execute("select count(*) from fts where fts match ?", ('"%s"' % t,)).fetchone()[0]  # noqa: E731
    idf = {t: math.log(1 + n / (1 + df(t))) for t in qs}
    return sum(idf[t] for t in qs if t in have) / sum(idf.values())


def split_query(query: str) -> List[str]:
    """Deterministic stand-in for LLM query decomposition: split a compound question at clause words."""
    parts = re.split(r",|;| and | then | with | using | out of | that ", query)
    return [p for p in (x.strip() for x in parts) if len(terms(p)) >= 2]


def retrieve(index: Index, query: str, mode: str = "hybrid+rerank", k: int = 5, pool: int = 30) -> List[Hit]:
    if mode == "hybrid+rerank+split":
        lists = [retrieve(index, p, "hybrid+rerank", 3) for p in split_query(query)]
        lists.append(retrieve(index, query, "hybrid+rerank", k))
        merged, seen = [], set()
        for rank in range(k):                       # round-robin, so every clause gets a seat
            for lst in lists:
                if rank < len(lst) and lst[rank].id not in seen:
                    seen.add(lst[rank].id)
                    merged.append(lst[rank])
        return merged[:k]
    dense = index.dense(query, pool)
    sparse = index.bm25(query, pool)
    if mode == "dense":
        cands = dense
    elif mode == "bm25":
        cands = sparse
    else:
        cands = rrf([[c for c, _ in dense], [c for c, _ in sparse]])
    hits = [Hit(cid, s, 0.0) for cid, s in cands[:pool]]
    if mode.endswith("rerank"):
        for h in hits:
            h.rerank = coverage(index, query, index.get(h.id).text)
        hits.sort(key=lambda h: (-h.rerank, -h.score))
    return hits[:k]

The ablation, computed by running the code.

## Corpus: 36 chunks from 12 documents; golden set 50 questions (42 answerable, 8 unanswerable), 10 held out

## Retrieval ablation (section-level gold, k=5; recall@5 mean with 95% bootstrap CI; n=42 answerable)
| retriever | recall@5 | MRR | recall@5 held-out |
|---|---|---|---|
| dense | 0.651 (0.51-0.79) | 0.580 | 0.500 (n=8) |
| bm25 | 0.897 (0.81-0.97) | 0.893 | 0.875 (n=8) |
| hybrid | 0.845 (0.74-0.94) | 0.732 | 0.875 (n=8) |
| hybrid+rerank | 0.897 (0.81-0.97) | 0.881 | 0.875 (n=8) |
| hybrid+rerank+split | 0.921 (0.84-0.98) | 0.827 | 0.938 (n=8) |

Paired recall@5, dense -> hybrid+rerank: mean diff +0.246, 95% CI +0.119..+0.373, wins 11, losses 0, ties 31
Paired recall@5, hybrid+rerank -> +split: mean diff +0.024, 95% CI +0.000..+0.060, wins 2, losses 0, ties 40

## Recall@5 by question type: before (hybrid+rerank) and after (+split)
| type | n | before | after |
|---|---|---|---|
| lookup | 20 | 1.000 | 1.000 |
| multi | 6 | 0.611 | 0.778 |
| exact | 8 | 1.000 | 1.000 |
| paraphrase | 8 | 0.750 | 0.750 |

## Chunk-size sweep, hybrid+rerank+split (words per chunk; sections are 40-90 words)
| max words | chunks | recall@5 |
|---|---|---|
| 12 | 123 | 0.909 |
| 25 | 100 | 0.909 |
| 50 | 52 | 0.909 |
| 120 | 36 | 0.921 |

## End to end with the scripted extractive model (tau=0.15 chosen on the dev split only)
answer-correct (all required facts present): 0.595 (0.45-0.74), n=42
citation precision 0.625, citation recall 0.837 (answered questions)
abstained on unanswerable: 2 of 8 (wilson 0.07-0.59); wrongly abstained on answerable: 0 of 42
abstention precision 1.000
error buckets over 42 answerable: all gold sections in the 4 context chunks and answer correct 24; gold missing from context (retrieval) 7; gold present but answer incomplete or wrong (generation stand-in) 11
held-out only: answer-correct 0.500 (n=8), abstained 0 of 2 unanswerable, wrongly abstained 0 of 8
generation calls 48, tokens per call 482, assumed-price cost per 1,000 answers $0.091; retrieval latency on this machine p50 16.2 ms, p95 26.3 ms (index of 36 chunks)

## Retrieval misses after the split fix (recall@5 < 1): 5 of 42
- q21 [multi] recall 0.67  I want to load a JSON config from disk with pathlib and report bad JSO
- q24 [multi] recall 0.50  I need the few earliest deadlines out of many, held as date objects. W
- q25 [multi] recall 0.50  Write a command line tool with a verbose switch that turns on DEBUG me
- q36 [paraphrase] recall 0.00  Tally how often each element shows up in a sequence
- q37 [paraphrase] recall 0.00  Make a function remember its earlier answers

GATE: PASS {'recall5': 0.9206, 'answer_correct': 0.5952, 'tokens_per_answer': 482, 'false_abstain': 0.0}

How to read the retrieval part.

• The dense leg here is weak, and that is a fact about my embedder, not about dense retrieval. HashEmbedder is a hashed bag of words; its recall@5 of 0.651 says nothing about a real embedding model, which you should measure on your own set (Choosing, Evaluating and Fine-Tuning an Embedding Model). The paired comparison dense versus hybrid+rerank is +0.246 (95 percent bootstrap interval +0.119 to +0.373; 11 questions better, none worse), so on this fixture the gain is real.
• Plain fusion made it worse than BM25 alone (0.845 versus 0.897, MRR 0.732 versus 0.893). Reciprocal rank fusion gives the weak leg a vote, and a bad voter dilutes a good ranking. Hybrid is not automatically better than its best leg; the ablation is how you find out. The reranker recovers the loss (0.897, MRR 0.881).
• By type, lookups and exact identifiers are solved (1.000); multi-section and paraphrase questions are where the misses live. The five questions still below full recall are listed at the end of the report.
• The chunk-size sweep is flat (0.909 at 12, 25 and 50 words, 0.921 at 120). On a corpus whose sections are 40 to 90 words, chunk size cannot matter much, so I would not tune it here; the sweep is a template to rerun on the real corpus, where sections are long.
• Held-out numbers are shown beside the overall ones. With 8 answerable held-out questions they are a smoke test, not a result: one question is 12.5 points.

Milestone 5: cited answers, a verifier, and abstention

Make citations a contract the code checks, not a request the model may ignore. The prompt tells the model to answer only from the sources and to put a source id after every sentence. Then a verifier checks three things per sentence: it has at least one citation, every cited id was actually retrieved, and the sentence's content words are mostly (at least 60 percent) found in the cited text. If any sentence fails, the answer is not shown.

# -*- coding: utf-8 -*-
"""rag.generate: cited answers, a citation verifier, and an abstain path."""
import re
from dataclasses import dataclass, field
from typing import Dict, List

from llmkit import LLMClient

from .index import Index, terms
from .retrieve import Hit, retrieve

SYSTEM = ("Answer using ONLY the sources. After every sentence put the id of the source that supports it, "
          "like [json#1]. If the sources do not contain the answer, say you do not know.")
ABSTAIN = "I do not know: the documentation I have does not answer that."
CITE = re.compile(r"\[([a-z0-9_]+#\d+)\]")


def build_messages(question: str, ctx: List[tuple]) -> List[dict]:
    src = "\n".join("[%s] %s" % (cid, " ".join(text.split())) for cid, text in ctx)
    return [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": "Question: %s\n\nSources:\n%s" % (question, src)}]


def sentences(text: str) -> List[str]:
    return [s for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s]


def verify(answer: str, ctx: Dict[str, str], min_support: float = 0.6) -> List[str]:
    """Problems found in a cited answer. Empty list means every sentence is cited and supported."""
    problems = []
    for s in sentences(answer):
        ids = CITE.findall(s)
        body = set(terms(CITE.sub("", s)))
        if not ids:
            problems.append("uncited: " + s[:60])
        elif any(i not in ctx for i in ids):
            problems.append("cites a source that was not retrieved: " + s[:60])
        elif body:
            have = set(terms(" ".join(ctx[i] for i in ids)))
            if len(body & have) / len(body) < min_support:
                problems.append("not supported by its citation: " + s[:60])
    return problems


class ExtractiveModel:
    """Scripted stand-in for an LLM: quotes the best-matching sentence of the top sources, with citations.
    It tests retrieval, prompting plumbing and the verifier; it says nothing about a real model."""

    def __call__(self, messages) -> str:
        user = messages[-1]["content"]
        q = set(terms(user.split("\n")[0].replace("Question:", "")))
        srcs = []
        for line in user.split("Sources:\n")[1].split("\n"):
            m = re.match(r"\[([^\]]+)\] (.*)", line)
            if m:
                srcs.append((m.group(1), sentences(m.group(2))))
        # a query term that appears in few sentences of the context is more informative than one in many
        df = {t: sum(t in set(terms(s)) for _, ss in srcs for s in ss) for t in q}
        weight = lambda s: sum(1.0 / df[t] for t in q & set(terms(s)))  # noqa: E731
        out, first = [], 0.0
        for cid, ss in srcs:
            best = max(ss, key=weight)
            w = weight(best)
            first = first or w
            if w and w >= 0.6 * first:
                out.append("%s [%s]." % (best.rstrip("."), cid))
            if len(out) == 2:
                break
        return " ".join(out) or ABSTAIN


@dataclass
class Answer:
    text: str
    abstained: bool
    cited: List[str] = field(default_factory=list)
    problems: List[str] = field(default_factory=list)
    hits: List[Hit] = field(default_factory=list)


def answer(index: Index, llm: LLMClient, question: str, tau: float = 0.5, k_ctx: int = 4,
           mode: str = "hybrid+rerank+split") -> Answer:
    hits = retrieve(index, question, mode, k=k_ctx)
    if not hits or max(h.rerank for h in hits) < tau:
        return Answer(ABSTAIN, True, hits=hits)
    ctx = [(h.id, index.get(h.id).text) for h in hits]
    text = llm.complete(build_messages(question, ctx), temperature=0, max_tokens=300).text
    problems = verify(text, dict(ctx))
    if problems:
        return Answer(ABSTAIN, True, problems=problems, hits=hits)   # an unverifiable answer is not shown
    return Answer(text, False, CITE.findall(text), [], hits)

The model in the tests is a scripted stand-in. ExtractiveModel quotes the best-matching sentences of the top sources with their citations. It exercises the prompt plumbing, the verifier and the metrics, and it tells you nothing about a real model's answers; for those, run your eval through OpenAICompatClient and label the result your own measurement.

The tests. test_verifier feeds four answers (supported, uncited, citing an id that was never retrieved, citing a real id with an unrelated claim) and asserts each verdict. test_uncited_model_answer_is_not_shown scripts a model that answers without citations and asserts the pipeline abstains and records the problem. test_below_threshold_abstains_without_calling_the_model asserts that a question with no support makes zero model calls.

Choosing the abstention threshold honestly. tau is the minimum best-chunk coverage (the signature default of 0.5 is only a placeholder; the eval passes the value pick_tau chose, 0.15). pick_tau sweeps it using only the dev questions (the 40 not held out) and takes the largest value that wrongly abstains on at most 5 percent of answerable dev questions; the held-out questions then get a single look. Abstention is a trade: raising tau refuses more unanswerable questions and more answerable ones.

What it printed for three questions (tau 0.15):

Q: How do I make a pattern succeed only when the entire string fits it?
   retrieved: [('re#1', 0.22), ('re#0', 0.22), ('heapq#2', 0.12), ('datetime#1', 0.1)]
   A: sub replaces every match with a replacement string or a function [re#1]. re.match only succeeds if the pattern matches at the very beginning of the string [re#0]. | abstained: False | cited: ['re#1', 're#0']
Q: I want to load a JSON config from disk with pathlib and report bad JSON properly. What do I need?
   retrieved: [('json#0', 0.33), ('json#1', 0.34), ('json#2', 0.14), ('argparse#1', 0.2)]
   A: json.loads parses a string, while json.load reads from a file object that is already open [json#0]. The exception carries msg, doc, pos, lineno and colno attributes, so a program can report exactly where the parser gave up [json#1]. | abstained: False | cited: ['json#0', 'json#1']
Q: How do I send an email with smtplib?
   retrieved: [('logging#2', 0.45), ('sqlite3#1', 0.0), ('heapq#0', 0.0), ('datetime#2', 0.0)]
   A: Prefer these to printing the exception, because the traceback goes wherever the handlers send it [logging#2]. | abstained: False | cited: ['logging#2']

Read those three carefully, because they are the lesson in miniature. The first question asks about matching the whole string; the model cited re#1 and re#0 and the quoted sentences are supported by their citations, yet the best answer (fullmatch) is in re#0's section and the first sentence quoted is about sub: supported is not the same as relevant. The second works, including the multi-section case. The third is an unanswerable question about smtplib, and the system did not abstain: the reranker found a chunk containing the word 'send' ('...goes wherever the handlers send it'), gave it coverage 0.45, and the verifier approved the answer because the sentence really is in that chunk. A citation check proves that a sentence came from the source, not that it answers the question.

Milestones 6 and 7: the end-to-end report, the error analysis, and the gate

Read the end-to-end block in the report, then classify every miss before you change anything. With the scripted model and tau 0.15 chosen on dev: answer-correct (all required facts present) is 0.595 with interval 0.45 to 0.74 on 42 answerable questions; citation precision is 0.625 and citation recall 0.837; abstention caught 2 of 8 unanswerable questions (Wilson interval 0.07 to 0.59) and wrongly refused none of the 42 answerable ones. Held-out only: answer-correct 0.500 on 8 questions and 0 of 2 unanswerable refused, which is far too small to say anything.

The error taxonomy answers 'retrieval is fine but answers are wrong'. The report sorts the 42 answerable questions into buckets: 24 have all gold sections in the four context chunks and a correct answer; 7 are missing a gold section from the context (a retrieval or context-assembly problem); 11 have the gold sections present but an incomplete or wrong answer (here the scripted extractor quoting one sentence per source). Measure each stage separately: is the gold chunk in the top k, is it in the prompt, did the model use it, is the gold label itself wrong?

The fix I made, and what the numbers say about it. The biggest retrieval class is the multi-section question. Its misses happen because one long natural-language query is a poor match for any single section, and the reranker then favours chunks that share many generic words. I added hybrid+rerank+split: split a compound question at clause words (and, with, commas), retrieve for each part, and interleave the results so every clause gets a seat. The table above shows the result. Multi-section recall rose from 0.611 to 0.778; overall recall@5 rose from 0.897 to 0.921; MRR fell from 0.881 to 0.827; the paired interval for overall recall is +0.000 to +0.060 with 2 questions better and none worse. In plain words: a small gain concentrated where intended, a ranking cost elsewhere, and an interval that touches zero. I kept it because it targets the named class and no question got worse on recall, but I would not claim it as a proven win, and in the README it appears with exactly that sentence. A real system would use an LLM for decomposition; this deterministic splitter is the stand-in.

What I did not fix. The two paraphrase questions with zero recall ('Tally how often each element shows up...', 'Make a function remember its earlier answers') share no vocabulary with their sections; only a real embedding model, query rewriting or synonyms can close that gap, and I have no measurement for them, so the README lists it as open. Abstention (2 of 8) is the other open item for the same reason.

The gate. evals/baseline.json stores recall@5 0.9206, answer-correct 0.5952 and 482 tokens per answer. python evals/run.py exits 1 if recall@5 falls more than 0.03, answer-correct falls more than 0.02, or tokens per answer rise more than 20 percent. The thresholds are my judgement: with 42 questions one question moves recall by up to 2.4 points, so 3 points tolerates one flipped question and catches two; on a real 120-question set you would tighten them and say why. A gate that has never failed is unproven, so I simulated the commonest regression, someone removing the BM25 leg (--degrade runs the dense-only retriever):

GATE: FAIL - recall@5 fell 0.921 -> 0.651; answer-correct fell 0.595 -> 0.452 {'recall5': 0.6508, 'answer_correct': 0.4524, 'tokens_per_answer': 469, 'false_abstain': 0.0714}

The recall drop is the 0.651 dense row from the ablation table; add this command after the unit tests in your CI workflow and keep the failing output (or a screenshot) in the README. A pull request that improves recall but doubles the context size fails the token check, which is the point.

Cost, latency, scale, observability and deployment

Budget the tokens first. As a planning assumption (not a measurement), one answer carries about 8 chunks of about 350 tokens plus a prompt, roughly 3,000 context tokens, and about 300 output tokens. At assumed prices of $0.15 per million input and $0.60 per million output tokens, that is (3,000 x 0.15 + 300 x 0.60) / 1,000,000 = $0.00063 per answer, $0.63 per 1,000 answers; check your provider's pricing page and date it. The fixture run is smaller: the scripted model sees 4 short chunks, 482 tokens per call (estimated at four characters per token), $0.091 per 1,000 answers at the same assumed prices. A test caps the largest prompt in the golden run below a token budget so a change that stuffs the context fails CI.

Latency. On the 36-chunk fixture, retrieval took a median of 16.2 ms and 95th percentile of 26.3 ms on my Windows laptop. Those are real measurements on that machine and will differ on yours (a rerun on a different day gave a median near 90 ms, so treat them as one noisy sample). Generation latency from a model is not measured here, so it is not quoted.

What happens at real scale. I ran ingest_docs.py on the Python 3.15 text documentation (the library folder): 328 files became 7,023 chunks, the build took 66.2 seconds and the SQLite file is 43.9 MB. A query then took 0.7 to 1.1 seconds in pure Python (668, 1,059 and 716 ms for three test queries). My target was retrieval under 150 ms, and a pure-Python scan over 7,000 vectors plus per-term document-frequency lookups misses it by a large factor. That is the signal to move dense search to numpy, sqlite-vec or FAISS and to precompute idf; I did not measure those replacements, so I make no claim about their speed. I also did not build a golden set for the real corpus, so I make no recall claim for it. Embedding the real corpus with a hosted model would cost roughly chunk count times average chunk tokens times price: 7,023 chunks at an assumed 200 tokens is about 1.4 million tokens, which at an assumed $0.10 per million is about $0.14. Both inputs are assumptions.

Observability. rag/api.py is a WSGI app: GET /ask?q=... streams server-sent events (token events, then citations, then done), GET /health, a 300-character question limit. It appends one trace line per request: the question, retrieved ids with fused and rerank scores, whether it abstained, verifier problems, tokens, dollars and milliseconds. TestApi calls it directly and asserts the event order, the trace content and the 400 on an empty or oversized question. A thumbs-up/down endpoint that writes failures to a table, exported as candidate golden-set cases, closes the loop (Observability and Cost covers the pattern).

Deployment. Build the index in CI and bake the SQLite file into the container image, so the instance never builds an index at boot; add a per-IP rate limit, the question limit, a daily token cap with a fallback to cached answers, and run python -m unittest plus python evals/run.py in the workflow.

:::key Verified 24 October 2026 Render free web services spin down after 15 minutes without traffic and take about a minute to wake, have no persistent disk and get 750 free instance hours a month per workspace (render.com/docs/free). A Hugging Face Docker Space needs a paid plan for a new account; static Spaces are free (huggingface.co/docs/hub/spaces-overview). Both are the primary pages fetched on that date. Re-check before you rely on them, and put the cold-start behaviour in the README. :::

Present it: README numbers, interview questions, pitch and resume bullet

The README results block, filled from this repository's own run (24 October 2026, scripted model, 50-question fixture).

Five interview questions.

@@ 1 | Why hybrid retrieval, and how did you choose chunk size? The textbook argument is that dense search handles paraphrase and BM25 handles exact identifiers; RRF needs no score calibration. But be honest that my fixture did not show the identifier half (hashed dense scored 1.000 on exact rows) and show the ablation: plain fusion scored below BM25 alone here because my dense leg was weak, and a reranker recovered it. Chunk size was swept (flat at 0.909 to 0.921 on a corpus of short sections), so I did not tune it; the sweep is for the real corpus.

@@ 2 | How do you know it is not hallucinating? Citations are required and verified (cited, retrieved, supported); unsupported answers are suppressed; unanswerable questions test abstention. Then be candid: the verifier checks support, not relevance, and my abstention caught only 2 of 8; the remaining risk is wrong-but-supported answers.

@@ 3 | Walk me through your eval gate. Golden set provenance and held-out split, metrics and intervals, thresholds and why (one flipped question is 2.4 points), the baseline file, and the regression I simulated and the output that turned red.

@@ 4 | Retrieval is fine but answers are wrong. Debug it. Separate the stages: is the gold chunk in the top k, in the prompt, used by the model, or is the label wrong? My buckets: 24 correct, 7 retrieval or context misses, 11 gold present but answer wrong. Fix the largest bucket after checking labels.

@@ 5 | How would this change with permissions and 10 million documents? Push ACL filters into the index, partition per tenant, incremental ingestion with deletes (the content hash already gives incremental), an ANN index, and the lesson I measured: pure-Python scan took about a second at 7,000 chunks. See Case Study: Enterprise Search and Q&A Over 20 Million Documents With Permissions.

ILLUSTRATIVE resume bullet (replace with your own measured numbers): Built a hybrid-retrieval RAG assistant over the Python standard-library docs (SQLite FTS5 + vector index, citation verifier, abstention); lifted recall@5 from 0.65 (dense only) to 0.92 on a 50-question golden set with intervals, 10 questions held out; CI gate blocks recall, answer-correct and token-cost regressions.

ILLUSTRATIVE 90-second pitch. Problem: engineers ask the same stdlib questions; I built a cited assistant. Decision: BM25 plus dense with a reranker, because my hashed dense leg alone reached recall@5 0.65 against 0.90 for BM25 on my fixture (a weak stand-in embedder, so not a claim about dense retrieval). Measurement: 50 questions, section-level gold, intervals, a regression gate. Failure: multi-topic questions pulled in the wrong sections; splitting at clause words lifted that class from 0.61 to 0.78, with MRR cost. Next: a real embedding model, because paraphrase recall and abstention are my open gaps.

Pitfalls. Golden questions written by reading the chunk text; evaluating generation without separating retrieval failures; an unvalidated LLM judge (here the 'judge' is a required-facts check, which is why I call it answer-correct and not faithfulness); building the index at container start; no unanswerable questions, so abstention is never tested; one run of a non-deterministic metric with no interval.

Stretch goals. A real embedding model behind the same embed interface and a measured ablation; LLM query rewriting replacing the splitter; numpy or sqlite-vec dense search with a recall and latency sweep; a golden set for the real 7,000-chunk corpus; permission-aware retrieval with two fake tenants; contextual chunk headers written by a model.

Time estimate. About 20 hours following this code, 30 typical with a real embedding model and a real golden set, 45 with the stretch goals.

How to say it in the interview

"The assistant is only as good as what I can measure, so I built the golden set before tuning and gate every change on recall and answer-correct. I ablated dense, BM25 and fusion separately: on my fixture BM25 beat my weak dense leg, plain fusion fell below BM25, and a reranker recovered it; every claim must cite a retrieved chunk and a verifier rejects uncited or unsupported sentences. I report intervals, I know my abstention is weak, and I can show the exact questions where retrieval, not generation, is the bottleneck."

Checklist

• Chose a corpus whose licence I verified and recorded (PSF licence, examples also Zero-Clause BSD), and kept the fixture used in tests original
• Chunks never cross a heading, carry the heading in their text, and ingestion is idempotent with a content hash
• Golden set has lookup, multi-section, exact-identifier, paraphrase and unanswerable questions, with gold labelled by section and a held-out slice
• Checked that no golden question was written by copying the chunk's words
• Reported dense, BM25, hybrid and hybrid+rerank separately, with intervals and a paired comparison
• Required citations, verified them in code, and tested the verifier with scripted bad answers
• Wrote down that citation support is not relevance and showed one failure where the verifier passed an irrelevant answer
• Chose the abstention threshold on the dev split only, reported held-out once, and stated the abstention result honestly (2 of 8)
• Classified every miss into retrieval, context assembly, generation or bad gold before changing anything
• Added a CI gate on recall, answer-correct and tokens, and kept evidence of it failing on a deliberate regression
• Measured scale on the real corpus (chunks, build time, query time) and labelled what I did not measure
• Built the index in CI rather than at container start, and put a rate limit, question limit and spend cap in front of any real-model path

Related lessons

• Build Projects That Get You Hired: The Process, the Shared Test Seam and How to Show It
• Project: A Document Extraction Pipeline With Validation and a Field-Level Scorecard
• Project: A Tracing and Cost Dashboard for LLM Apps (Build Your Own Mini-Langfuse)
• Project: An Eval Harness and LLM-as-Judge Toolkit You Can Trust
• Project: A Text-to-SQL Analyst With Read-Only Guardrails and Execution-Based Evals
• Project: A Tool-Using Agent With Guardrails, an Approval Gate and an Audit Log