Cruxly › AI engineering › Portfolio Projects

Project: A Document Extraction Pipeline With Validation and a Field-Level Scorecard

schema, repair loop, arithmetic and evidence checks, a second read, confidence routing and a scorecard that reports error among auto-accepted documents

Project: "Our finance team keys in about 2,000 supplier invoices a month. A vendor demo extracted them with an LLM and looked great on five samples. I need to know whether I can let the system post invoices without a person looking, and which ones a person must still see. Build it, and prove the numbers."

Time-box: 14 hours minimum, about 22 typical, 34 with the stretch goals. You will have a tested Python package (extractor, standard library plus llmkit, 22 passing tests), a seeded synthetic invoice corpus with perfect ground truth, a validation and repair loop, deterministic cross-checks, a router with three exits (accepted, repaired, human), an eval script that prints a field-level scorecard and a CI gate, a small SQLite review queue behind a WSGI app, and a README with real numbers.

Why this project, what it proves, and where its edges are

Extraction is the most common real enterprise LLM job, and it is unusually easy to evaluate, which makes it a strong first portfolio piece. The output is a record with known fields, so you can compare it to a label field by field; the failure modes are concrete (a wrong total, an invented tax number, a misread currency); and the engineering that matters (validation, repair, routing, measurement) is exactly what interviewers ask about. What the project proves to an employer is that you can make an LLM produce data a downstream system can trust, and that you know how often it is wrong.

Scope. One document type (invoices), three layouts (a plain list, a table-like layout and an Indian GST layout with a GSTIN), four currencies including INR, optional fields that are really absent (a missing due date, no GSTIN), and 1 to 5 line items. The input to the model is document text. Non-goals: training a model; handling handwriting; multi-page tables; Hindi or regional-language invoices (a stretch goal); a polished UI.

What a reviewer will probe, so build the answers in. How do you know the total is right? Why a repair loop and what are its limits? What happens when all the checks pass and the value is still wrong? How did you build the golden set and protect it? What changes at 100 times the volume or for a new layout? The interview section answers each with numbers from this project.

The success metric, written first. I fixed these before building: document fully correct after the pipeline, reported with a Wilson interval; error rate among auto-accepted documents at or below 1 percent as the product target; tokens per document under a budget; and a CI gate that fails if document-correct drops by more than 2 points or the auto-accepted error rate rises. You will see below that the product target is not met by this offline fixture, and why that is a result worth reporting.

Builds on. Structured Outputs: Schemas, Validation Loops and What 'Guaranteed' Means (why schema-valid is not true), Document Understanding: OCR, Layout Models, VLMs and the Hybrid That Usually Wins (where the text comes from), Case Study: Invoice and Contract Extraction at 500,000 Pages a Month (the production design reasoning), Evaluation Foundations: Golden Sets, Error Analysis and Reading Benchmarks and Evaluation Beyond Accuracy. This lesson builds and measures; it does not re-teach them.

Why the code is standard library only. The usual stack for this job is Pydantic, pdfplumber, FastAPI and Faker. I verified the current facts on 24 October 2026 (box below) and then deliberately wrote the schema with dataclasses and decimal, the review app as WSGI, and the generator with random. Reasons: the offline tests then run anywhere, I can show the whole validation logic in a few lines instead of hiding it behind a library, and the swap to Pydantic or FastAPI later is mechanical.

:::key Verified 24 October 2026 Pydantic on PyPI: latest 2.14.1 (released 11 October 2026); its 2.14.0 release dropped Python 3.9, which is one reason this lesson's 3.9-compatible schema uses dataclasses. pdfplumber on PyPI: 0.11.10 (15 June 2026), MIT licence, tested on Python 3.10 to 3.14 per its README. The PDF text-layer function in this lesson (layer_pages) uses pdfplumber but was not run in my offline suite, because I generated no PDFs; the tested part is the text-layer-first, OCR-fallback decision (choose_text) with a stub. Re-check versions when you build. :::

Architecture and milestones: three exits, eight proofs

The pipeline has one rule: nothing the model says reaches a downstream system until deterministic code has checked it.

The eight milestones, each with the test or eval that proves it. Commit after each with its test green.

Milestones 1 and 2 in prose. The schema (extractor/schema.py, 123 lines, not pasted) is a dataclass Invoice with a parse_invoice(dict) function that collects every problem into one ValidationFailure instead of raising on the first. Money must arrive as a JSON string such as "1250.00": a JSON number is a float in most parsers, and 0.1 + 0.2 is not 0.3, so a float is rejected with an instruction the model can follow. Optional fields (due_date, gstin) may be null; required ones may not. An unknown key is an error, which is how a hallucinated field gets caught. The generator (extractor/synth.py) builds invoices from a per-document seeded random.Random, computes the totals with Decimal and half-up rounding, and for the GST layout builds a real GSTIN by computing its check character.

The GSTIN check. A GSTIN is 15 characters; the last is a check character computed over the first 14 by a mod-36 weighted sum. Test it with a known-good value (27AAPFU0939F1ZV passes in TestCrosschecks) and a one-character change (fails). This is the kind of deterministic check that stops a model inventing a plausible tax number, a classic pitfall: letting the model fill a missing field instead of returning null.

Milestone 3, text. extractor/pdf.py has layer_pages(path) (pdfplumber, not run here) and choose_text(pages, ocr_page, min_chars): keep a page's text layer when it has at least min_chars non-space characters, else call OCR for that page. The decision is cheap to test and the expensive part (Tesseract or a vision model) is injected.

Milestones 4 and 5: the repair loop and the checks the model cannot talk past

The repair loop is a conversation with an auditor: the reply is checked, and if it fails, the exact error text goes back. Read extract() and check() first.

# -*- coding: utf-8 -*-
"""extractor.extract: ask, validate, and feed the exact error text back (at most twice)."""
import json
import re
from dataclasses import dataclass, field
from typing import List, Optional

from llmkit import LLMClient

from .schema import Invoice, ValidationFailure, parse_invoice
from .validate import crosschecks

SYSTEM_A = ("You extract invoices into JSON. Reply with one JSON object and nothing else. Keys: vendor, "
            "invoice_no, issue_date (YYYY-MM-DD), due_date, currency (INR, USD, EUR or GBP), gstin, "
            "line_items [{description, quantity, unit_price, amount}], subtotal, tax, total. "
            "Money is a JSON string like \"1250.00\". Use null for anything the document does not "
            "state. Never guess.")
SYSTEM_B = SYSTEM_A + " Read the document again from scratch and answer independently."


@dataclass
class Extraction:
    invoice: Optional[Invoice]
    status: str                      # "ok" | "repaired" | "failed"
    repairs: int = 0
    errors: List[str] = field(default_factory=list)   # errors of the LAST failed attempt
    raw: str = ""


def parse_json(text: str) -> dict:
    t = re.sub(r"^```(?:json)?|```$", "", text.strip(), flags=re.M).strip()
    a, b = t.find("{"), t.rfind("}")
    if a < 0 or b < a:
        raise ValidationFailure(["no JSON object found in the reply"])
    try:
        return json.loads(t[a:b + 1])
    except json.JSONDecodeError as e:
        raise ValidationFailure(["invalid JSON: %s" % e])


def check(text: str, source: Optional[str] = None) -> Invoice:
    inv = parse_invoice(parse_json(text))
    errs = crosschecks(inv, source)
    if errs:
        raise ValidationFailure(errs)
    return inv


def extract(llm: LLMClient, doc_text: str, max_repairs: int = 2, system: str = SYSTEM_A,
            ground: bool = False) -> Extraction:
    msgs = [{"role": "system", "content": system}, {"role": "user", "content": doc_text}]
    errors: List[str] = []
    raw = ""
    for attempt in range(max_repairs + 1):
        raw = llm.complete(msgs, temperature=0, max_tokens=700).text
        try:
            return Extraction(check(raw, doc_text if ground else None), "ok" if attempt == 0 else "repaired", attempt, [], raw)
        except ValidationFailure as e:
            errors = e.errors
            msgs = msgs + [{"role": "assistant", "content": raw},
                           {"role": "user", "content": "Your JSON failed validation:\n- "
                            + "\n- ".join(errors) + "\nReturn the corrected JSON only."}]
    return Extraction(None, "failed", max_repairs, errors, raw)

Three decisions are encoded there. The system prompt tells the model to use null for anything the document does not state and never to guess; without that sentence a model fills the gap. parse_json tolerates a code fence or a sentence around the JSON, because real models add them; a reply with no JSON is itself a repairable validation error. And the loop appends the model's own reply plus the validator's message, so the second attempt sees what it said and why it was rejected. The cap of two repairs is a budget decision: a document that fails three times usually has a problem the model cannot fix (an unreadable scan, a contradiction in the document), and a fourth attempt costs money for no information.

See it work. Document doc-0000, with a scripted model whose first reply has a total that is $10.00 too high:

MEGHNA FOODS
Invoice
Invoice No: INV-1000
Date: 15 Mar 2026
Due: 29 Apr 2026
packing tape  qty 12 @ $413.56 = $4962.72
hosting (month)  qty 4 @ $160.64 = $642.56
pallet freight  qty 20 @ $577.50 = $11550.00
Subtotal: $17155.28
Tax: $857.76
TOTAL DUE: $18013.04
---
status: repaired repairs: 1 calls: 2
fed back to the model:
Your JSON failed validation:
- subtotal 17155.28 + tax 857.76 = 18013.04 but total is 18023.04
Return the corrected JSON only.

The validator message names the three numbers involved. A vague 'the JSON is invalid' would give the model nothing to act on; this one lets it recompute.

The cross-checks are plain arithmetic and plain text matching.

# -*- coding: utf-8 -*-
"""extractor.validate: deterministic cross-checks the model cannot talk its way past."""
import re
from datetime import date
from decimal import Decimal
from typing import List, Optional

from .schema import Invoice

TOL = Decimal("0.01")
GSTIN_RE = re.compile(r"^\d{2}[A-Z]{5}\d{4}[A-Z][1-9A-Z]Z[0-9A-Z]$")
_ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"


def gstin_check_char(first14: str) -> str:
    """GSTIN checksum: mod-36 weighted digit sum over the first 14 characters."""
    total = 0
    for i, ch in enumerate(first14):
        p = _ALPHABET.index(ch) * (1 if i % 2 == 0 else 2)
        total += p // 36 + p % 36
    return _ALPHABET[(36 - total % 36) % 36]


def gstin_ok(g: str) -> bool:
    return bool(GSTIN_RE.match(g)) and gstin_check_char(g[:14]) == g[14]


SYMBOL = {"INR": "Rs ", "USD": "$", "EUR": "EUR ", "GBP": "GBP "}


def grounding_errors(inv: Invoice, source: str) -> List[str]:
    """Evidence checks against the document text itself."""
    lines = {" ".join(l.casefold().split()) for l in source.splitlines()}
    errs = []
    if " ".join(inv.vendor.casefold().split()) not in lines:
        errs.append("vendor %r is not a line of the document" % inv.vendor)
    if inv.invoice_no not in source:
        errs.append("invoice_no %r does not appear in the document" % inv.invoice_no)
    if SYMBOL[inv.currency] not in source:
        errs.append("currency %s: its marker %r never appears in the document" % (inv.currency, SYMBOL[inv.currency].strip()))
    return errs


def crosschecks(inv: Invoice, source: Optional[str] = None) -> List[str]:
    errs = []
    for i, it in enumerate(inv.line_items):
        if abs(it.quantity * it.unit_price - it.amount) > TOL:
            errs.append("line_items[%d]: quantity x unit_price = %s but amount is %s"
                        % (i, (it.quantity * it.unit_price).quantize(TOL), it.amount))
    s = sum((i.amount for i in inv.line_items), Decimal("0"))
    if abs(s - inv.subtotal) > TOL:
        errs.append("line item amounts sum to %s but subtotal is %s" % (s, inv.subtotal))
    if abs(inv.subtotal + inv.tax - inv.total) > TOL:
        errs.append("subtotal %s + tax %s = %s but total is %s"
                    % (inv.subtotal, inv.tax, inv.subtotal + inv.tax, inv.total))
    if inv.due_date and inv.due_date < inv.issue_date:
        errs.append("due_date %s is before issue_date %s" % (inv.due_date, inv.issue_date))
    if not (date(2015, 1, 1) <= inv.issue_date <= date(2035, 12, 31)):
        errs.append("issue_date %s is outside 2015-2035" % inv.issue_date)
    if inv.gstin is not None and not gstin_ok(inv.gstin):
        errs.append("gstin %r fails the format or checksum test; use null if the document has none" % inv.gstin)
    if source is not None:
        errs += grounding_errors(inv, source)
    return errs

Notes. Tolerance is one paisa (0.01), so a one-cent rounding difference passes while two fail (both cases are in the table-driven test). The line check multiplies quantity by unit price, so a model that copies the right amount but misreads a quantity is caught. The gstin_check_char function implements the mod-36 checksum. The source argument switches on grounding_errors, the evidence checks: the vendor must equal a whole line of the document (not a substring, because a truncated name such as 'Sharma Textile' is a substring of 'SHARMA TEXTILES'), the invoice number must appear in the text, and the currency's marker (Rs , $, EUR , GBP ) must appear. Grounding checks are what you add after error analysis shows that arithmetic alone lets consistent misreads through; the next sections show that happening.

Why checks beat asking the model to double-check. A model asked 'are you sure?' tends to agree with itself. A deterministic check has no opinion to defend, costs nothing in tokens, and gives a message specific enough to drive the repair. The structured-output lesson makes the same point about 'guaranteed' schemas: the guarantee is about syntax, not truth.

Milestone 6: confidence and routing

Auto-accept only when the checks pass and an independent second read agrees on the fields that matter. Self-reported model confidence is a weak signal on its own, so the router uses evidence: did validation pass, how many repairs did it take, and did two reads agree.

# -*- coding: utf-8 -*-
"""extractor.route: auto-accept only when the checks pass AND an independent second read agrees."""
from dataclasses import dataclass
from typing import List, Optional

from llmkit import LLMClient

from .extract import SYSTEM_B, Extraction, extract
from .schema import Invoice

KEY_FIELDS = ("vendor", "issue_date", "total", "currency")


@dataclass
class Decision:
    route: str            # "auto" | "review"
    score: float
    reasons: List[str]


def disagreements(a: Invoice, b: Invoice) -> List[str]:
    return [f for f in KEY_FIELDS if getattr(a, f) != getattr(b, f)]


def score_of(first: Extraction, second: Optional[Extraction]) -> Decision:
    """Truth table. 1.0 clean+agree, 0.8 repaired+agree, 0.5 valid but the reads disagree, 0.0 failed."""
    if first.invoice is None:
        return Decision("review", 0.0, ["validation failed after %d repairs" % first.repairs])
    if second is None or second.invoice is None:
        return Decision("review", 0.5, ["second read unavailable or failed validation"])
    diff = disagreements(first.invoice, second.invoice)
    if diff:
        return Decision("review", 0.5, ["two reads disagree on " + ", ".join(diff)])
    s = 1.0 if first.status == "ok" and second.status == "ok" else 0.8
    return Decision("auto", s, [] if s == 1.0 else ["accepted after repair"])


def process(llm: LLMClient, text: str, threshold: float = 0.8, ground: bool = True):
    first = extract(llm, text, ground=ground)
    second = extract(llm, text, system=SYSTEM_B, ground=ground) if first.invoice else None
    d = score_of(first, second)
    if d.route == "auto" and d.score < threshold:
        d = Decision("review", d.score, d.reasons + ["score below threshold %.1f" % threshold])
    return first, d

The score is a small truth table, not a probability. 1.0: both reads clean and in agreement. 0.8: valid and in agreement but at least one read needed a repair. 0.5: both valid but the reads disagree on a key field (vendor, issue date, total, currency), or the second read failed validation. 0.0: the first read failed after two repairs. Only 1.0 and 0.8 can auto-accept, and the threshold (default 0.8) is a parameter. The second read uses SYSTEM_B, which asks the model to read the document again from scratch and answer independently. In a real deployment you would also vary temperature or use a different model for the second read; two identical greedy calls are not independent.

Why only the key fields are compared. The second read costs a full call. Requiring it to agree on everything would send nearly every document with a harmless wording difference in a line description to a person. The fields chosen are the ones whose error causes a wrong payment or a wrong ledger. Whether that choice is right for your business is a decision, and it is also a hole: in the results below, an error in a field the router does not compare gets through, and the report shows exactly that.

Cost of the second read. The second pass runs only when the first produced a valid invoice, so a document that fails outright costs three calls and a clean one costs two.

The eval harness: a simulated model, a golden set and what the numbers mean

A scripted model tests the pipeline, never the model. To exercise the repair loop, the checks and the router deterministically, extractor/sim.py defines a stand-in that knows each document's truth and injects named faults. Read it, then read what it can and cannot tell you.

# -*- coding: utf-8 -*-
"""extractor.sim: a SCRIPTED stand-in for a model. It knows the truth and injects named faults.

Everything measured with it tests the PIPELINE (validation, repair, routing, metrics),
never a real model. The fault mix below is my assumption, not a measurement."""
import copy
import json
import random
from decimal import Decimal
from typing import Dict, List

from .synth import SynthDoc

# (fault, probability). Remaining probability mass is a clean read.
FAULTS = [("bad_total", .12), ("bad_total_sticky", .04), ("missing_date", .05), ("float_money", .04),
          ("hallucinated_gstin", .04), ("silent_vendor", .05), ("silent_currency", .03), ("silent_item", .03),
          ("unstable_date", .05)]
BAD_GSTIN = "29ABCDE1234F1Z5"   # right shape, wrong checksum (asserted in tests)


def _swap_day_month(iso: str) -> str:
    y, m, d = iso.split("-")
    return iso if int(d) > 12 else "%s-%s-%s" % (y, d, m)


def fault_for(doc: SynthDoc, seed: int) -> str:
    r = random.Random("fault:%d:%s" % (seed, doc.doc_id)).random()
    acc = 0.0
    for name, p in FAULTS:
        acc += p
        if r < acc:
            if name == "hallucinated_gstin" and doc.truth["gstin"]:
                return "clean"
            if name == "silent_currency" and doc.truth["currency"] != "INR":
                return "clean"
            if name == "unstable_date" and _swap_day_month(doc.truth["issue_date"]) == doc.truth["issue_date"]:
                return "clean"
            return name
    return "clean"


def reply_for(doc: SynthDoc, fault: str, attempt: int, second_pass: bool) -> str:
    t = copy.deepcopy(doc.truth)
    if fault == "bad_total" and attempt == 0:
        t["total"] = str(Decimal(t["total"]) + 10)
    elif fault == "bad_total_sticky":
        t["total"] = str(Decimal(t["total"]) + 10)
    elif fault == "missing_date" and attempt == 0:
        t["issue_date"] = None
    elif fault == "float_money" and attempt == 0:
        t["total"] = float(t["total"])
    elif fault == "hallucinated_gstin" and attempt == 0:
        t["gstin"] = BAD_GSTIN
    elif fault == "silent_vendor":
        t["vendor"] = t["vendor"][:-1]                       # same misread on both reads
    elif fault == "silent_currency":
        t["currency"] = "USD"                                # arithmetic still adds up
    elif fault == "silent_item":
        t["line_items"][0]["description"] += "s"             # no check looks at descriptions
    elif fault == "unstable_date" and not second_pass:
        t["issue_date"] = _swap_day_month(t["issue_date"])
    return json.dumps(t)


class SimModel:
    """Use as FakeLLM(default=SimModel(corpus)). Finds the document, counts repair turns."""

    def __init__(self, corpus: List[SynthDoc], seed: int = 7):
        self.by_no: Dict[str, SynthDoc] = {d.truth["invoice_no"]: d for d in corpus}
        self.faults = {d.doc_id: fault_for(d, seed) for d in corpus}

    def __call__(self, messages) -> str:
        text = messages[1]["content"]
        doc = next(d for no, d in self.by_no.items() if no in text)
        attempt = sum(1 for m in messages if m["role"] == "assistant")
        second = "again from scratch" in messages[0]["content"]
        return reply_for(doc, self.faults[doc.doc_id], attempt, second)

The fault mix is my assumption, not a measurement of any real model: 12 percent bad totals that a repair fixes, 4 percent that never get fixed, 5 percent missing dates, 4 percent float money, 4 percent invented GSTINs, 5 percent a vendor name misread the same way on both reads, 3 percent a currency misread, 3 percent a line-item description misread, and 5 percent a day and month swap on the first read only. The remaining mass is a clean read. With 120 documents and seed 7 the draw gave 78 clean documents and the fault counts in the table below.

The metrics, verified by hand. extractor/metrics.py scores each field by normalised exact match (case and whitespace folded, money compared as Decimal to the paisa) and the line items by set F1 on description plus amount.

# -*- coding: utf-8 -*-
"""extractor.metrics: field-level scoring. Verified by hand in tests/test_metrics.py."""
from collections import Counter
from decimal import Decimal, InvalidOperation
from typing import Dict, List, Optional

from llmkit import wilson

FIELDS = ("vendor", "invoice_no", "issue_date", "due_date", "currency", "subtotal", "tax", "total", "gstin")
MONEY = ("subtotal", "tax", "total")


def norm(field: str, v):
    if v is None:
        return None
    if field in MONEY:
        try:
            return Decimal(str(v)).quantize(Decimal("0.01"))
        except InvalidOperation:
            return "?"
    return " ".join(str(v).casefold().split())


def item_keys(d: Optional[dict]) -> Counter:
    return Counter((" ".join(i["description"].casefold().split()), norm("total", i["amount"]))
                   for i in (d or {}).get("line_items", []))


def line_item_f1(pred: Optional[dict], truth: dict) -> float:
    p, t = item_keys(pred), item_keys(truth)
    hit = sum((p & t).values())
    if hit == 0:
        return 1.0 if not p and not t else 0.0
    prec, rec = hit / sum(p.values()), hit / sum(t.values())
    return 2 * prec * rec / (prec + rec)


def score_doc(pred: Optional[dict], truth: dict) -> Dict[str, float]:
    s = {f: float(pred is not None and norm(f, pred.get(f)) == norm(f, truth.get(f))) for f in FIELDS}
    s["line_items_f1"] = line_item_f1(pred, truth)
    s["doc_correct"] = float(all(s[f] == 1.0 for f in FIELDS) and s["line_items_f1"] == 1.0)
    return s


def field_table(scores: List[Dict[str, float]]) -> List[str]:
    rows = ["| field | correct | n | 95% Wilson |", "|---|---|---|---|"]
    for f in FIELDS + ("doc_correct",):
        k = int(sum(s[f] for s in scores))
        p, lo, hi = wilson(k, len(scores))
        rows.append("| %s | %.3f | %d | %.2f-%.2f |" % (f, p, len(scores), lo, hi))
    f1 = sum(s["line_items_f1"] for s in scores) / len(scores)
    rows.append("| line_items (mean F1) | %.3f | %d | n/a |" % (f1, len(scores)))
    return rows

test_metrics checks it against a 5-document example worked on paper: documents 1 and 3 fully correct (document 3 only after normalising case), document 2 with a wrong total, document 4 failed outright, document 5 with a wrong date, giving 2 fully correct documents, 3 correct totals, 4 correct vendors, 3 correct issue dates and line-item F1 summing to 4.0. A metric you have not checked against a hand count is a bug waiting to look like a result.

The run. The printed modules are the decision-bearing ones; schema.py, synth.py, pdf.py, trace.py, api.py, the tests and evals/run.py are plain plumbing that I do not print, so the exact figures below come from my full repository and your own generator will draw different documents (expect similar shape, not identical counts). python evals/run.py generates 120 documents (seed 7), runs a baseline (one prompt, trust the reply), pipeline v1 (schema, arithmetic, repairs and second read) and pipeline v2 (v1 plus the evidence checks), and prints everything below. The last 20 documents are marked held out.

## Baseline (one prompt, trust the reply) vs pipeline, n=120 simulated documents
| system | doc fully correct | auto-accept rate | wrong among auto-accepted | review rate |
|---|---|---|---|---|
| baseline | 0.675 (0.59-0.75) | 1.000 | 0.325 | 0.000 |
| pipeline v1: schema+arithmetic+repair+2nd read | 0.867 (0.79-0.92) | 0.950 | 0.088 (10 of 114; 0.048-0.154) | 0.050 |
| pipeline v2: v1 + grounding checks | 0.867 (0.79-0.92) | 0.900 | 0.037 (4 of 108; 0.014-0.091) | 0.100 |

Paired, baseline -> v2 on doc_correct: mean diff +0.192, 95% bootstrap CI +0.125..+0.267, v2 wins 23, baseline wins 0, ties 97, sign-test p=2.4e-07

## Where each fault ends up (v1 | v2): count, auto-accepted wrong, sent to review
| fault | n | v1 auto-wrong | v1 review | v2 auto-wrong | v2 review |
|---|---|---|---|---|---|
| bad_total | 12 | 0 | 0 | 0 | 0 |
| bad_total_sticky | 4 | 0 | 4 | 0 | 4 |
| clean | 78 | 0 | 0 | 0 | 0 |
| float_money | 3 | 0 | 0 | 0 | 0 |
| hallucinated_gstin | 1 | 0 | 0 | 0 | 0 |
| missing_date | 10 | 0 | 0 | 0 | 0 |
| silent_currency | 1 | 1 | 0 | 0 | 1 |
| silent_item | 4 | 4 | 0 | 4 | 0 |
| silent_vendor | 5 | 5 | 0 | 0 | 5 |
| unstable_date | 2 | 0 | 2 | 0 | 2 |

## Coverage vs accuracy as the score threshold moves (v2, all 120 docs)
| accept if score >= | coverage | wrong among accepted |
|---|---|---|
| 1.0 | 0.683 | 0.049 |
| 0.8 | 0.900 | 0.037 |
| 0.5 | 0.900 | 0.037 |
| 0.0 | 1.000 | 0.133 |

Held-out last 20 docs, v2: auto-accept 18, wrong among auto 0, review 2

## Per-field accuracy, v2 pipeline output (all docs, n=120)
| field | correct | n | 95% Wilson |
|---|---|---|---|
| vendor | 0.900 | 120 | 0.83-0.94 |
| invoice_no | 0.900 | 120 | 0.83-0.94 |
| issue_date | 0.900 | 120 | 0.83-0.94 |
| due_date | 0.900 | 120 | 0.83-0.94 |
| currency | 0.900 | 120 | 0.83-0.94 |
| subtotal | 0.900 | 120 | 0.83-0.94 |
| tax | 0.900 | 120 | 0.83-0.94 |
| total | 0.900 | 120 | 0.83-0.94 |
| gstin | 0.900 | 120 | 0.83-0.94 |
| doc_correct | 0.867 | 120 | 0.79-0.92 |
| line_items (mean F1) | 0.885 | 120 | n/a |

## Cost (token counts are ~4-chars-per-token estimates; prices ASSUMED $0.15/$0.60 per 1M)
tokens per doc: 579 in + 327 out; $0.000283 per doc; $0.283 per 1,000 docs; repair rate 0.317

GATE: PASS {'doc_correct': 0.8667, 'auto_error': 0.037, 'tokens_per_doc': 906}

How to read it, in order.

• The baseline is wrong a third of the time and says nothing. Of 120 documents, 0.675 are fully correct (Wilson 95 percent interval 0.59 to 0.75). Because it trusts every reply, every wrong document is auto-accepted: 32.5 percent error among accepted.
• v1 fixes the loud errors. Bad totals, missing dates, float money and the invented GSTIN all get repaired (the fault table shows zero auto-accepted errors for them). Of 114 auto-accepted documents, 10 are wrong, 8.8 percent (interval 4.8 to 15.4 percent). Look at which: all five misread vendors, the one misread currency and four line-item descriptions. Every one passes every check because the numbers are consistent and both reads repeated the same mistake.
• v2 closes two of those holes. Evidence checks send the five vendor errors and the currency error to review, so auto-accepted errors drop to 4 of 108, 3.7 percent (interval 1.4 to 9.1 percent). The four that remain are the line-item description errors, and no check in the pipeline looks at descriptions. That is the named blind spot. Review rate rose from 5 percent to 10 percent: the price of catching them.
• Paired baseline versus v2. Mean improvement in document-correct is 0.192, bootstrap interval 0.125 to 0.267; v2 wins on 23 documents, the baseline on none, 97 ties; sign-test p about 2.4e-07. The improvement is real in this simulation.
• Note that document-correct is identical for v1 and v2 (0.867). v2 did not make the output more correct, it changed which wrong outputs were accepted without a person. That is why the headline is error among auto-accepted, not accuracy.
• Coverage versus accuracy. Accepting everything (score at least 0.0) gives 100 percent coverage and 13.3 percent error; accepting only 1.0 gives 68.3 percent coverage and 4.9 percent error; 0.8 gives 90 percent and 3.7 percent. The 0.5 row equals the 0.8 row because no document in this run scored 0.5: score counts were 82 at 1.0, 26 at 0.8 and 12 at 0.0 (the day-and-month swaps produced a due date before the issue date, so they failed validation outright rather than reaching a disagreement; the disagreement path is covered by the router test instead). The curve is coarse because the score takes only four values.
• Held-out slice. 18 of the last 20 were auto-accepted with no wrong ones, 2 went to review. Treat that with suspicion: with 18 documents the interval on 0 errors still reaches about 18 percent. Also, I added the evidence checks after reading v1's fault table, which includes those 20 documents, so in this demonstration the held-out label is only nominal. In your own run, look at failures on the dev slice only and score the held-out slice once.

What this does not prove. The simulated model makes mistakes of types I chose, at rates I chose. These numbers show that the pipeline catches the faults it was designed to catch and misses the one it has no check for. They are not a claim about how often a real model misreads an invoice, which you measure with a live run (a real client through OpenAICompatClient) on your own documents, and the comparison above is illustrative of method.

The error-analysis round, the CI gate and the blind spot

The loop that earns the before-and-after row: read the failures, name the largest class, fix it, rerun. In v1, the fault table shows where the 10 auto-accepted errors came from: 5 misread vendors, 1 misread currency, 4 line-item descriptions. The first two share a cause (arithmetic is blind to text that is internally consistent) and a cheap deterministic remedy (check the text against the source), so that became v2. The third class has a remedy too (ground each description against the document), and that is the natural next round and a stretch goal; I left it in on purpose so the report names an honest remaining weakness.

The gate. evals/baseline.json stores the last accepted numbers (document-correct 0.8667, auto-accepted error 0.037, 906 tokens per document). run.py exits non-zero when document-correct falls by more than 0.02 from that file, when auto-accepted error exceeds the larger of 1 percent and the stored baseline, or when tokens per document rise more than 20 percent. The gate is a regression gate: it does not say 3.7 percent is acceptable (the product target of 1 percent is not met), it says the pipeline got no worse than the accepted version. Simulating someone removing the evidence checks (python evals/run.py --ungrounded) produces:

GATE: FAIL - auto-accepted error 0.088 exceeds max(1%, baseline 0.037) {'doc_correct': 0.8667, 'auto_error': 0.0877, 'tokens_per_doc': 906}

and exit code 1. Put python evals/run.py in your CI workflow after python -m unittest discover -s tests, and take a screenshot of that failure for the README, because it proves the gate works.

How to protect the golden set. Seeded synthetic documents give known truth; add 30 realistic hand-labelled ones (public sample invoices, anonymised) so the layouts are not all ones your generator wrote, otherwise you evaluate on the generator that also shaped your prompt examples. Stratify by layout and difficulty, hold out 20 you never look at while tuning, and add corrections from the review UI as new cases. Report n and an interval on every number.

Failure analysis for the README. Three failures, each with cause, fix and metric: (1) bad totals that a repair fixes, with the exact validator message; (2) vendors misread consistently, which passed arithmetic and both reads, fixed by a whole-line evidence check (auto-accepted error 8.8 to 3.7 percent in this simulation); (3) line-item descriptions, still open, with the proposed check.

Cost, latency, observability and deployment

Budget: about 900 tokens and a fraction of a cent per document in this fixture; prices are assumptions. The offline run measured, per document, 579 input and 327 output tokens (four-characters-per-token estimates), a repair rate of 31.7 percent, and a cost of $0.000283 per document or $0.283 per 1,000 documents at the assumed prices of $0.15 and $0.60 per million input and output tokens. Real invoices are longer than my synthetic ones; as a rough planning figure (an assumption, not a measurement) take about 2,000 input and 500 output tokens for a one-to-two page invoice, times about 1.4 with repairs and the second pass. A test asserts that the worst document in a 40-document run stays under a token budget, so a prompt change that doubles the cost fails CI. Check current prices on your provider's page and date them in the README.

Latency. The offline latency is simulated and must not be quoted. As written, the second read waits for the first to validate, so a clean document costs two serial model calls; run several documents concurrently to hide it. Measure p50 and p95 on your real model and put the model name and date next to them.

Observability. extractor/trace.py writes one JSONL line per document with the route, repairs, reasons and dollars, and rollup() prints review rate, repair rate, cost per document and the top three reasons. The review rate by layout is your drift alarm: if a new supplier layout appears, its review rate jumps before anyone notices wrong data.

The rollup returns docs, review_rate, repair_rate, usd_per_doc and top_reasons, and TestPdfTrace checks it against hand values.

The API. extractor/api.py is a WSGI app (standard library) with POST /extract (document text in, route and data out), GET /review (an HTML table of documents in the queue), POST /correct (stores a field correction), GET /health, and a size limit that returns 413. TestApi calls it directly through the WSGI interface, with no server, and walks the whole flow. Swapping in FastAPI is mechanical: keep process() and the store, change the transport. Corrections land in a corrections table; export them as new golden cases.

Deployment. A slim Python container with the WSGI app behind a production server, secrets in environment variables, uploads discarded within an hour, and the fake client as the default so the public demo cannot spend money; use the real model only behind a daily spend cap (CostMeter(limit_usd=...)).

:::key Verified 24 October 2026 Render's free-instances page: a free web service spins down after 15 minutes of no inbound traffic and takes about a minute to start again; no persistent disk; 750 free instance hours a month per workspace. So keep the SQLite queue on ephemeral disk for the demo only, and say so in the README. Hugging Face: a Docker Space needs a paid plan for a new account (Spaces overview, same date). Check current limits before you choose. :::

Present it: README numbers, interview questions, pitch and resume bullet

The README results block, filled with what this repository's offline run printed. Replace the model-dependent rows with your own measured values and date them.

Five interview questions.

@@ 1 | How do you know the extracted total is right? Never trust the model: Decimal money, arithmetic cross-checks (items to subtotal, subtotal plus tax to total), an independent second read on the key fields, and a measured error rate among auto-accepted documents. Then name the case where everything passes and the value is wrong (a consistently misread currency or description) and say how a sample audit of auto-accepted documents catches it.

@@ 2 | Why a repair loop, and what are its limits? Feed back the exact validator message, cap at two, and measure repair success (here 31.7 percent of documents needed one). Repeated failure signals an input or prompt problem, not noise. A schema guarantee means syntax, not truth.

@@ 3 | OCR plus an LLM, or a vision model directly? Text layer first (cheap and exact), OCR for scans, a vision model for layout-heavy pages; decide by comparing cost and field accuracy on your own golden set. Document Understanding explains why the hybrid usually wins.

@@ 4 | How did you build and protect the golden set? Seeded synthetic documents for known truth plus hand-labelled realistic ones, stratified, a held-out slice scored once, review corrections fed back as new cases, n and intervals on every number. Admit what I did not protect: my held-out slice was nominal because I read the whole fault table.

@@ 5 | What changes at 100 times the volume or on a new layout? Queueing and batch concurrency, review rate per layout as a drift alarm, a canary prompt on shadow traffic, caching the static prompt prefix, and fine-tuning a small model once there is labelled data.

ILLUSTRATIVE resume bullet (swap in your own measured numbers): Built an invoice-extraction pipeline (Python, schema validation, repair loop, second-read routing) with a field-level scorecard; cut auto-accepted error from 32.5% to 3.7% on a 120-document simulated golden set (95% interval 1.4-9.1%), about $0.28 per 1,000 documents at assumed prices; CI gate blocks regressions.

ILLUSTRATIVE 90-second pitch. Problem: finance teams key in invoices by hand. Decision that mattered: arithmetic checks plus an independent second read before auto-accept, because a plain prompt posted wrong records a third of the time in my simulation. Measurement: 120 documents, intervals on every number, a gate in CI. Failure: five misread vendors passed every arithmetic check; a whole-line evidence check sent them to review (8.8 to 3.7 percent). Next: ground line descriptions, then audit a sample of auto-accepted documents weekly.

Pitfalls. Evaluating on the same generator that wrote your prompt examples; float money; reporting accuracy over all documents instead of error among auto-accepted; letting the model fill missing fields instead of returning null; no held-out split; storing real customer data (use synthetic or anonymised documents only).

Stretch goals. Fine-tune a small model on corrected outputs and compare cost and accuracy; contract clause extraction with page and span citations; an active-learning loop that picks the next documents to label; Hindi and regional-language invoices; ground line-item descriptions.

Time estimate. As in the time-box above: 14, 22 and 34 hours.

How to say it in the interview

"I treat the model as an untrusted parser. The reply has to pass a schema with Decimal money, arithmetic cross-checks, and evidence checks against the document text; if it fails, the exact error goes back to the model, twice at most. A second independent read has to agree on vendor, date, total and currency before anything is auto-accepted. The number I report is not overall accuracy but the error rate among auto-accepted documents, with an interval, and I know the one class of error my checks cannot see."

Checklist

• Money is Decimal parsed from JSON strings, and a float in the reply is rejected with a message the model can act on
• Every validation message names the numbers involved and goes back to the model verbatim in the repair turn
• Repairs are capped (two) and a document that still fails is routed to a person, with the failure reason stored
• Missing fields come back as null, and a test shows an invented GSTIN is caught by the checksum
• Cross-checks have a table-driven test with rounding and one-paisa-tolerance cases
• Evidence checks compare extracted text against the source (whole-line vendor, invoice number, currency marker)
• Auto-accept requires validation to pass AND an independent second read to agree on the key fields
• The headline metric is the error rate among auto-accepted documents with an interval, not overall accuracy
• Metrics are verified against a hand-computed example before any number is reported
• The README states that the model was simulated, lists the fault mix as an assumption, and names the remaining blind spot
• A CI gate runs the offline eval, and I have a screenshot of it failing on a deliberate regression
• No real customer documents appear in the repo, the demo defaults to the fake client, and a daily spend cap protects any real-model path

Related lessons

• Build Projects That Get You Hired: The Process, the Shared Test Seam and How to Show It
• Project: A RAG Assistant Over Real Documents With an Eval Gate in CI
• Project: A Tracing and Cost Dashboard for LLM Apps (Build Your Own Mini-Langfuse)
• Project: An Eval Harness and LLM-as-Judge Toolkit You Can Trust
• Project: A Text-to-SQL Analyst With Read-Only Guardrails and Execution-Based Evals
• Project: A Tool-Using Agent With Guardrails, an Approval Gate and an Audit Log