Cruxly

Cruxly › AI engineering › Multimodal & Document Understanding

Document Understanding: OCR, Layout Models, VLMs and the Hybrid That Usually Wins

route by page, extract with citations and abstention, and price 100,000 pages a month at $1,120, $56 or about $109

Open the interactive lesson

Prompt: "A lender receives 100,000 pages a month: bank statements, invoices, salary slips and scanned contracts, some typed, some photographed, some handwritten. Extract a fixed set of fields into JSON, tell me which pages need a human, make the scanned contracts searchable, and tell me what it costs. Do we use OCR, a vision-language model, or both?"

40 minutes first. Price the volume at two model rates, then design a router that sends each page to the cheapest path that meets your accuracy bar. The trap is picking a tool and quoting its demo accuracy; the interviewer wants per-page routing, abstention, and a number for how many pages reach a human.

Three approaches and what each one costs you

There are three ways to turn a document page into fields, and they fail differently.

1. Classic OCR plus layout parsing. An OCR engine finds text and its positions, a layout model finds regions (paragraphs, tables, headings), and rules or a small model assign meaning. It is cheap per page, fast, deterministic, and every output has coordinates you can show a reviewer. It is brittle on layouts it has not seen: a new bank's statement template breaks the rules, and it reads a table as a bag of words unless a table model is added.

2. An end-to-end vision-language model. You send the page image and ask for JSON. It handles layout variety without templates, reads context ("Total" next to a number), and can follow a schema. It costs real tokens per page, is slower, can misread small digits with full confidence (the previous lesson, Vision-Language Models: How Images Become Tokens, What They Cost, and Where They Fail, covers why), and gives no coordinates unless you ask for them and verify them.

3. The hybrid. Run OCR (or extract the native text layer) and send both the extracted text and the page image to the model. The text gives exact characters, the image gives layout and context, and the model reconciles them. Anthropic's PDF support does this by default: it converts each page to an image and also extracts the page's text, and gives the model both.

:::key Verified October 2026 From Anthropic's PDF support documentation, read October 2026: each page is processed as both text and image; text costs roughly 1,500 to 3,000 tokens per page depending on density, and the image cost is calculated as for any image (the formula from the previous lesson), with no separate PDF fee. Limits are 32 MB per request and 600 pages per request (100 when the request's context window is under 1M tokens). Dense PDFs can fill the context window before the page limit, so split large documents. :::

Do the arithmetic for a sent PDF page. A text-dense page is about 1,500 to 3,000 text tokens plus about 1,560 image tokens (a 1456 x 819 equivalent), so roughly 3,060 to 4,560 tokens, about $0.006 to $0.009 per page of input at $2 per million. If you already extracted the text yourself, you can send just the image (about 1,560) or just the text (1,500 to 3,000), which is the lever the router below pulls.

Pick on four axes. Cost (OCR cheapest), accuracy on hard content (VLM or hybrid best), latency (OCR fastest), and debuggability (OCR gives coordinates and deterministic output; a VLM gives you a prompt to tune and an eval to run). The honest summary: OCR where layout is stable and volume is high, VLM where layout varies, the hybrid where an exact digit matters and the layout varies, and a router so you do not choose once for everything.

Route per page: native text, scan, or hard, and what 100,000 pages cost

Detect what the page is before choosing a model, because the cheapest correct path differs by page. Three checks in code decide the route.

@@ 1 | Does the page have a native text layer? Extract text with a PDF library and test it: enough characters per page, mostly dictionary words, text positions inside the page box. A scanned PDF often has none, or a poor embedded OCR layer full of garbage. Native-text pages need no image reading; you send the extracted text to a small model to structure it.

@@ 2 | If scanned, how good is the image? Resolution, skew, blur, contrast. A page photographed at an angle needs deskewing before anything reads it; a page below a quality threshold should skip straight to a human or a re-capture request, not burn tokens on guesses.

@@ 3 | Is the content hard? Dense tables, handwriting, stamps and overlapping text are routed to the stronger path, as the next section details.

Now price it. Assume 100,000 pages a month, and for an image page about 1,600 input tokens and 800 output tokens.

:::key Verified October 2026 Prices per million tokens (input / output) from Anthropic's pricing table, read October 2026: Claude Sonnet 5.5 $2 / $10; Claude Haiku 5.5 $0.10 / $0.50 for prompts up to 100K tokens (higher above that). Prices change often and vary by provider; recompute with the current table and with your own measured token counts before quoting a budget. :::

The 20x gap invites the obvious design. Route, and escalate only what the cheap path cannot handle. Suppose 60 percent of pages (60,000) have a native text layer, 40 percent (40,000) are scans, and 15 percent of scans fail validation on the first pass and escalate to the strong model. These splits are assumptions, and for simplicity the native pages never escalate (the diagram below lets them; add their escalations once you measure them).

Total $108.80 a month, about a tenth of the all-strong bill, and the strong model is spent where it is needed. The 15 percent is an assumption; it is the number to measure, because at 50 percent escalation the escalated line alone is $224 and the saving shrinks. Which pages escalate depends on the small model's accuracy on hard fields, not its average: if it is, say, 20 points behind on those fields (an assumed gap), those pages should go up.

The routing signal has to be something other than the model's own claim of confidence, which is poorly calibrated. Use validation failures (totals do not reconcile, a required field is null, a date is invalid), disagreement between two cheap reads (OCR text versus VLM), and a quality score from the image.

What breaks: tables, forms, handwriting, stamps, columns and rotation

Each hard content type has a characteristic failure, and the fix is usually a targeted step, not a bigger model.

A concrete illustration. A bank statement table has 40 rows, each with date, narration, debit, credit and balance. If each cell is read correctly 99.5 percent of the time, a row of 5 cells is right 0.995^5 = 97.5 percent of the time, and all 40 rows are right 0.975^40 = 36 percent of the time (37 percent without rounding the row figure). A table that looks almost perfect per cell is usually wrong somewhere, which is why the check that matters is arithmetic: opening balance plus credits minus debits must equal closing balance, and each row's running balance must follow from the last. One failed identity pinpoints the row to re-read.

:::warn Averages hide the pages that hurt Reporting "98 percent field accuracy" for a pipeline where handwritten cheque amounts are 80 percent accurate is the average doing its usual work. Slice accuracy by document type, field type and capture quality, and set the human-review rule on the slices that matter. :::

Structured extraction: schemas, citations and the right to say null

An extraction that cannot be checked or refused is a liability, so design the output contract before the prompt. Four properties make an extraction auditable.

A strict schema. Define fields, types and enums, and let a field be null. Tools for schema-constrained output exist (the schema and its quality trade-offs are in Cost Attribution and Runaway Prevention for constrained decoding, and the general pattern in the track's structured-output material), but the schema design matters more than the mechanism: separate amount_text (what the page says, "1,25,000.00") from amount_value (the number), because Indian digit grouping and currency symbols are a normalisation problem you handle in code.

A citation per field. Each returned field carries page, a short quote copied from the page, and optionally a bounding box. The quote is checkable: a script searches the OCR or native text for it. If the quote does not appear in the text of the cited page, the field is rejected without human involvement. That one rule catches a large share of hallucinations, because an invented value rarely comes with a verbatim quote that exists on the page. For boxes, the vision documentation says localisation outputs are approximate, so verify them against the OCR positions rather than trusting them.

Abstention. Tell the model that null with a reason (not_present, illegible, ambiguous) is the right answer when the page does not support a value, and measure how often it abstains. A model never asked to abstain fills the blank; an abstention rate of zero is a warning, not a success.

Validators in code. Format (PAN, IFSC, GSTIN patterns), arithmetic (line items sum to the subtotal; subtotal plus tax equals total), cross-field logic (invoice date not after due date), and cross-document logic (the employer on the salary slip matches the one on the bank credit). A validator failure is the escalation trigger from the routing section.

Per-field confidence is best built from signals, not from the model's self-report: did the quote verify, did two reads agree, did validators pass, was the image quality high? Combine them into a three-level status (accepted, needs second read, needs human) and calibrate the thresholds on labelled data until accepted fields meet your error target.

{
  "net_pay": {"value": 84250.00, "text": "84,250.00", "page": 1,
              "quote": "Net Pay  84,250.00", "status": "accepted"},
  "employer_gstin": {"value": null, "reason": "illegible", "page": 1,
                     "quote": null, "status": "needs_human"}
}

Searching scanned documents: parse-then-embed versus page-image retrieval

To search 2 million pages of scanned contracts you can index their text, their images, or both. The standard route is parse-then-embed: OCR the page, chunk the text (the chunking problems are in Chunking and Document Parsing), embed the chunks, and search. It is cheap and easy to debug, and it loses whatever the parser loses: a table flattened to a string, a chart dropped, a stamp ignored, layout meaning gone, and any OCR error becomes a retrieval miss.

Page-image retrieval skips parsing. ColPali (Faysse et al., arXiv 2407.01449, ColPali: Efficient Document Retrieval with Vision Language Models, ICLR 2025) embeds page images directly with a vision-language model, producing many vectors per page, and scores a query against a page by late interaction, the multi-vector method described in Hybrid Search, Rerankers and Late Interaction. The paper introduces the ViDoRe benchmark for page-level retrieval and reports that the approach substantially outperforms the text-extraction pipelines it compared against on it. Those are the authors' own results on their benchmark; treat them as evidence the idea works, and test it on your own scanned contracts.

Late interaction by hand (MaxSim). Take a query with two token vectors, q1 = [1, 0] and q2 = [0, 1], and a page with three patch vectors [0.9, 0.1], [0.2, 0.8], [0.5, 0.5]. For each query vector, take the maximum dot product over the page's patches: q1 gives max(0.9, 0.2, 0.5) = 0.9, q2 gives max(0.1, 0.8, 0.5) = 0.8. The page score is the sum, 0.9 + 0.8 = 1.7. Each query token finds its best-matching region, so a query about "interest rate table" can match the table region of the page without a text parser having understood the table.

The cost is storage and compute. Use illustrative settings, not the paper's exact ones: 1,000 vectors of 128 dimensions in float16 is 1,000 x 128 x 2 = 256,000 bytes a page. For 1 million pages that is 256 GB, against one 1,024-dimension float32 vector per page at 4,096 bytes, 4.1 GB. About 60 times the storage, which is why practitioners prune or compress the vectors, use the multi-vector search as a re-ranker over a cheaper first stage, or keep it for the high-value collection.

A pragmatic design for the contracts. Index parsed text with a keyword and dense hybrid for the bulk, add page-image retrieval where tables and layout carry the meaning, return the page image and the cited page to the user, and evaluate recall on queries that need layout ("the clause in the second table of the annexure"). Retrieval hands pages to extraction, so the cited-page rule from the previous section carries through.

Evaluating the pipeline: fields, cells, sampling, and cost per correct field

Measure field-level accuracy after normalisation, per slice, with a sampled human audit on production traffic. The metrics follow from the output contract.

Sizing the audit. For a simple random sample, and an error rate assumed to be near 2 percent, to estimate it to within plus or minus 1 point at 95 percent confidence you need about 1.96^2 x 0.02 x 0.98 / 0.01^2 = 753 pages. To get within 0.5 points you need 3,012. Sample from accepted pages (not only the hard ones) every week, since drift in the document mix is what silently breaks a pipeline; review budget for 753 pages at 2 minutes each is about 25 hours.

Cost per correct field is the number that settles routing arguments. Divide spend by the count of correct fields, not by pages. For 12 fields, with assumed accuracies of 98 percent for the strong model and 90 percent for the small one: the strong model at $0.0112 a page and 98 percent is $0.0112 / (12 x 0.98) = $0.00095 per correct field; the small model at $0.00056 and, say, 90 percent on the same fields is $0.00056 / (12 x 0.90) = $0.000052. The small model is about 18 times cheaper per correct field and, on these assumed figures, wrong eight points more often; whether that matters depends on what a wrong field costs downstream. A wrong loan amount that reaches a credit decision may cost thousands of rupees, so the extra error rate is paid for by review hours, which is why the router escalates validation failures instead of choosing one model.

A human review queue is part of the system. Send pages with status needs_human, capture the corrected value, and feed those corrections back as labelled data. Track reviewer disagreement too: if two reviewers disagree on 8 percent of handwritten amounts, no model will beat the label. (The throughput, SLA and queue design of this pipeline are the subject of the document-extraction case study later in the track.)

How to say it in the interview

"Neither alone. I would first detect whether a page has a native text layer, because for those pages I can extract text for free and only need the model to structure it. For scans I would run a cheap vision-language model on the page image with a strict schema, every field returned with its source page and a quoted snippet, and the right to answer null. Validation in code, such as totals reconciling and formats matching, plus a confidence signal decides which pages escalate to a stronger model or a human. At 100,000 pages and 1,600 tokens in and 800 out, a 2 and 10 dollar per million model costs about 1,120 dollars a month, a 10 and 50 cent one costs 56, and routing with 15 percent escalation lands near 110. For search I would index page images with a late-interaction retriever alongside parsed text, and I would measure field-level accuracy after normalisation on a labelled sample, plus the review rate."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons