Cruxly

Cruxly › AI engineering › Portfolio Projects

Project: A Tracing and Cost Dashboard for LLM Apps (Build Your Own Mini-Langfuse)

span trees, token and cost accounting you can verify to the cent, percentiles, budget stops, redaction and sampling, in a library with a measured overhead

Open the interactive lesson

Project: "Our RAG assistant works, but nobody can say what a question costs, which feature burns the money, why Tuesday's p95 doubled, or whether the agent that looped all night is running again. Build the telemetry layer: record every step of every request, price it, show it, stop runaway spend, and keep customer data out of the logs."

Time-box: 14 hours minimum, about 20 typical, about 30 with the stretch goals. You will have a tested package (tracekit, standard library plus llmkit, 34 passing tests): spans that nest across async and threads, an LLM wrapper recording tokens, retries, cost and time to first token, a dated price table, a SQLite store behind a non-blocking writer, percentile and cost-by-tag queries, budget and loop guards, redaction before write, a review sampler, a JSON API with a waterfall page, a 1,000-span accounting fixture, a benchmark gate, and the three lines that instrument the previous project's RAG assistant.

Why this project, scope, the success metric and what a reviewer will probe

Anyone can plug in a vendor dashboard; this project shows that you know what the dashboard is made of. Hiring-side guidance in the first lesson of this topic keeps naming the same differentiators: cost and latency discussed with numbers, evidence that a system is measured, and candour about where it broke. A tracing layer is where all three come from. Building a small one proves you understand trace trees, token accounting, cost attribution, tail latency, sampling and privacy, and it becomes the telemetry module that the gateway, agent and capstone projects in this topic reuse. It also earns a better answer to the build-versus-buy question, because you will have felt where the hard parts are.

Scope. A Python library that instruments any function or LLMClient; a store; a handful of queries; budget and loop guards; redaction and sampling; a read-only JSON API and one HTML page. Non-goals: a multi-tenant SaaS, a distributed collector, a query language, a polished front end, long-term retention policy. What a reviewer will probe: what you log and what you do not; how you attribute cost; how you stop a looping agent; why p95 and how to compute it cheaply; what you would adopt instead in production.

Success metric, set before building (assumed targets, mine to defend). Accounting must match a hand-computed fixture exactly: total cost, p50, p95 and p99 latency, error rate. Tracer overhead must stay at or below 200 microseconds per span and under 1 percent of a 2-second LLM call. A caller must never be blocked by the writer, and a drop must be counted, not silent. A looping agent must be stopped inside its budget. No test may need a network or a key, and the CI gate fails on any accounting deviation or an overhead regression.

Builds on. Observability and Cost Governance for LLM Systems, Cost Attribution and Runaway Prevention, TTFT, TPOT, and the Latency Budget, Prompt Caching: Paying for the Same Tokens Once and Streaming Interfaces That Show Their Work. The seam (LLMClient, FakeLLM, BudgetExceeded) comes from llmkit; this project wraps it and does not redefine it.

What the offline tests prove. Accounting and plumbing only. Every price here is an assumed round number, and the latency of a FakeLLM call is the speed of Python, not of a model.

:::warn What this lesson prints, and what it does not tracekit is eight modules plus tests, the fixture, the benchmark and a web page, roughly 1,200 lines of library code. This lesson prints the core of each idea (tracer, wrapper, cost function, writer, guards, redaction, key tests) and describes the rest by the test that proves it. It does not print pricing.py (the PriceTable class around cost(): load, age_days, is_stale), the SQLite schema and row_of, queries.py, api.py, evals/fixture.py, evals/bench.py, web/index.html or most tests, so you write those from the milestone text and the numbers below. The package __init__.py is short and exact names matter, so here it is, with the price file the examples load:

# -*- coding: utf-8 -*-
from .trace import Tracer, Span, tags, current_span, in_context, fanout  # noqa: F401
from .pricing import PriceTable  # noqa: F401
from .llm import TracedLLM, traced_stream  # noqa: F401
from .store import Writer, connect  # noqa: F401
from .guard import BudgetGuard, GuardedLLM, LoopDetector, LoopDetected  # noqa: F401
from .privacy import make_transform, redact, ReviewSampler  # noqa: F401
{
  "as_of": "2026-10-24",
  "source": "ASSUMED round numbers for teaching. Replace with the provider's pricing page on the day you measure.",
  "usd_per_million_tokens": {
    "small":  {"input": 0.15, "output": 0.60, "cached_input": 0.015},
    "large":  {"input": 2.50, "output": 10.00, "cached_input": 0.25},
    "fake-1": {"input": 0.15, "output": 0.60, "cached_input": 0.015}
  }
}

fanout (used in the RAG example) is a small helper in trace.py that returns a sink calling several sinks; ReviewSampler is a callable sink, as described under milestone 7. The llmkit package is the one from the first project of this topic. The numbers printed below are what my complete copy produced; your own build will match the deterministic ones (costs, percentiles, counts) and not the timings. :::

Architecture and what the library records

One span is one row, and the row is designed around the questions you will ask. A span has a trace id (shared by the whole request), a span id, a parent id (empty for the root), a name, a kind (llm, retrieval, tool or chain), a wall-clock start, a duration, a status, the exception type if it failed, free-form tags, and attributes. For LLM spans the attributes follow the OpenTelemetry names, and the store promotes the ones you will aggregate (model, input tokens, output tokens, cached tokens, cost, whether the cost was known) into real columns so a query is one GROUP BY, not a scan of JSON.

:::key Verified 24 October 2026 The OpenTelemetry GenAI semantic conventions now live in their own repository, open-telemetry/semantic-conventions-genai. Its span document is marked Status: Development, so names can still change. It says the client span name SHOULD be {gen_ai.operation.name} {gen_ai.request.model}; the attributes this project uses are gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens; input and output messages are Opt-In. Two details matter for the cost engine: gen_ai.usage.input_tokens SHOULD include cached tokens, and the cache-read count is a subset of it. Source: the repository's docs/gen-ai/gen-ai-spans.md and client-inference.md, read 24 October 2026 and re-read 25 October 2026 (the inference page says the span name follows that pattern unless a specific provider defines another, and describes gen_ai.usage.cache_read.input_tokens as a subset of the input total). I could not confirm a latest-commit date, so I give none. Because the status is Development, check the current names before you ship an exporter. :::

What you log, and what you do not. Log model, tokens in, out and cached, latency, time to first token, retries, status and tags (user, feature, prompt version). Do not log prompt text by default: it holds customer data and multiplies storage, and the specification makes message content opt-in. The transform has three modes: full (development), redacted (regexes scrub emails, Indian mobile numbers, Aadhaar-like and PAN-like strings) and metadata (content keys dropped). Start production at metadata.

The data layout. SQLite in WAL mode, one table. Its documentation says readers do not block writers, there is only one writer at a time, and WAL does not work over a network filesystem (sqlite.org/wal.html, read 24 October 2026). One writer thread, one host: that fits, and it tells you where the design stops.

Milestones 1 and 2: span nesting that survives async and threads, and an LLM wrapper

Parent ids must come from the call stack you did not write, which is why they live in a context variable. Tracer.span is a context manager. On entry it reads the current span from a ContextVar to learn its parent, creates the new span, and installs it as current; on exit it records the duration, resets the variable, and hands the span's dictionary to the sink. If the body raises, the status becomes error, the exception type is recorded, and the exception is re-raised: a tracer that swallows errors changes the program it observes. The @tracer.trace decorator is the same thing for functions, and it checks for coroutine functions so async def works.

_current = contextvars.ContextVar("tracekit_span", default=None)
_tags = contextvars.ContextVar("tracekit_tags", default={})
_ids = itertools.count(1)
_PID = uuid.uuid4().hex[:6]   # distinguishes processes that write to one store


def new_id(prefix: str) -> str:
    return "%s%s-%d" % (prefix, _PID, next(_ids))


class Span:
    __slots__ = ("trace_id", "span_id", "parent_id", "name", "kind", "start",
                 "status", "error", "attrs", "tags", "_t0", "_dur", "_tracer")

    def __init__(self, tracer, name, kind, parent, tags, attrs):
        self._tracer = tracer
        self.name, self.kind, self.attrs, self.tags = name, kind, attrs, tags
        self.parent_id = parent.span_id if parent else None
        self.trace_id = parent.trace_id if parent else new_id("t")
        self.span_id = new_id("s")
        self.status, self.error, self._dur = "ok", None, 0.0
        self._t0 = tracer.clock()
        self.start = tracer.wall()

    def set(self, **attrs) -> "Span":
        self.attrs.update(attrs)
        return self

    def to_dict(self) -> Dict[str, Any]:
        return {"trace_id": self.trace_id, "span_id": self.span_id, "parent_id": self.parent_id,
                "name": self.name, "kind": self.kind, "start": self.start, "duration_ms": self._dur,
                "status": self.status, "error": self.error, "tags": self.tags, "attrs": self.attrs}


class Tracer:
    def __init__(self, sink: Callable[[dict], None], clock: Callable[[], float] = time.perf_counter,
                 wall: Callable[[], float] = time.time, transform: Optional[Callable[[dict], dict]] = None):
        self.sink, self.clock, self.wall, self.transform = sink, clock, wall, transform

    @contextmanager
    def span(self, name: str, kind: str = "chain", **attrs):
        parent = _current.get()
        sp = Span(self, name, kind, parent, dict(_tags.get()), dict(attrs))
        token = _current.set(sp)
        try:
            yield sp
        except BaseException as e:            # status=error, but always re-raise
            sp.status, sp.error = "error", type(e).__name__
            raise
        finally:
            sp._dur = (self.clock() - sp._t0) * 1000.0
            _current.reset(token)
            d = sp.to_dict()
            self.sink(self.transform(d) if self.transform else d)

Tags use a second context variable, so with tags(feature="support"): labels every span opened inside it. That is how cost reaches a feature or customer without threading identifiers through signatures.

Context crosses await for free and crosses threads only if you carry it. Python's documentation says context variables are natively supported in asyncio, so tasks created inside a span see it as their parent. A new thread does not automatically share the creator's context, and the documentation's own advice for handing one over is copy_context().run. The helper in_context(fn) does that, and the part that is easy to get wrong is where it runs: it must be called in the thread that owns the span, not inside the worker.

@contextmanager
def tags(**kv):
    """Tags (user, feature, prompt_version) attach to every span opened inside the block."""
    token = _tags.set({**_tags.get(), **kv})
    try:
        yield
    finally:
        _tags.reset(token)


def current_span() -> Optional[Span]:
    return _current.get()


def in_context(fn: Callable) -> Callable:
    """Capture the caller's context now, run fn inside it later (for threads and executors)."""
    ctx = contextvars.copy_context()
    return lambda *a, **k: ctx.run(fn, *a, **k)

The test below proves all three behaviours at once: the linked threads parent correctly, the plain submissions do not, and the exception test (shown in the repo) proves a failed span closes and does not leak into the next one.

def test_threads_need_in_context_and_get_the_parent_with_it(self):
    tr, out = collect()

    def work(i):
        with tr.span("w%d" % i):
            pass
    with tr.span("root"):
        jobs = [in_context(lambda i=i: work(i)) for i in range(2)]      # context captured HERE, in the caller
        with ThreadPoolExecutor(2) as ex:
            list(ex.map(lambda job: job(), jobs))                       # ...and replayed in the worker
            list(ex.map(lambda i: work(i + 10), range(2)))              # plain submit: context is lost
    root = next(s for s in out if s["name"] == "root")
    linked = [s for s in out if s["name"] in ("w0", "w1")]
    lost = [s for s in out if s["name"] in ("w10", "w11")]
    self.assertEqual(len(linked) + len(lost), 4)
    self.assertTrue(all(s["parent_id"] == root["span_id"] for s in linked))
    self.assertTrue(all(s["parent_id"] is None for s in lost))

:::warn A failure I hit while building this My first version of that test wrote ex.map(lambda i: in_context(work)(i), range(2)). It looked right and the spans came out orphaned: in_context ran inside the worker thread, so it captured the worker's empty context, not the caller's. The fix is to build the wrapped jobs in the calling thread (jobs = [in_context(...) ...]) and only then hand them to the pool. Whenever you add a thread pool to an instrumented app, test the parent id. :::

Wrapping the model call. TracedLLM takes any llmkit client, opens an llm span named chat <model>, calls the inner client with a retry loop, and then writes the usage and cost onto the span. Retries are counted on the span (retries), and the final failure marks the span as an error with the retry count preserved, so a dashboard can tell 'failed after three tries' from 'failed immediately'. It also records usage_source, because a FakeLLM and some servers report estimated token counts; the field makes an estimate visible instead of letting it pass as a provider figure.

class TracedLLM(LLMClient):
    def __init__(self, inner: LLMClient, tracer: Tracer, prices: PriceTable, provider: str = "fake",
                 retries: int = 0, backoff_s: float = 0.0, usage_source: str = "estimated"):
        self.inner, self.tracer, self.prices = inner, tracer, prices
        self.provider, self.retries, self.backoff_s = provider, retries, backoff_s
        self.usage_source, self.model = usage_source, inner.model

    def complete(self, messages, **opts):
        attrs = {"gen_ai.operation.name": "chat", "gen_ai.provider.name": self.provider,
                 "gen_ai.request.model": self.model}
        with self.tracer.span("chat " + self.model, kind="llm", **attrs) as sp:
            attempt = 0
            while True:
                try:
                    c = self.inner.complete(messages, **opts)
                    break
                except Exception:
                    if attempt >= self.retries:
                        sp.set(retries=attempt)
                        raise
                    attempt += 1
                    time.sleep(self.backoff_s)
            cached = getattr(c, "cached_input_tokens", 0)
            cost = self.prices.cost(c.model or self.model, c.input_tokens, c.output_tokens, cached)
            sp.set(**{"gen_ai.response.model": c.model,
                      "gen_ai.usage.input_tokens": c.input_tokens,
                      "gen_ai.usage.output_tokens": c.output_tokens,
                      "gen_ai.usage.cache_read.input_tokens": cached},
                   retries=attempt, usage_source=self.usage_source,
                   cost_usd=cost, cost_known=cost is not None, llm_latency_ms=c.latency_ms)
            return c

Streams. traced_stream wraps a token iterator and writes ttft_ms and tpot_ms; the test drives it with a fake clock (300 ms to the first token, 20 ms per token after) and asserts exactly 300, 20 and a 380 ms span. Limitation for your README: a generator that opens a span must be consumed in the context that created it, so this is for synchronous streams.

Milestone 3: a cost engine you can verify, and the cache discount

Compute cost at write time from tokens and a dated table, store the tokens, and you can always recompute. The price table is a JSON file with an as_of date and a source note; cost() returns dollars, or None for a model that is not in the table. None is the point: pricing an unknown model at zero would make a new model's spend invisible, so the span records cost_known=False, the dashboard counts those calls, and the budget guard (later) charges a fallback amount instead of letting them through free.

    def cost(self, model: str, input_tokens: int, output_tokens: int,
             cached_input_tokens: int = 0) -> Optional[float]:
        """Dollars, or None when the model is not in the table. input_tokens INCLUDES the cached
        ones (the OpenTelemetry GenAI convention), so the uncached part is the difference."""
        p = self.models.get(model)
        if p is None:
            return None
        fresh = input_tokens - cached_input_tokens
        assert fresh >= 0, "cached tokens cannot exceed input tokens"
        return (fresh * p["input"] + cached_input_tokens * p.get("cached_input", p["input"])
                + output_tokens * p["output"]) / 1e6

The cached-token line is where most hand-rolled calculators are wrong. Following the convention verified in the box above, input_tokens includes the cached tokens, so the fresh part is the difference. A cached input token is billed at a lower rate by providers that offer prompt caching (Prompt Caching: Paying for the Same Tokens Once explains the mechanics); counting cached tokens at the full rate overstates cost, and counting them as extra tokens double-counts. The function asserts that cached never exceeds input, because that state means an instrumentation bug upstream and you want it loud.

The prices in pricing.json are assumed round numbers ($0.15 and $0.60 per million input and output tokens for 'small', cached input a tenth of input), dated 24 October 2026 and deliberately not copied from a vendor. Replace them on the day you measure; is_stale() lets the dashboard warn after 90 days.

Prove it with arithmetic you did on paper. Eight table-driven cases, each worked by hand in the comment next to it, for example 1,000 input and 100 output tokens on 'small' is 1000 x 0.15 + 100 x 0.60 = 210 micro-dollars, and the same call with 800 cached tokens is 200 x 0.15 + 800 x 0.015 + 60 = 102 micro-dollars.

class TestCost(unittest.TestCase):
    CASES = [  # (model, input, output, cached, dollars worked out by hand)
        ("small", 1000, 100, 0, 0.00021),                 # 1000*0.15 + 100*0.60 = 210 micro-dollars
        ("small", 1000, 100, 800, 0.000102),              # 200*0.15 + 800*0.015 + 60 = 102
        ("small", 0, 0, 0, 0.0),
        ("small", 1_000_000, 0, 0, 0.15),
        ("large", 2000, 500, 0, 0.01),                    # 2000*2.5 + 500*10 = 10000 micro
        ("large", 2000, 500, 2000, 0.0055),               # 2000*0.25 + 5000 = 5500
        ("fake-1", 10, 3, 0, 0.0000033),                  # 1.5 + 1.8 = 3.3 micro
        ("small", 0, 1_000_000, 0, 0.6),
    ]

    def test_eight_hand_computed_cases(self):
        for m, i, o, c, want in self.CASES:
            self.assertAlmostEqual(PRICES.cost(m, i, o, c), want, places=10, msg=str((m, i, o, c)))

    def test_unknown_model_is_flagged_not_free(self):
        self.assertIsNone(PRICES.cost("mystery-9", 1000, 100))
        tr, out = collect()
        TracedLLM(FakeLLM(default="x", model="mystery-9"), tr, PRICES).complete([{"role": "user", "content": "q"}])
        self.assertFalse(out[0]["attrs"]["cost_known"])
        self.assertIsNone(out[0]["attrs"]["cost_usd"])

    def test_cached_cannot_exceed_input(self):
        with self.assertRaises(AssertionError):
            PRICES.cost("small", 10, 1, 11)

    def test_staleness_uses_the_as_of_date(self):
        self.assertFalse(PRICES.is_stale(datetime.date(2027, 1, 1)))
        self.assertTrue(PRICES.is_stale(datetime.date(2027, 3, 1)))

Beside it: an unknown model is flagged and not free, cached above input fails loudly, and staleness takes the date as an argument so it never depends on today.

Milestones 4 and 5: a writer that never blocks, and queries with known answers

The tracer sits on the request path, so the write must not. Writer.submit is put_nowait on a bounded queue. If the queue is full it increments a drop counter and returns; it never blocks and never raises. A background thread takes up to 500 spans at a time and writes them in one transaction with executemany. The thread opens its own SQLite connection, because SQLite connections belong to the thread that made them. flush() waits for the queue to drain, close() flushes and stops, and health() returns queue depth, drops, rows written and the last write time, which is the observer observing itself. Limitation: as printed, a span that cannot be turned into a row would raise inside the writer thread and stop it (and flush() would then wait forever), so a production version catches, counts and continues.

    """submit() never blocks and never raises: a full queue drops the span and counts it."""

    def __init__(self, path: str, max_queue: int = 10000, batch: int = 500, flush_s: float = 0.05):
        self.path, self.batch, self.flush_s = path, batch, flush_s
        self.q = queue.Queue(maxsize=max_queue)
        self.dropped, self.written, self.last_write_ms = 0, 0, 0.0
        self._lock, self._stop = threading.Lock(), threading.Event()
        connect(path).close()                      # create the file and schema up front
        self._t = threading.Thread(target=self._run, name="tracekit-writer", daemon=True)
        self._t.start()

    def submit(self, span: dict) -> None:
        try:
            self.q.put_nowait(span)
        except queue.Full:
            with self._lock:
                self.dropped += 1

    def _run(self) -> None:
        con = connect(self.path)                   # SQLite connections stay in the thread that made them
        while not (self._stop.is_set() and self.q.empty()):
            rows: List[tuple] = []
            try:
                rows.append(row_of(self.q.get(timeout=self.flush_s)))
            except queue.Empty:
                continue
            while len(rows) < self.batch:
                try:
                    rows.append(row_of(self.q.get_nowait()))
                except queue.Empty:
                    break
            t0 = time.perf_counter()
            with con:
                con.executemany("INSERT OR REPLACE INTO spans VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", rows)
            self.last_write_ms = (time.perf_counter() - t0) * 1000.0
            self.written += len(rows)
            for _ in rows:
                self.q.task_done()
        con.close()

The proof has two halves, and the numbers disagree on purpose. The unit test stalls the consumer, submits 10,000 spans into a queue of 100, and asserts three things: it finished in under a second (no blocking), every span is accounted for as written, queued or dropped, and at least 9,899 were dropped. The benchmark then measures what a real service would see. This is the output of python evals/bench.py on my machine (Python 3.9.13, Windows 11). Treat every timing in it as a one-machine sample: when I re-ran it in a separate review the accounting block was identical, the overhead was 5.4 and 10.6 microseconds, the burst took 0.500 s (about 100,000 spans/s into the queue) with the same 39,500 dropped and 10,500 written, and the paced run emitted 19,846 spans instead of 19,967, still with zero drops:

accounting       OK
  total_usd    got 0.174036   want 0.174036   
  p50          got 500.0      want 500.0      
  p95          got 950.0      want 950.0      
  p99          got 990.0      want 990.0      
  error_rate   got 0.037      want 0.037      
  spans        got 1000       want 1000       
  support_usd  got 0.033966   want 0.033966   
overhead per span (best of 3 runs of 50,000):
  tracer + no-op sink          5.6 us
  tracer + Writer.submit      11.0 us   (gate: <= 200 us)
burst of 50000 spans, queue 10,000: producer time 0.941 s (53116 spans/s into the queue), dropped 39500, written 10500
paced 10,000 spans/s for 2 s: emitted 19967, written 19967, dropped 0
machine: Python 3.9.13, Windows-10-10.0.22631-SP0
GATE PASSED

Read the two throughput lines separately. In a tight loop of 50,000 spans with a 10,000-slot queue, the producer finished in about a second or less but the single writer thread could not keep up and the tracer dropped 39,500 spans. Paced at 10,000 spans per second for two seconds, which is far above what an LLM application emits, it wrote every one and dropped none. Both are honest: a burst faster than the writer drains is the case the drop counter exists for, and the README should say the sustained rate the writer holds, not the burst rate the queue absorbs. If you need to survive bursts, raise max_queue, raise the batch size, or sample; do not make submit block.

Queries with known answers. The fixture builds 1,000 LLM spans through the real tracer with a fake clock, so ground truth is arithmetic: span i takes i ms (nearest-rank p50, p95, p99 = 500, 950, 990); every 27th span fails (37 in 1,000); feature is support, search or sales by i mod 3; support calls carry 800 cached tokens. Tag names are validated before they reach a JSON path, so a hostile tag cannot become SQL.

def test_cost_by_feature_matches_hand_totals(self):
    rows = {r["key"]: r for r in queries.cost_by(self.con, "feature")}
    self.assertEqual({k: v["calls"] for k, v in rows.items()}, {"support": 333, "search": 334, "sales": 333})
    self.assertAlmostEqual(rows["support"]["usd"], 0.033966, places=9)      # 333 * 0.000102
    self.assertAlmostEqual(rows["search"]["usd"], 0.07014, places=9)        # 334 * 0.00021
    self.assertAlmostEqual(sum(r["usd"] for r in rows.values()), 0.174036, places=9)
    self.assertNotIn("(untagged)", rows)

Why p95, and cheaply. The mean hides the tail users feel. Exact percentiles need every duration, fine for a day in SQLite; LogHistogram keeps counts in 5 percent geometric buckets and is never below the exact value or more than one bucket above it (tested). A t-digest is a stretch goal.

:::warn Two failures from the build A float clock leaks into your numbers. My first percentile test expected exactly 950.0 and got 949.9999999999886, because durations are differences of floating-point seconds. The test now rounds to six places and says why. My own arithmetic was wrong once. The test expecting a grand total of 0.173736 failed against the code's 0.174036; the code was right and my hand sum of three subtotals was off by 0.0003. A check by hand is only as good as the hand: add the subtotals twice, or compute them a second way. :::

Milestone 6: budgets, a runaway guard and the 80 percent alert

A budget is a check before the call and a charge after it, and it must not be possible to configure it off by omission. BudgetGuard keeps spend per key and per calendar day, fires on_alert once when spend crosses 80 percent of the limit, and fires on_stop at 100 percent; check() raises the shared BudgetExceeded from llmkit.cost so callers handle one exception type across projects. GuardedLLM wraps a client: it calls check first, so a key that is already over its cap never reaches the model, and charges the real cost afterwards. For one caller at a time the overshoot is therefore bounded by one call, which a test asserts. As printed, BudgetGuard keeps its totals in a plain dictionary with no lock and in one process's memory: concurrent callers can race the read-modify-write and each in-flight call can overshoot, and several worker processes each keep their own total. For a threaded service add a lock (or a shared store) before you claim a cap. An unpriced model is charged a fallback (unknown_usd) rather than zero, for the reason in the cost section.

EPS = 1e-12      # eight charges of 0.1 sum to 0.7999999999999999, and the 80 percent alert must still fire


class LoopDetected(RuntimeError):
    pass


class BudgetGuard:
    def __init__(self, daily_usd: float, warn_at: float = 0.8,
                 on_alert: Optional[Callable[[str, float], None]] = None,
                 on_stop: Optional[Callable[[str, float], None]] = None,
                 today: Callable[[], str] = lambda: datetime.date.today().isoformat()):
        self.limit, self.warn_at, self.on_alert, self.on_stop, self.today = daily_usd, warn_at, on_alert, on_stop, today
        self.spent: Dict[tuple, float] = {}
        self.alerted: set = set()

    def used(self, key: str) -> float:
        return self.spent.get((key, self.today()), 0.0)

    def check(self, key: str) -> None:
        if self.used(key) >= self.limit - EPS:
            raise BudgetExceeded("%s is at its daily cap of $%.4f" % (key, self.limit))

    def charge(self, key: str, usd: float) -> None:
        k = (key, self.today())
        self.spent[k] = self.spent.get(k, 0.0) + usd
        if self.spent[k] >= self.warn_at * self.limit - EPS and k not in self.alerted:
            self.alerted.add(k)
            if self.on_alert:
                self.on_alert(key, self.spent[k])
        if self.spent[k] >= self.limit - EPS and self.on_stop:
            self.on_stop(key, self.spent[k])

The loop detector is the cheap first line; the budget is the backstop. A runaway agent usually repeats itself: same tool, same arguments, over and over. LoopDetector hashes (tool, arguments) and counts consecutive repeats; a changed argument resets the count; the fifth identical call raises LoopDetected. The simulation is the proof the plan asks for: a scripted agent that never finishes, run twice. With the detector it stops after exactly 5 steps. With the detector off, the budget stops it, and the total spent is at most the limit plus one call.

def test_looping_agent_is_stopped_inside_its_budget(self):
    # A scripted agent that never finishes: ask the model, call the same tool, repeat.
    def run(detector):
        guard = BudgetGuard(0.002, today=lambda: "d")
        llm = GuardedLLM(FakeLLM(default="call search(refund)" * 30, model="small"), guard, PRICES, "key-a")
        steps = 0
        try:
            while steps < 1000:
                llm.complete([{"role": "user", "content": "what next? " * 200}])
                steps += 1
                if detector:
                    detector.observe("search", {"q": "refund"})
        except (BudgetExceeded, LoopDetected) as e:
            return steps, guard.used("key-a"), type(e).__name__
        return steps, guard.used("key-a"), "ran away"
    steps, spent, why = run(LoopDetector(5))
    self.assertEqual((steps, why), (5, "LoopDetected"))
    steps2, spent2, why2 = run(None)                              # detector off: the budget is the backstop
    self.assertEqual(why2, "BudgetExceeded")
    self.assertLessEqual(spent2, 0.002 + spent2 / steps2 + 1e-12)  # overshoot is at most one call
    self.assertLess(steps2, 1000)

Why both. The detector catches the obvious loop in five calls; the daily cap catches loops that vary their arguments. Add per-run step limits (the agent project does), a human alert path and a kill switch. A guard that only logs is a report.

:::warn A failure I hit while building this The first version of the 80 percent alert never fired at 80 percent. Eight charges of $0.10 add up to 0.7999999999999999 in floating point, which is below 0.8 x 1.0, so the alert waited until the ninth charge and reported 90 percent. The fix is a tolerance (EPS = 1e-12) on every threshold comparison, with the reason in a comment. For real money, accumulate integer micro-dollars and avoid the problem; for a guard, a tolerance is enough, but you must test the boundary, and 'eight charges of ten cents' is exactly the test that finds it. :::

Milestone 7: redaction before write, and sampling for review

Scrub in the tracer's transform, so the unredacted text never reaches the queue, the database, a log line or a crash dump. The Tracer takes an optional transform that runs on the finished span dictionary before the sink sees it. In redacted mode it applies four regular expressions to the content keys only: emails, Indian mobile numbers (with or without +91, and tolerating the common 5-5 grouping), 12-digit sequences grouped 4-4-4 (Aadhaar-like), and PAN-like strings (five letters, four digits, a letter). In metadata mode content keys are dropped entirely. The test writes a span through a real Writer and then reads the raw SQLite row to prove the email is not there.

def redact(text: str) -> str:
    for label, pat in PATTERNS:
        text = pat.sub("[%s]" % label, text)
    return text


def make_transform(mode: str = "redacted") -> Callable[[dict], dict]:
    """mode: 'full' keeps content as is (development only), 'redacted' scrubs it,
    'metadata' drops content keys entirely (the production default to start from)."""
    assert mode in ("full", "redacted", "metadata")

    def transform(d: dict) -> dict:
        if mode == "full":
            return d
        attrs = {}
        for k, v in d["attrs"].items():
            if k in CONTENT_KEYS:
                if mode == "metadata":
                    continue
                v = redact(v) if isinstance(v, str) else v
            attrs[k] = v
        d["attrs"] = attrs
        return d
    return transform

Be honest in the README. These are pattern matches, not a PII detector: the Aadhaar-like pattern has no checksum so any 12-digit number matches, and names in free text match nothing. Treat the regexes as a floor.

Sampling decides which traces a human or a judge will read. ReviewSampler hashes the trace id with a salt and compares to the rate, so every process makes the same decision for the same trace with no shared random generator; as a sink it looks at root spans only. The test draws 10,000 trace ids at 5 percent and asserts a count within about 435 to 565 (500 plus or minus three binomial standard deviations, 21.8 each; the test uses the slightly wider 434 to 566) and that two samplers with the same salt choose the identical set.

Milestone 8: the dashboard, the three-line integration, and the accounting gate

The API is a thin read layer over the queries, with two secrets and no framework. make_server returns a standard-library ThreadingHTTPServer: /api/health is open; /api/summary?by=feature, /api/traces?top=5 and /api/trace/<id> need a bearer read token; POST /api/ingest needs a different write key, so a leaked dashboard token cannot inject data. Tests start the server on a free port and check the JSON shape, the 401, the trace tree and the write-key rule. The HTML page draws a cost table and a waterfall (bars positioned by start and duration, indented by depth). I tested its data path and that it is served; I did not render it in a browser for this lesson, so take your own screenshot for the README.

Instrumenting the RAG assistant takes three lines and no edits to it. From the previous project, rag.generate.answer calls retrieve and then the model. The example script builds a tracer with a writer, wraps the model client in TracedLLM, and decorates the two functions (this works against the previous lesson's repository as I built it; the paths are in the script):

writer, sampler = Writer(db), ReviewSampler(0.25)
tracer = Tracer(fanout(writer.submit, sampler), transform=make_transform("redacted"))        # line 1
prices = PriceTable.load(os.path.join(HERE, "..", "pricing.json"))
llm = TracedLLM(FakeLLM(default=gen.ExtractiveModel()), tracer, prices)                       # line 2
gen.retrieve = tracer.trace("retrieve", kind="retrieval")(gen.retrieve)                       # line 3
answer = tracer.trace("rag.answer", kind="chain")(gen.answer)

Fifty golden questions later, this is the output of python examples/instrument_rag.py (the trace id prefix is random per run, and the review-queue count varies with it; the durations vary a lot: my first run is below, and a later re-run on the same machine gave an llm p50 near 10 ms, retrieval near 90 to 110 ms and a whole rag.answer near 100 to 125 ms, while the span counts, kinds, costs and token totals were identical):

questions asked: 50 | spans stored: 148
by span kind: {'chain': 50, 'llm': 48, 'retrieval': 50}
cost by feature: [{"key": "rag-assistant", "calls": 35, "usd": 0.003196049999999999, "input_tokens": 15379, "output_tokens": 1482}, {"key": "rag-eval", "calls": 13, "usd": 0.0011925000000000002, "input_tokens": 5702, "output_tokens": 562}]
llm span duration in ms (a FakeLLM call, so this times Python, not a model): {'p50': 1.0193999999998926, 'p95': 1.6173000000000437, 'p99': 2.1979999999999222, 'n': 48}
calls that answered without an llm span (abstained before generation): 2
queued for review (25% sample): 9
most expensive trace t7b41c8-157
  rag.answer           chain       31.33 ms
    retrieve             retrieval   29.73 ms
    chat fake-1          llm          1.35 ms

Read it the way you would read a real one. 148 spans for 50 questions: every question has a chain and a retrieval span, and 48 have an llm span; the other two abstained before generation, which the dashboard can now count. The most expensive trace shows retrieval taking 29.7 of the 31.3 milliseconds. That is a property of this setup, not of RAG in general: the retrieval is pure Python over 36 chunks and the model is a scripted fake that takes a millisecond or a few (retrieval dominated in every run I made). With a real model the generation span would dominate, and the point of the waterfall is that you stop guessing which one it is.

The gate. python evals/bench.py exits non-zero on any accounting deviation, overhead above 200 microseconds per span, or a drop in the paced run. The overhead is best-of-three on one laptop (10.6 to 11.0 microseconds measured in two runs, against 200 allowed), so name the machine. The RAG example also needs the previous project's repository, which that lesson prints only in part (its four main modules, not the corpus, golden set or evals/run.py); set RAG_REPO to your own build and expect your own counts.

Cost, latency, deployment and the observability of the observer

The budget is small and almost all of it is storage. Overhead is about 11 microseconds per span in my run (a tracer with a no-op sink costs about 6), so a request with 10 spans pays roughly 110 microseconds, which is 0.006 percent of a 2-second model call (an assumed latency). The rows in the fixture database take about 430 bytes per span without content (the database file was 430,080 bytes for 1,000 spans after a checkpoint), so one million spans a day is about 430 MB a day before indexes grow and before any content capture, which multiplies it. Hence a retention job (delete rows older than N days) and the metadata default are not optional extras. Runtime cost is zero dollars: SQLite and the standard library.

Deployment, with the free-tier caveats. One container serving the API and page, with the SQLite file on a volume.

:::key Verified 24 October 2026 (check current limits before you rely on any of this) Render. The free-instances page says a Free web service spins down after 15 minutes without inbound traffic and takes about one minute to wake; it does not support a persistent disk, so local files are lost on redeploy, restart or spin-down; each workspace gets 750 free instance hours per calendar month; and free Render Postgres databases expire 30 days after creation, followed by a 14-day grace period. Consequences for this project: a SQLite file on a free instance is a demo database, not a record, so ship a seed dataset (the fixture) that the container loads at start, and put the real store on a paid persistent disk or another database if you ever need history. Source: render.com/docs/free, read 24 October 2026. Langfuse, the tool this project imitates, describes itself as an open source LLM engineering platform, MIT licensed except for its ee folders, and self-hostable by Docker Compose, a VM or Kubernetes (github.com/langfuse/langfuse README, read 24 October 2026). :::

Keep the read token and write key in environment variables, rate-limit the public demo, and make it read-only over seed data.

The observer needs its own health. /api/health exposes queue depth, drops, rows written and the last write time; alert on drops above zero and on a queue that stays near full. A tracer that fails silently is worse than none, because you will trust the dashboard that it has quietly stopped feeding.

Failure analysis, pitfalls and stretch goals

Failures for docs/failures.md (each with a guarding test): orphaned thread spans, the alert at 90 percent, and the throughput claim that hid 79 percent loss, all described above.

Pitfalls. Synchronous writes on the request path; undated prices; raw prompts with personal data; wrong streamed-token counts (prefer provider usage, flag estimates); an unbounded queue; a regex redactor sold as PII protection; a burst rate reported as sustained; p95s from different windows or sample rates.

Stretch goals. An OpenTelemetry exporter that maps spans to the GenAI attributes above (check the status first, because the conventions are in Development). DuckDB for analytics and a cohort cost report. A prompt-version comparison view, built on the paired comparison in the next project. A t-digest in place of the log histogram. Wiring the review queue to the judges from the eval project. An async-safe stream wrapper.

Present it: README numbers, interview questions, pitch and resume bullet

The README results table, filled with what this repository regenerates. Run python -m unittest discover -s tests and python evals/bench.py and paste your own output; mine, on Python 3.9.13 and Windows 11 (the numbers that depend on the machine are marked):

itemvalue
tests34 passing, offline, no key
accounting fixture1,000 spans; total $0.174036, p50 500 ms, p95 950 ms, p99 990 ms, error rate 0.037; all match by construction
price tableassumed prices, as of 24 October 2026 (replace with yours)
tracer overhead (this machine)about 11 microseconds per span with the writer (10.6 to 11.0 in two runs), about 5 to 6 with a no-op sink; gate 200
sustained throughput10,000 spans per second for 2 seconds, 0 dropped (this machine)
burst behaviour50,000 spans in a tight loop into a 10,000 queue: 39,500 dropped, counted
storageabout 430 bytes per span, metadata only
failure analysis3 failures with tests, in docs/failures.md

The five questions this project prepares you for, with outlines.

@@ 1 | What do you log for an LLM call, and what do you not? Model, tokens in, out and cached, latency, TTFT, retries, status, tags. Content opt-in and redacted. Cost derived from tokens and a dated table, so it is recomputable.

@@ 2 | How do you attribute cost to a feature or customer? Tags propagate by context; cost is computed at write time; group by tag with an explicit '(untagged)' bucket whose size measures completeness. State a policy for shared-prefix cache credit.

@@ 3 | Your agent loops and burns money overnight. Prevent it. Per-run step and token limits, a daily budget checked before each call with 80 percent alert and hard stop, a loop detector, a kill switch. You proved it with a simulated loop; overshoot is bounded by one call.

@@ 4 | Why p95 and not the average, and how do you compute it cheaply? The tail drives experience. Exact nearest-rank for a day of data; geometric histogram for constant memory with bounded error; t-digest for finer tails.

@@ 5 | Build or buy? Building teaches the model. In production, adopt the OpenTelemetry GenAI conventions (still Development) and a vendor or self-hosted tool for retention, auth and UI; self-hosting is a privacy option.

The 90-second pitch (ILLUSTRATIVE). 'Teams ship LLM features and cannot say what a question costs or why it was slow. I built a small tracing library where every call is a span with tokens, retries and cost, written without blocking the request. The decision that mattered was making accounting testable: a 1,000-span fixture with known totals. The failure I am proudest of finding was a throughput claim that hid a 79 percent drop in a burst; I now report sustained and burst separately. Overhead is about 11 microseconds a span on my laptop, a guard stops a looping agent inside its cap, and redaction runs before anything is written.' Hinglish opener: 'Maine ek chhota tracing library banaya jo har LLM call ka token, cost aur latency record karta hai, aur request ko block nahi karta.'

The resume bullet (ILLUSTRATIVE shape; numbers are what this repo prints on my machine). Built tracekit (Python, SQLite WAL, OpenTelemetry GenAI names): span trees across async and threads, dated price table with cache discounts, budget and loop guards, redaction before write; accounting verified against a 1,000-span fixture, about 11 microseconds overhead per span, 34 offline tests and a CI gate.

An honest time estimate. About 17 to 18 hours: spans and wrapper 4, cost engine 2, writer and queries 3, guards and privacy 3, API and integration 3, benchmark and write-up 2 to 3. Stretch goals add the rest of the 30.

How to say it in the interview

"Observability for an LLM app is mostly arithmetic you can check. Every call becomes a span with tokens, retries and latency; cost is computed at write time from a dated price table and is recomputable from the tokens; an unknown model is flagged, never priced at zero. I test the accounting against a 1,000-span fixture whose totals I know by construction, I measured about 11 microseconds of overhead per span against a 200 microsecond gate, and the writer never blocks the request: a full queue drops a span and counts it. Budgets stop a looping agent, redaction runs before anything is written, and the numbers on my dashboard are numbers I can regenerate with one command."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons