Cruxly

Cruxly › AI engineering › Portfolio Projects

Project: A Cost-Aware LLM Gateway With Routing, Caching, Failover and Budget Caps

An OpenAI-compatible proxy that routes by difficulty, caches safely, fails over, and enforces budgets under concurrency

Open the interactive lesson

Project: "My team calls a model API from six services. Last month one provider had an outage, one service sent a loop that cost a day's budget, and a customer's answer came from another customer's cache. I want one gateway in front of all of them that routes, caches, fails over and enforces budgets, and I want proof that each of those works."

Time-box: about 36 hours for the core, 52 with the stretch goals. You will have a running gateway written in Python 3.9 with the standard library plus llmkit (aip1) and tracekit (aip2) imported, not copied. It has 43 unit tests, a probe script that tries to break it, an eval harness that prints a summary, and a list of the claims it does not support.

Why this project, and what it proves to an employer

A gateway proves judgement about money, failure and isolation, which are the three things that go wrong in production. Anyone can call one API. Fewer people can explain what happens when the provider returns 429 halfway through a stream, when two requests race for the last cent of a budget, or when the cache is asked a question that looks like a stored one but differs by one number.

Reviewers look for circuit breakers, caches with safety rules, a router whose quality loss is measured, and cost accounting that reconciles. This project covers each, and it is where I broke my own code most, so the failure section is long. Background lessons: Multi-Provider Failover That Actually Fails Over, Cost Attribution and Runaway Prevention and Prompt Caching.

Scope, metrics and non-goals (numbers first)

Before any code, I wrote down the numbers the gateway must meet, and I committed them to the design notes. Some of them I could not meet offline, and I say which.

Non-goals. No real provider key, no Redis (SQLite is enough for one process), no learned router, no hedged requests, no FastAPI. I also name what a reviewer will probe: cross-tenant leaks, prompts that cross the cache boundary, provider overspend, and every provider being down.

Architecture and the seam: what is fake, what is real

The request path has five exits, and each exit is recorded on a trace span, so a reviewer can see why a request cost what it cost. The exits are a cache hit, a routed call, a retried call, a failover to the next provider, and a rejection by the budget or the rate limit.

What is fake and what is real. The providers are FakeProvider objects with a script: each call consumes the next entry, a success, 429, 500, timeout or malformed stream. The fake clock advances by the scripted latency, so every timing in the tests is simulated. The live adapter wraps OpenAICompatClient from llmkit, so the gateway can point at a hosted API, vLLM or Ollama through environment variables; the tests never call it. The embedder is HashEmbedder, lexical and so a weak paraphrase model.

The gateway imports llmkit.client, llmkit.embed and tracekit.trace.Tracer; nothing is copied. llmkit sits at <root>/aip1work/llmkit and tracekit at <root>/aip2work/tracekit, and tests/_paths.py finds them two folders above the project or in the folder named by AIP_SCRATCH.

# -*- coding: utf-8 -*-
"""Shared import paths: this project's src, plus tracekit (aip2) and llmkit (aip1), which are imported, not copied."""
import os
import sys

HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
SCR = os.environ.get("AIP_SCRATCH", os.path.abspath(os.path.join(ROOT, "..", "..")))
for p in (os.path.join(ROOT, "src"), os.path.join(SCR, "aip2work", "tracekit", "src"),
          os.path.join(SCR, "aip1work", "llmkit")):
    if p not in sys.path:
        sys.path.insert(0, p)

What is printed and what is not. The lesson prints about 230 lines of the 1,300-line project: the code where a decision lives, plus __init__.py and clock.py. The other six package files are in the repository and only described: providers.py (fakes, live adapter, ProviderError, classify), ledger.py, cache.py, gateway.py (the Gateway class, parse_request, route, price_micro), resilience.py (token bucket, retry policy) and api.py (HTTP). Typing only the printed parts does not give a runnable package. llmkit is printed in full in the first lesson; tracekit only in part in the tracing lesson, so for it the repository is the source.

# -*- coding: utf-8 -*-
from .gateway import Gateway, GatewayError, route, price_micro  # noqa: F401
from .ledger import Ledger, hash_key  # noqa: F401
from .providers import FakeProvider, ProviderError  # noqa: F401
from .clock import FakeClock, RealClock  # noqa: F401
# -*- coding: utf-8 -*-
"""Clocks. Production uses the monotonic clock; tests use a fake one that only moves when told."""
import time


class RealClock:
    def now(self) -> float:
        return time.monotonic()


class FakeClock:
    def __init__(self, start: float = 0.0):
        self.t = float(start)

    def now(self) -> float:
        return self.t

    def advance(self, seconds: float) -> None:
        self.t += seconds

Milestones 1 and 2: adapters, fakes and streaming passthrough

The adapter contract is one method, complete, and every adapter raises the same small set of error kinds. A failure is a ProviderError with a kind: rate_limit, server, timeout or malformed. The live adapter turns an HTTP status into the same kinds through classify, so the retry code never reads an HTTP status directly.

Streaming needs one more rule. Once the first byte has reached the client, the gateway cannot retry, so the stream path takes one provider, does not retry, and reports an upstream failure inside the stream. The important test is client disconnect: when the reader goes away, the upstream generator must be closed, and whatever tokens were produced must still be billed. The code below is that path. The finally block is what runs when the server's write fails and the generator is closed.

def stream(self, raw_key: str, body: dict):
    """Streaming path: no cache, one provider (the first healthy one in the tier), no retry after the
    first byte. Yields text pieces. If the consumer stops early (client disconnect), closing this generator
    closes the upstream stream and settles the tokens actually produced."""
    self._admit(raw_key)
    req = parse_request(body)
    msgs, max_tokens, request_bytes = req["messages"], req["max_tokens"], req["request_bytes"]
    tier = route(req)
    key_hash = hash_key(raw_key)
    reserved = price_micro(tier, request_bytes, max_tokens, round_up=True)
    if not self.ledger.reserve(key_hash, reserved):
        self.counts["budget_rejected"] += 1
        raise GatewayError(402, "budget_exceeded")
    prov = next((p for p in self.providers.get(tier, []) if self._breaker(p.name).allow()), None)
    if prov is None:
        self.ledger.settle(key_hash, reserved, 0)
        raise GatewayError(503, "all_providers_failed")
    from llmkit.client import approx_tokens, messages_tokens
    tin, tout = messages_tokens(msgs), 0
    gen = prov.stream(msgs, max_tokens=max_tokens)
    try:
        for piece in gen:
            tout += approx_tokens(piece)
            yield piece
        self._breaker(prov.name).success()
    except ProviderError as e:
        if classify(e) != "client":
            self._breaker(prov.name).failure()
        raise GatewayError(503, classify(e))
    finally:
        gen.close()
        self.ledger.settle(key_hash, reserved, price_micro(tier, tin, min(tout, max_tokens)))

The test that proves it starts a real ThreadingHTTPServer, opens a raw socket, reads one chunk, closes it, and waits until the provider's stream has closed and the reservation has settled.

Milestones 3: retries, timeouts, circuit breaker, failover

A retry without a breaker turns an outage into a storm, so the breaker opens first and the retries stay small and jittered. The breaker is a three-state machine. It is closed while calls succeed, it opens after a run of failures, and after a cooldown it lets exactly one probe through. The probe decides whether it closes again or re-opens.

class CircuitBreaker:
    """closed -> open after `threshold` consecutive failures; open -> half-open after `cooldown` seconds;
    half-open lets exactly one probe through: success closes it, failure re-opens it."""

    def __init__(self, clock, threshold: int = 3, cooldown: float = 30.0):
        self.clock, self.threshold, self.cooldown = clock, threshold, cooldown
        self.state, self.failures, self.opened_at = "closed", 0, 0.0
        self._probe_out, self._probe_at = False, 0.0
        self._lock = threading.Lock()

    def allow(self) -> bool:
        with self._lock:
            if self.state == "open" and self.clock.now() - self.opened_at >= self.cooldown:
                self.state, self._probe_out = "half-open", False
            if self.state == "closed":
                return True
            # a probe whose outcome never arrived (client disconnect, crash) expires after one more cooldown,
            # or the breaker would stay half-open forever
            if self.state == "half-open" and (not self._probe_out or self.clock.now() - self._probe_at >= self.cooldown):
                self._probe_out, self._probe_at = True, self.clock.now()
                return True
            return False

    def success(self) -> None:
        with self._lock:
            self.state, self.failures, self._probe_out = "closed", 0, False

    def failure(self) -> None:
        with self._lock:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.threshold:
                self.state, self.opened_at, self._probe_out = "open", self.clock.now(), False

The retry policy uses full jitter: each wait is a uniform draw between zero and an exponential cap. The draw is seeded, so the outage simulation is reproducible. Failover walks an ordered list of providers in the same tier and skips any whose breaker is open. The loop below is the part a reviewer should read first, because it decides what the client sees.

def _try_tier(self, tier: str, req: dict):
    """Returns (result, provider_name, attempts), or None when every provider in the tier failed."""
    attempts = []
    pool = [p for p in self.providers.get(tier, []) if not req["tools"] or p.supports_tools]
    for i, prov in enumerate(pool):
        br = self._breaker(prov.name)
        if not br.allow():
            attempts.append((prov.name, "breaker_open"))
            continue
        for n in range(self.retry.max_retries + 1):
            try:
                res = prov.complete(req["messages"], max_tokens=req["max_tokens"],
                                    temperature=req["temperature"])
            except ProviderError as e:
                kind = classify(e)
                if kind != "client":     # a 4xx says the request was bad, not that the provider is sick
                    br.failure()
                self.counts["upstream_errors"] += 1
                attempts.append((prov.name, kind))
                if kind not in RETRYABLE or n == self.retry.max_retries:
                    break
                self.sleep(self.retry.delay(n))
                continue
            br.success()
            attempts.append((prov.name, "ok"))
            if i > 0:
                self.counts["failover"] += 1
            return res, prov.name, attempts
    return None

Three rules above came from probes. A provider 4xx does not count against the breaker, or one tenant's bad requests would open it for everyone. A half-open probe that never reports back (a client that disconnects mid-stream) expires after one more cooldown. And the unprinted tier loop in _run moves to the next tier when a whole tier fails or a validator rejects the answer, at that tier's price; a 503 comes only after the strongest tier fails. One limit remains: a timed-out call may still finish and be billed upstream, and the ledger books only answers it received.

The failover claim is tested in three ways: a primary that returns 500 three times is retried and then bypassed, a breaker that has opened stops receiving calls, and a primary that is down for the whole run of 200 requests still gives 200 successes. The last one is a fact about the fixture, not about a real provider, and the lesson says that. What the test does prove is the mechanics. The eval harness also runs ten failure scenarios; the summary below shows that every scenario with a healthy second provider succeeds, and the one where both providers are down returns zero successes, as it should.

overhead (one machine, one run): p50 8.2 ms, p95 12.5 ms, n=400
concurrency: 50 threads, accepted 12, rejected 38, max overshoot 0 micro, reserved left 0
S01 primary down (500 always): 50/50 succeeded
S02 primary rate-limited (429 always): 50/50 succeeded
S03 primary timeout (slow tail): 50/50 succeeded
S04 primary malformed stream: 50/50 succeeded
S05 primary flaps: 500 then ok: 50/50 succeeded
S06 primary 500 once then ok: 50/50 succeeded
S07 primary 429 twice then ok: 50/50 succeeded
S08 primary slow but under timeout: 50/50 succeeded
S09 primary healthy: 50/50 succeeded
S10 secondary also down: 0/50 succeeded

Milestones 4 and 5: exact cache, then semantic cache, with the safety rules

A cache key must contain everything that can change the answer, and a cache hit must be checked again before it is returned. The exact cache hashes a canonical form of the request: the tenant, the routed tier, the prompt version, the model, every message including the system prompt, and the sampling parameters. Whitespace is normalised. Nothing else is normalised, because lowercasing a name can change its meaning.

def canonical(tenant: str, tier: str, prompt_version: str, model: str, messages: List[dict],
              temperature: float, max_tokens: int) -> str:
    """Everything that can change the answer goes into the key: the tenant (isolation), the tier and model,
    the prompt version, EVERY message (system prompt included), and the sampling parameters."""
    body = {"t": tenant, "tier": tier, "pv": prompt_version, "m": model, "temp": temperature,
            "max": max_tokens,
            "msgs": [[m["role"], normalise_text(m["content"])] for m in messages]}
    return json.dumps(body, sort_keys=True, ensure_ascii=False)

Four requests are never cached: anything sampled at a temperature above zero, anything that carries tools (a tool call can have a side effect), anything the caller marks private, and anything whose answer failed validation. The rule is in one function so that a reviewer can read it in one place.

def cacheable(temperature: float, tools, private: bool) -> bool:
    """Never cache sampled output, calls with tools (side effects), or user-private requests."""
    return temperature == 0 and not tools and not private

The semantic cache is where most of the work went, and where the honest result is not a success. It embeds the question, finds a stored question above a cosine threshold, and then runs a verification step: the numbers and the capitalised words in the two questions must match. The partition is the tenant, the tier, the prompt version, a digest of the system prompt and earlier turns, and max_tokens, because an answer cut off at 4 tokens must not be served to a caller who allowed 1,000. The verification step is signature:

def signature(s: str) -> Tuple[Tuple[str, ...], Tuple[str, ...]]:
    """What a verification step compares: the numbers AND the capitalised words (names, places, codes).
    Two questions that differ in either are near-misses, however similar their embeddings are."""
    body = s.split(None, 1)[1] if len(s.split(None, 1)) > 1 else ""   # the first word is capitalised by grammar
    return numbers(s), tuple(sorted(w.lower() for w in CAP.findall(body)))

The first word of a sentence is always capitalised, so the code skips it. My first version counted "What" as an entity and recall was zero at every threshold.

cache: 200 pairs; thresholds passing precision>=0.98 and recall>=0.5 on dev: none
cache dev threshold 0.5: hits 48 precision 0.667 recall 0.780
cache dev threshold 0.7: hits 37 precision 0.568 recall 0.512
cache dev threshold 0.9: hits 1 precision 1.000 recall 0.024
near-miss false hits at 0.5: {'entity': 0, 'negation': 0, 'number': 0, 'word': 25}
near-miss false hits at 0.8: {'entity': 0, 'negation': 0, 'number': 0, 'word': 24}

This is the dev half of a labelled set of 200 pairs generated from templates: paraphrases, and near-misses that change a number, an entity, one word, or add a negation. No threshold reaches precision 0.98 with recall 0.5, so the gate fails. The number and entity checks catch every near-miss of those kinds, and the negation near-misses show zero false hits only because the template writes NOT in capitals; a lower-case not passes the verifier (probe 13). The lower-case word change ("refund" for "fee") is caught by neither check. The fix is a stronger verifier, such as a second model judging borderline pairs, measured on your own data. The gate stays failed and the semantic cache stays off in production. The exact cache expires entries after an hour; the semantic cache has no expiry or size limit, so it is a teaching structure.

Milestone 6: the router, and a measured quality loss

A router that works on length and tags cannot see difficulty, so its quality loss is large, and the gate has to say so. The rules are short. Tools go to the strong tier. Tags such as legal or math go to strong. Long prompts go to standard. Everything else goes to cheap.

def route(req: dict) -> str:
    """Rules first: tools and legal/maths tags go strong, long prompts go standard, the rest cheap."""
    if req.get("tools"):
        return "strong"
    if set(req.get("tags", [])) & {"legal", "math"}:
        return "strong"
    if sum(len(m["content"]) for m in req["messages"]) > 800:
        return "standard"
    return "cheap"

The eval harness builds 150 synthetic items with a hidden difficulty, and scripts three tiers: cheap is right only on easy items, standard on easy and medium, strong on all. That is a construction, not a measurement of any model, and the lesson labels it that way. The result is still informative. Always-strong scores 1.00 by construction, the routed mix scores 0.49, and the loss of about 51 points fails the gate of 2. The summary line is printed below. The lesson does not claim the router works. It claims the gate catches a router that does not, and the next step is an escalation on a failed validation, which the code has a hook for but the eval does not exercise.

near-miss false hits at 0.8: {'entity': 0, 'negation': 0, 'number': 0, 'word': 24}

Milestone 7: the budget ledger under concurrency

A budget check must be one transaction: check the balance and book the reservation together, or two requests can both pass the check and both spend the money. The protocol is reserve the worst case before the call, and settle the actual cost after it. The worst case is priced from the request's byte count (a token covers at least one byte, so for byte-level tokenizers the byte count bounds the message tokens; the tools schema and chat-template overhead are not counted, so a request with large tools can be under-reserved) and the maximum output tokens. Money is kept in integer micro-dollars, so sums never drift.

class LedgerTests(unittest.TestCase):
    def setUp(self):
        self.led = Ledger(os.path.join(tempfile.mkdtemp(), "l.db"))

    def test_fifty_parallel_reserves_book_exactly_capacity(self):
        raw = self.led.create_key("acme", 10_000)
        h = hash_key(raw)
        results = []
        lock = threading.Lock()
        barrier = threading.Barrier(50)

        def go():
            barrier.wait()
            ok = self.led.reserve(h, 1_000)
            with lock:
                results.append(ok)

        ts = [threading.Thread(target=go) for _ in range(50)]
        for t in ts:
            t.start()
        for t in ts:
            t.join()
        self.assertEqual(sum(results), 10)
        snap = self.led.snapshot(h)
        self.assertEqual(snap["reserved_micro"], 10_000)
        self.assertLessEqual(snap["spent_micro"] + snap["reserved_micro"], snap["limit_micro"])

    def test_release_returns_the_reservation(self):
        raw = self.led.create_key("acme", 5_000)
        h = hash_key(raw)
        self.assertTrue(self.led.reserve(h, 4_000))
        self.assertFalse(self.led.reserve(h, 2_000))
        self.led.release(h, 4_000)
        self.assertTrue(self.led.reserve(h, 4_000))

    def test_raw_key_is_never_stored(self):
        raw = self.led.create_key("acme", 1_000)
        with open(self.led.path, "rb") as f:
            self.assertNotIn(raw.encode(), f.read())
        self.assertEqual(self.led.tenant_of(raw), "acme")
        self.assertIsNone(self.led.tenant_of(raw + "x"))
def reserve(self, key_hash: str, amount: int) -> bool:
    with self._lock:
        con = self._con()
        try:
            con.execute("BEGIN IMMEDIATE")
            spent, reserved, limit = con.execute(
                "SELECT spent_micro, reserved_micro, limit_micro FROM keys WHERE key_hash = ?",
                (key_hash,)).fetchone()
            if spent + reserved + amount > limit:
                con.execute("ROLLBACK")
                return False
            con.execute("UPDATE keys SET reserved_micro = reserved_micro + ? WHERE key_hash = ?",
                        (amount, key_hash))
            con.execute("COMMIT")
            return True
        except Exception:
            con.execute("ROLLBACK")
            raise

The reserve transaction uses BEGIN IMMEDIATE, so SQLite takes the write lock before the balance is read, and a lock in the process serialises threads. The ledger test fires 50 threads at a barrier and checks that exactly as many reservations succeed as the balance allows. The whole-gateway test fires 50 requests and checks three invariants: spent plus reserved never exceeds the limit, no reservation is left open, and the overshoot is zero. In the eval the provider pauses 50 real milliseconds so the reservations overlap; my reruns admitted 12 and refused 38, but that count is timing-dependent and only the invariants are claimed. An earlier version without the pause admitted anywhere from 15 to 50, which made zero overshoot a weak statement.

Milestone 8: reconciliation and the savings waterfall

A savings number is only believable if every stage of the waterfall is priced with the same usage numbers. My first waterfall priced the baseline with assumed token counts and the cache stages with measured approximations, and the comparison was meaningless. The fix was to price every request, in every stage, from the same usage pair: the approximate input tokens of its messages and the approximate output tokens of its answer.

The workload is 1,000 requests drawn from 42 questions with a Zipf-like weight. The stages are always-strong with no cache, then routing, then the exact cache, then the semantic cache; the numbers are below: about 163 thousand micro-dollars, 9.7 thousand, 750 and 400. The semantic stage helps only because the workload contains spacing variants of the same question. That is a weak test of semantic caching, and I say so.

workload micro-dollars: baseline 163405, +routing 9698, +exact 748, +semantic 407
semantic hits in workload: {'exact': 673, 'miss': 42, 'semantic': 285}

Reconciliation against provider-reported usage is not done: the fake provider reports exactly the approximation it was given, so it cannot disagree. It is a stretch goal.

Adversarial probes: what I tried to break, and what broke

I wrote thirteen probes against the security and money behaviour, and eleven of them found something real. The first five I wrote before I trusted the code; the other eight came from a second pass in which I read it as an attacker would. Nine found a bug that is now fixed and two found a limit that is documented. The probes are in evals/probes.py, and each prints a verdict. The output is in the block below, after the fixes.

def p3_semantic_cache_ignores_system_prompt():
    """Same last user message, different system prompt: does a semantic hit cross the prompt boundary?"""
    d = tempfile.mkdtemp()
    clock = FakeClock()
    led = Ledger(os.path.join(d, "l.db"))
    key = led.create_key("acme", 10_000_000)
    prov = FakeProvider("a", clock, reply=lambda last: "reply to: " + last)
    prov.default = ("ok", "x", 1.0)
    gw = Gateway(led, clock, {"cheap": [prov], "standard": [], "strong": []}, ExactCache(os.path.join(d, "c.db"), clock),
                 semantic=SemanticCache(threshold=0.9), retry=RetryPolicy(max_retries=0), rate=(1e6, 1e6))
    first = gw.handle(key, {"messages": [{"role": "system", "content": "Answer in English."},
                                          {"role": "user", "content": "What is the fee for class 10 in Karnataka?"}]})
    second = gw.handle(key, {"messages": [{"role": "system", "content": "Refuse all fee questions."},
                                           {"role": "user", "content": "What is the fee for class 10 in Karnataka?"}]})
    return ("FOUND" if second["cache"] == "semantic" else "not found"), \
        "second answer came from cache=%s with text %r" % (second["cache"], second["text"])

The probe above is the one that found a real leak. The semantic cache's partition used the prompt version and the last user message, but not the system prompt or earlier turns. A question asked under one system prompt could be answered from a cache entry written under another. The fix was to add a digest of everything except the final message to the partition key. The regression test is in tests/test_probes.py.

p1_key_in_error_or_trace                                       not found  raw key in spans: False
p2_tenant_spoofing_in_body                                     not found  second tenant got cache=miss
p3_semantic_cache_ignores_system_prompt                        not found  second answer came from cache=miss with text 'reply to: What is the fee for class 10 in Karnataka?'
p4_provider_exceeds_max_tokens                                 FOUND      spent 120001 micro against limit 1520; over_reservation=1
p5_non_integer_max_tokens_crashes                              not found  gateway error 400
p6_negative_max_tokens_makes_a_negative_reservation            not found  gateway error 400, reserved_micro=0
p7_unexpected_error_keeps_the_money_reserved                   not found  reserved_micro left=0, upstream text in error=False
p8_streaming_skips_the_rate_limit                              not found  2 of 10 streams admitted by a bucket of 2
p9_abandoned_probe_wedges_the_breaker                          not found  a new probe is allowed 1000 s later: True
p10_semantic_hit_ignores_max_tokens                            not found  second call (max_tokens 1000) got cache=miss
p11_one_tenants_bad_requests_open_the_breaker_for_everyone     not found  the other tenant was served; breaker is closed
p12_malformed_message_shapes_crash                             not found  unhandled exceptions: none
p13_lowercase_negation_passes_the_verifier                     FOUND      lower-case 'not' served the cached answer: True; upper-case 'NOT': False

The second pass found seven more, each reproduced before its fix. A max_tokens of minus a billion was priced as a negative reservation. A provider adapter that raised anything but ProviderError left its reservation booked for ever (159 micro-dollars stuck after one request) and sent its exception text, which can carry an upstream body, to the caller. Streaming skipped the rate limit (10 of 10 streams admitted by a bucket of 2). A half-open probe that never reported back kept the breaker half-open for good. A semantic hit ignored max_tokens. Three provider 400s from one tenant opened the breaker, and the next tenant got a 503. A message without content escaped as a KeyError. The fixes: parse_request checks every client field (max_tokens below 1 is a 400), a try around the tier call settles the reservation for zero and raises a generic 502, one _admit serves both paths, the breaker's probe expires, the semantic scope includes max_tokens, and a 4xx is not a provider failure. The HTTP layer caps the body at 64,000 bytes before reading it and turns any other exception into a generic 500. The "before" figures are from my run before the fixes; the block below is after.

Two properties hold by construction: the server's log_message does nothing, so the Authorization header never reaches an access log, and a key is looked up only by its SHA-256 digest. Probe 13 is the one I could not close.

The other results: the raw key does not appear in traces or errors, a tenant named in the body is ignored because the tenant comes from the key, and a non-integer max_tokens is now a 400. The fourth probe, a provider that bills 200,000 output tokens against a request that asked for 16, overspent the limit. The gateway detects that (the over_reservation counter goes up and the next request is refused) but cannot undo a bill: a limit, not a fix.

The table below is the regression test log, which includes the probe regressions and the concurrency tests.

...........................................
----------------------------------------------------------------------
Ran 43 tests in 7.162s

OK

Cost, latency and deployment

The gateway's own cost is the SQLite round trips, and my first version was too slow because it opened a connection for every call. One connection per thread plus write-ahead logging brought the overhead to single-digit milliseconds at p50: 8.2 ms at p50 and 12.5 ms at p95 in the printed run, and 7.5 to 9.4 ms and 8.4 to 13.9 ms across reruns, so the 10 ms p50 target is met narrowly and the 40 ms p95 target comfortably, on this machine only. A first profiling run showed 46 ms at p50; the printed files do not reproduce that figure, so it is a note and not a result.

:::key Verified 28 October 2026 Python 3.9 reached end of life on 31 October 2025, and 3.9.25 was its final release, according to PEP 596 on python.org, which I opened on this date. The project still runs on 3.9 because the reader's machine does, but a new project should target a supported version. Check python.org before you rely on that date. :::

Deployment would be a Dockerfile, a free-tier container host, a half-dollar budget on the demo keys, and an admin endpoint behind a secret. I did not deploy this project or check any host's limits; read the host's pricing page on the day.

Observability and failure analysis

Every request writes one span, holding tenant, cache result, tier, provider, cost and attempts, so a failure can be traced without reading logs. The sink here is an in-memory list; pointing it at the tracekit store is one argument.

The three failures worth writing up: the semantic-cache leak (found by sending one question under two system prompts), the overhead (a connection per call, fixed by keeping connections open) and the capitalised first word that made recall zero. Each has a cause, a fix and a before-and-after number.

How to present it: README numbers, the pitch, the resume bullet

A reviewer should find the failed gate above the fold, because a gateway that hides its failing numbers is the one people stop trusting. The README's results table has the five numbers from the scope section, each with its status: passed, failed, or not measured on this machine.

The 90-second pitch (ILLUSTRATIVE: use only numbers your own repository prints) has five beats: the problem (six services, one key, no budget), the design decision that mattered (reserve before the call, because a check-then-call race is the failure a reviewer tests), the measurement (50 concurrent requests, zero overshoot, eval invariants), the failure found (the semantic-cache leak from probe 3, or the negative reservation from probe 6), and the number that remains open (the semantic gate fails, and the router loses about 51 points).

The resume bullet follows the formula in the first lesson of this topic, and every number in it must come from a file in the repository. An ILLUSTRATIVE example, not a claim: built a multi-provider LLM gateway (Python, SQLite, stdlib HTTP) with routing, exact and semantic caching, circuit breakers and atomic per-key budgets; 50 concurrent requests cannot overspend a budget (checked by a 50-thread test); a semantic-cache gate that is reported as failing until a verifier passes.

The five interview questions, with outlines:

Pitfalls and stretch goals

The pitfalls are the ones this project hit, and the stretch goals are the ones it did not reach. Caching a sampled or personalised answer. Counting a provider 4xx as a provider outage. A retry storm with no jitter and no retry budget. Failing over to a model with a different tool format. Counting savings without a quality column. A check-then-act budget race. A stream buffered so the first token waits for the whole answer. A semantic cache whose gate checks precision and not recall. A demo key with no spend cap.

Stretch goals, none done: a learned router trained on judge scores (the judgekit project in aip2); hedged requests; per-tenant data-residency routing; prompt-cache-aware routing; reconciling the ledger with provider-reported usage, which needs a real provider.

How to say it in the interview

A gateway is the infrastructure side of LLM work, so I measure it like infrastructure. The reliability story is a fake provider that fails on command. The cost story is a ledger that reserves money before each call. The cache story is a set of rules about what may never be cached. Where a quality gate fails, I say so in the README, because a gate that fails and blocks the feature is working.

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons