Cruxly › AI engineering › Inference Serving

Autoscaling a GPU Fleet Without Burning the Budget

cold starts, queue-depth scaling, spot draining, and the break-even math

Prompt: you are serving a fine-tuned 70B model plus fourteen customer LoRA adapters. Traffic is business-hours-shaped — roughly 20:1 peak to trough — and the current fleet is provisioned for peak and costs a fortune overnight. Design the autoscaling, then justify self-hosting at all against simply paying a token-metered API provider. Show the break-even.

40 minutes first. CPU-style autoscaling rules do not merely underperform here; they actively make things worse, and the build-vs-buy half of the question is where most candidates have no numbers at all.

Why CPU-style autoscaling fails on GPUs

Three assumptions break, and they break in different directions.

The scaling signal is wrong. CPU utilization is meaningless on an inference server — it reads low while the GPU is saturated, and GPU utilization reads ~100% during bandwidth-bound decode whether you are serving 4 requests or 60. Scaling on either produces a fleet that either never scales up or never scales down. The signals that actually track saturation are queue depth, queue wait time, and KV cache utilization, in roughly that order of practicality, because all three are direct measures of "requests are waiting for a resource" rather than proxies for it.

Cold start is minutes, not seconds. Be specific about where the time goes for a 70B fp16 model: acquiring the instance (30 s to several minutes, or never during a crunch), pulling a multi-gigabyte container image, downloading ~140 GB of weights from object storage (at 1 GB/s that is 140 seconds, and cross-region it can be far worse), loading into VRAM across 8 GPUs, then engine warm-up — CUDA graph capture, kernel autotuning, and for TensorRT-LLM potentially an engine build. Five to fifteen minutes end to end is a realistic range. A spike that lasts three minutes is entirely over before your new capacity serves its first token, which means reactive autoscaling does not solve spikes at all; it solves sustained load changes.

The increment and the price are lumpy. You cannot add 15% of an 8-GPU node. You add a node, at a fixed hourly rate, billed whether or not it is busy. An on-demand 8×H100 node runs in the tens of dollars per hour — so an idle overnight node is a four-figure monthly line item doing nothing. There is also no burst credit, no shared-tenancy slack, and in a supply crunch no guarantee the capacity exists.

The consequence to state plainly: GPU autoscaling is predictive and queue-absorbing, not reactive. Scale on schedule and forecast for the shape you know (business hours, campaign launches), keep enough warm capacity to absorb the spike duration you cannot forecast, and use the queue — bounded, with load shedding by class — as the shock absorber during the minutes it takes real capacity to arrive.

Scale-to-zero versus warm pools, and how to cut the cold start

Scale-to-zero is correct for exactly one traffic shape: genuinely spiky, latency-tolerant, low-duty-cycle workloads. Internal tools, batch jobs, dev environments, and per-customer models that see a handful of requests a day. The first request after a scale-down pays the entire cold start, so the honest framing is "this endpoint's SLO includes a documented several-minute cold-start penalty on the first request after idle" — which is acceptable for an internal report generator and unacceptable for a customer-facing chat box.

For anything interactive, keep a warm floor and scale above it. Size the floor from the trough of your traffic curve plus enough headroom to absorb the spike duration you cannot predict — for the 20:1 business-hours shape in the prompt, that might be 1 warm node overnight against 12 at peak, plus a small pre-warmed buffer during business hours so a single node's worth of surge does not hit the queue.

Attack the cold start directly; several minutes is not a fixed cost. Pre-bake the model weights into the machine image or a pre-attached volume snapshot rather than downloading from object storage at boot — this is usually the single biggest win, often taking minutes off. Cache the container image on the node pool. Keep instances stopped rather than terminated where the provider supports it, so you skip acquisition and image pull. And pre-warm the engine: capture CUDA graphs and run a synthetic request before marking the pod ready, so the first real request is not also the one that pays for autotuning. With those in place, 5-15 minutes can become 60-90 seconds, which changes what autoscaling can do for you.

Use the cheapest capacity for the least urgent work, deliberately. A standing reservation or committed-use discount covers the warm floor at 40-60% off on-demand, since you will run it 24/7 regardless. On-demand covers the predictable daily ramp. Spot covers batch and overflow. Matching the commitment term to the confidence you have in the load is the cost lever that dwarfs most engine tuning.

Scale down slowly and asymmetrically. Scale up on a short window (a minute of elevated queue depth) and down on a long one (fifteen minutes of sustained low load), because a wrong scale-up costs one node-hour and a wrong scale-down costs a full cold start plus an SLO breach. Never scale down a node with active requests: drain it — stop admitting new work, let in-flight generations finish (bounded by a timeout, since a long generation can run for minutes), then terminate.

Multi-model serving and LoRA hot-swapping

Fourteen customer adapters is the case that makes or breaks the economics of this fleet. A separate full 70B deployment per customer means fourteen copies of 140 GB of identical weights and fourteen node groups sitting mostly idle — for most businesses that is immediately unaffordable, and it is the naive answer the prompt is constructed to punish.

LoRA changes the arithmetic completely. A LoRA adapter is a pair of low-rank matrices per targeted weight matrix. At rank 16 on a Llama-70B-shaped model in bf16:

• All linear layers targeted (attention + MLP, the current recommended default): about 207M parameters ≈ ~400 MB per adapter.
• Attention projections only: about 65M parameters ≈ ~130 MB.

Either way it's well under 1% of 140 GB of base weights. Load the base model once and hold many adapters in the same VRAM, selecting per request. Fourteen all-linear adapters cost roughly 6 GB on top of one deployment instead of fourteen deployments.

Multi-LoRA batching is the part worth naming precisely. Modern engines can serve requests using different adapters in the same batch, applying each sequence's adapter during the forward pass rather than grouping requests by adapter and running separate batches. That matters because per-adapter batching would fragment your batch — fourteen adapters each with 3 concurrent requests gives you fourteen batches of 3 instead of one batch of 42, which by the batching lesson's arithmetic is most of your throughput gone. If the engine cannot do mixed-adapter batching, a long-tail multi-tenant workload will disappoint you badly.

Hot-swapping and the tail. Keep hot adapters resident in VRAM, keep the long tail in host RAM with an LRU policy, and load on demand — an adapter load is hundreds of milliseconds, which is a TTFT hit on a cold adapter's first request but not a cold start. Bound the number of resident adapters explicitly, because adapter memory competes with the KV pool, and an unbounded adapter cache quietly reduces your concurrency ceiling.

Two operational cautions worth raising unprompted. First, adapter identity must be part of the prefix-cache key — sharing prefix blocks across adapters would serve one customer's cache to another customer's model, which is both a correctness and an isolation bug. Second, distinct base models cannot share anything: a fleet serving a 7B and a 70B needs separate pools, and the routing layer, not the engine, decides which pool a request belongs to.

Spot capacity, draining, and surviving preemption

Spot or preemptible GPU instances run at roughly 60-90% off on-demand, and they can be taken away with 30 seconds to 2 minutes of notice. The question is not whether to use them but which traffic can tolerate them, and the answer follows request class exactly: batch and offline work, yes, enthusiastically; overflow above the warm floor, cautiously; the warm floor itself, never.

Handle the preemption notice as a real code path, not a hope. Subscribe to the provider's termination signal, and on receipt: immediately deregister from the load balancer so no new requests arrive, stop admitting from the local queue, push queued-but-unstarted requests back to a shared queue so a sibling node can take them, and let in-flight generations finish if they fit in the notice window. Generations that will not finish in 30 seconds are the hard case — a request 200 tokens into a 2,000-token response has no good ending — so either fail it with a retryable error and let the client resubmit, or, if you have built KV-cache migration, ship its blocks to another node. The honest interview answer is that most production systems do the former and accept a small preemption-induced error rate on spot-served traffic, budgeted into the SLO.

Because nodes will vanish, keep the serving layer stateless above the KV cache. Conversation history lives in a shared store, not in a node's memory, and a client reconnect must be able to resume against a different node. This is the same statelessness discipline as any web tier, with the wrinkle that the KV cache genuinely is node-local state that cannot be replicated cheaply — so the design goal is making its loss recoverable (re-prefill on a new node) rather than preventing it.

Spread across zones and instance types, and mix capacity pools. A fleet pinned to one instance type in one availability zone can be preempted entirely at once when that pool is reclaimed, which converts a cost optimization into an outage. Accept several compatible accelerator types where your engine supports them, and keep the on-demand floor in a different pool from the spot overflow.

Build versus buy: the break-even you should be able to compute live

Do the arithmetic rather than expressing a preference. Here is the shape of it.

Take an 8×H100 node at about $32/hour on demand (Lambda lists $3.99 per H100 GPU-hour) — about $23,300/month running continuously, or an assumed $11,000-13,000/month with a committed-use discount (an illustrative range; discounts are negotiated). A well-batched FP8 70B on that node can sustain on the order of 10,000 aggregate output tokens/sec — but treat that as a placeholder and benchmark your own traffic, because it swings several-fold with batch size, context length and prefix-cache hit rate.

• At 100% utilization: 10,000 × 3,600 × 24 × 30 ≈ 26 billion output tokens/month.
• At a realistic 40% across a business-hours curve: about 10 billion output tokens/month for ~$13,000 ≈ $1.25 per million output tokens — before engineering cost.
• Don't forget input tokens. Prefill also consumes GPU time; a RAG workload with 6,000 input tokens per 300 output tokens spends much of the box on prefill, and the API bills those input tokens separately too.

Now compare against a token-metered API serving a comparable open-weights 70B, which in October 2026 is listed at roughly $0.32–$0.60 per million output tokens depending on the provider (DeepInfra: Llama 3.3 70B Instruct Turbo $0.32; Groq: the 120B-parameter gpt-oss-120b $0.60), with zero idle cost and zero operational burden. Write it as a formula so the numbers can be swapped. With H dollars per node-hour, T sustained output tokens/sec and utilization u, the cost per million output tokens is H / (3,600 × T × u / 1,000,000). With H = $18 (the committed rate above), T = 10,000: u = 0.4 gives $1.25, and the break-even utilization against an API price p is u* = 0.5 / p (p in dollars per million). At p = $0.60 that is 83%; at p = $0.32 it is 156%, above 100%, so self-hosting never wins on cost alone at that price.

:::key Verified October 2026 Read on 31 October 2026: Lambda's pricing page lists NVIDIA H100 SXM 80 GB on demand at $3.99 per GPU-hour ($31.92 for eight); DeepInfra's pricing page lists Llama-3.3-70B-Instruct-Turbo at $0.10 input / $0.32 output per million tokens; Groq's models page lists gpt-oss-120b at $0.15 / $0.60. Hyperscaler on-demand rates (AWS, Google Cloud) are not covered here and are often higher; and the 10,000 tokens/sec throughput and the committed-use rate are assumptions, not measurements. GPU and token prices change often, so treat H, T and p as inputs to re-price from two current providers before using the result. :::

Even the top of that API range ($0.60) undercuts $1.25. At 40% utilization, self-hosting is not cheaper — it's likely more expensive. Saying this out loud is the credibility move; a candidate who claims self-hosting is always cheaper has not run the numbers.

Self-hosting wins when one of these is true, and you should name which one applies:

And name the costs that never appear in the per-token comparison, because leaving them out is how self-hosting proposals get approved and then regretted: one to two engineers' sustained time on the serving layer, on-call for GPU failures and driver issues, model-update and re-benchmarking cycles, evaluation infrastructure to gate quantization changes, and the multi-month lead time on reserved capacity. A defensible answer is usually a hybrid: self-host the high-volume steady workload and the fine-tuned adapters on committed capacity, burst to an API provider above your own fleet's ceiling — which conveniently makes the provider-failover machinery from the backend-infrastructure topic your overflow path rather than a separate system.

The trigger to state for revisiting the decision: recompute the break-even whenever sustained utilization crosses 60%, whenever provider per-token pricing moves by more than about 30%, or whenever a new accelerator generation changes tokens-per-dollar — all three happen on a timescale of quarters, not years, so this is a standing review, not a one-time architecture decision.

How to say it in the interview

"Standard autoscaling assumes you can add 10% more capacity in 30 seconds for a proportional cost. With GPUs, none of the three assumptions hold: the increment is a whole accelerator at a fixed hourly rate, the startup is minutes because you are pulling a 140GB checkpoint, and during a capacity crunch the instance may not exist to buy at any price. So the design has to absorb spikes with queueing and warm capacity rather than reacting to them."

Checklist

• Scaled on queue depth, queue wait or KV utilization — never CPU or GPU percent
• Broke the cold start into its actual stages with a time for each
• Concluded that GPU autoscaling must be predictive and queue-absorbing, not reactive
• Scoped scale-to-zero to latency-tolerant workloads and documented the first-request penalty
• Attacked cold start directly by pre-baking weights and pre-warming the engine
• Made scale-up fast and scale-down slow, with draining before termination
• Served many LoRA adapters on one base model instead of one deployment each
• Required mixed-adapter batching and bounded the resident adapter count
• Made adapter identity part of the prefix-cache key
• Handled the spot preemption notice as a real drain path, including unfinishable generations
• Computed a build-vs-buy break-even with utilization in it, not just a per-hour rate
• Named the non-cost drivers and the hidden operational costs on both sides

Related lessons

• GPU Memory Math: Weights, KV Cache, and What Actually Fits
• Quantization Methods Compared: INT8, GPTQ, AWQ, FP8, FP4 and KV-Cache Quantization
• FlashAttention and Why Attention Is a Memory-Traffic Problem
• Continuous Batching and the Scheduler
• TTFT, TPOT, and the Latency Budget
• Mixture-of-Experts and Long-Context Serving