Cruxly › AI engineering › Fine-Tuning and Model Adaptation

LoRA and QLoRA: What Actually Changes During Fine-Tuning

all-linear targeting, the 10x learning rate, and serving many adapters on one base model

Prompt: a candidate says "I fine-tuned a 7B model with LoRA on our support data, targeted the attention layers like everyone recommends, used the learning rate we'd use for a full fine-tune, and it underperformed a full fine-tune by a wide margin — so I concluded LoRA just doesn't work as well." Evaluate that conclusion, then explain what LoRA actually does mechanically and how you'd run the experiment correctly.

40 minutes first. The conclusion is wrong, and the two specific reasons why are both checkable facts, not opinions — which is exactly why an interviewer can tell in thirty seconds whether you know them.

What LoRA actually does, mechanically

Freeze the base weights. Train a low-rank decomposition alongside each targeted weight matrix. For a weight matrix W (say, a d × d projection inside an attention or feedforward block), full fine-tuning updates all of W directly. LoRA instead freezes W completely and adds two small matrices, B (d × r) and A (r × d), where r — the rank — is far smaller than d. The effective weight at inference time is W + (α/r)·B·A (the convention of the LoRA paper and the PEFT library; B starts at zero so training begins exactly at the base model), and only A and B are trained; W never moves. Because r is small (commonly 4 to 64 against a d in the thousands), B·A has orders of magnitude fewer trainable parameters than W itself, which is the entire memory and compute saving.

This is why LoRA adapters are small, swappable files, not new models. The adapter is just A and B for each targeted matrix — typically tens of megabytes for a 7B-class model — and applying it is addition, not replacement. Removing it (setting the adapter's contribution to zero) exactly recovers the original base model, which is a property full fine-tuning does not have: there is no clean way to "subtract" a full fine-tune back to the base.

Correction one: all-linear targeting beats attention-only

The old default — LoRA adapters only on the attention projections — underperforms. Many early tutorials and configs attached adapters only to the query/key/value/output projections inside attention, leaving the feedforward (MLP) layers untouched. The evidence against that is not new:

• QLoRA (Dettmers et al., May 2023) already found that adapters on all linear layers were needed to match full fine-tuning quality.
• Thinking Machines' "LoRA Without Regret" (2025) reconfirmed it at larger scale: attention-only LoRA underperforms even at matched parameter counts, and the MLP layers are where adaptation capacity matters most.

The default persisted in copy-pasted configs long after the papers, which is exactly why it still shows up in interviews.

Why this was missed for so long is worth saying, because it's the actual lesson. Attention gets the mechanistic spotlight — it's the part of the architecture people draw diagrams of — so it became the default place to intervene. But the MLP blocks hold the majority of a transformer's parameters and, it turns out, a proportional share of what needs to adapt for a downstream task. An adapter that only touches attention is trying to shift the model's behavior through a narrower channel than the model actually offers.

The checkable takeaway. If someone describes a LoRA setup as "targeting q_proj, k_proj, v_proj, o_proj" and stops there, that is the old default, and the fix is one line in a config: target every Linear layer the framework can enumerate, not a hand-picked subset. Frameworks that expose a target_modules="all-linear" (or equivalent) option exist specifically because this correction is now mainstream enough to warrant a first-class flag.

Correction two: the ~10x learning rate rule

LoRA wants a learning rate roughly an order of magnitude higher than the equivalent full fine-tune. This is the single most common reason someone's "LoRA underperformed full fine-tuning" result is an artifact of their experiment rather than a property of LoRA. A full fine-tune of a 7B-class model commonly uses a learning rate on the order of 1e-5 to 2e-5; the equivalent LoRA run wants something on the order of 1e-4 to 2e-4 — roughly 10x higher.

Be honest about what's known and what isn't. The ~10x ratio is an empirical finding (reported consistently, including in Thinking Machines' 2025 study), not something derived from first principles. What you can say precisely is how LoRA's setup shapes it:

• The update is scaled by α/r. The adapter's contribution is (α/r) × B × A. With the standard α/r scaling, the optimal learning rate comes out roughly independent of rank — which is why one "~10x" rule works across ranks instead of needing re-tuning for each.
• B starts at exactly zero. With A random and B zero, the adapter begins as a no-op, and early training is dominated by how fast B moves away from zero — a full-fine-tune-scale learning rate leaves it barely moving.
• LoRA is less tolerant of very large batch sizes than full fine-tuning in the same study, so a recipe copied from a large-batch full fine-tune can under-power it a second way.

The practical rule: sweep the learning rate for LoRA separately, starting around 10x the full-fine-tune value. A LoRA run at the full-fine-tune rate is technically training but barely getting anywhere, which looks exactly like "LoRA has a lower ceiling" if you don't know to check.

The candidate's stated experiment, diagnosed precisely. Attention-only targeting and a full-fine-tune-scale learning rate are two independent ways to under-power a LoRA run, and the candidate in the prompt made both mistakes at once — so "LoRA underperformed by a wide margin" is not surprising, it's closer to expected. The correct experiment: all-linear targeting, a learning rate roughly 10x the full-fine-tune value, then compare — and only that comparison is evidence about LoRA's actual ceiling relative to full fine-tuning.

Rank selection and the memory math

Rank is the practical knob, and it has a well-understood shape. r=4 gives minimal memory overhead and moderate quality — usable for narrow, simple adaptations. r=8 is the good-balance default for most tasks. r=16 earns its cost on genuinely complex task adaptation. r=32 to r=64 approaches full-fine-tune quality, with diminishing returns above that range — past roughly r=64, you are paying for parameters that a full fine-tune would use more efficiently, and at that point the honest question is whether you should just do a full fine-tune.

Memory scales linearly with rank, and the numbers are concrete enough to quote in an interview. Each targeted matrix of shape d_in × d_out gets rank × (d_in + d_out) adapter parameters (A is r × d_in, B is d_out × r). Worked for a Llama-7B-shaped model (hidden 4096, MLP 11008, 32 layers), with all linear layers targeted — the default the previous sections recommend — stored in bf16:

Attention-only targeting (the four 4096×4096 projections: 8 × 8,192 × 4 × 32 ≈ 8.4M parameters at rank 8, about 17MB in bf16) is roughly 40% of the all-linear size — which is where the older, smaller figures in many tutorials come from. Either way, a rank-8 adapter is a tiny fraction of the 14GB base model, which is the entire reason LoRA makes multi-adapter serving feasible.

QLoRA: quantize the base, keep the adapter precise. QLoRA quantizes the frozen base model's weights to 4-bit (NF4 — a data type designed for the roughly-normal distribution of trained neural network weights) and trains the LoRA adapter in higher precision (bf16 or fp32) on top of that quantized base. The result is near-LoRA quality at a fraction of the base model's memory footprint — a 7B model's frozen weights shrink roughly 4x compared to fp16 — which is what makes fine-tuning a 7B-13B class model feasible on a single consumer-class GPU. The trade is real and worth naming: every forward pass now pays a dequantization cost (converting 4-bit weights back to a usable precision on the fly), so QLoRA training is meaningfully slower per step than full-precision LoRA, even though it uses far less memory. Use it when memory, not time, is the binding constraint.

Serving many adapters: why this matters for a SaaS with many tenants

The production case LoRA was built for: one base model, many small customers, each wanting a customized model. A SaaS with a thousand tenants, each wanting the model fine-tuned on their own tone or domain data, cannot afford a thousand full copies of a 7B-13B model in memory or storage. It can easily afford a thousand rank-8 all-linear adapters at ~40MB each — roughly 40GB total, against ~14TB for a thousand full 7B model copies — swapped onto one shared base model per request.

S-LoRA-style systems make this a serving architecture, not just a storage trick. The core idea is unified paging: manage adapter weights and the KV cache in the same paged-memory scheme, so the system can hold an adapter catalog larger than any single GPU's memory and page the right adapters in as requests for that tenant arrive, rather than requiring every adapter a request might need to already be resident. This connects directly to the KV-cache memory management covered in ai-serving — the same memory-bound-regime problem (many small, cheap operations competing for the same limited high-bandwidth memory) shows up again here, and the fix is the same family of idea: page it, don't pre-allocate it all.

The compute pattern is different enough to need its own kernels. Batching requests that each want a different adapter is not the same problem as batching requests against one shared model — the base model computation can still batch normally, but the per-adapter A×B correction is small, cheap, and memory-bound rather than compute-bound at typical batch sizes. Kernels specialized for this regime (computing many small, different low-rank corrections efficiently across a batch) are what makes serving hundreds of concurrently-active adapters practical rather than a theoretical possibility.

How to say it in the interview

"That conclusion is very likely just two configuration mistakes wearing the costume of a fundamental limitation. Attention-only targeting is the old default and it underperforms — current research shows targeting all linear layers, attention and the MLP blocks, gives meaningfully better results for about the same parameter budget. And LoRA needs a learning rate roughly 10x higher than an equivalent full fine-tune would use, so reusing the full-fine-tune learning rate systematically under-trains the adapter. Before concluding LoRA doesn't work, I'd re-run it targeting every linear layer at a 10x learning rate, because both of those are checkable, specific, and currently the most common reason someone's LoRA experiment looks worse than it should."

Checklist

• Explained LoRA mechanically before discussing tuning choices
• Diagnosed the candidate's underperforming LoRA run as two specific, checkable mistakes
• Named all-linear targeting as the current best practice, not attention-only
• Stated the ~10x learning rate rule and explained why, not just the multiplier
• Gave concrete rank-to-memory numbers rather than a vague 'higher rank is better'
• Named where rank selection stops paying off
• Explained QLoRA's trade precisely: quantized base, higher-precision adapter, and the dequantization cost
• Connected multi-adapter serving to a concrete SaaS scenario with real numbers
• Named unified paging and small-batch memory-bound kernels as the two things that make multi-adapter serving actually work

Related lessons

• Fine-Tune, RAG, or Prompt: The Decision Nobody Wants to Make Twice
• Training Data: Dedup, Contamination, and Knowing It Actually Worked
• RLHF, DPO and RL with Verifiable Rewards: How Post-Training Actually Works
• Distillation and Synthetic Data: Teachers, Students and the Quality Loop