Parameter-efficient fine-tuning

Chapters 13 and 14 left you with the recipes for post-training. Take a base model, run SFT to teach it the chatbot format, layer DPO or RLHF on top to teach it quality, and you have a useful assistant. The recipes are clear; the problem is that doing them at the parameter scale of modern models is operationally brutal. Full fine-tuning of a $70$B-parameter model holds parameters, gradients, and AdamW’s two optimizer moments in memory simultaneously, roughly $840$ GB before activations. That is $10\times$ A100s at minimum. Out of reach for most teams.

Most production fine-tuning is not done that way. Instead, teams freeze the base model (no gradient, no optimizer state for those parameters) and train only a small subset, typically $0.1\%$ to $1\%$ of the base model’s parameters. The dominant technique is LoRA (Low-Rank Adaptation), introduced by Hu et al. (2021). The hypothesis: fine-tuning updates have low intrinsic dimensionality. The matrix of changes $\Delta W = W_{\text{ft}} - W_{\text{base}}$ can be well approximated by a low-rank decomposition $\Delta W \approx BA$. Train $B$ and $A$, both small. Leave $W$ frozen and large.

Combined with QLoRA (Dettmers et al. 2023), which adds $4$-bit NF4 quantization of the base model to LoRA training, you can fine-tune a $70$B model on a single $48$ GB GPU. This is what made open-source post-training explode after early $2023$. This chapter walks through LoRA mechanics, the broader PEFT family (adapters, prefix tuning, (IA)³), QLoRA’s tricks, and the modern variants (DoRA, VeRA, AdaLoRA, MoLoRA). It is the practical-engineering counterpart to Chapters 13 and 14’s algorithmic content.

The setup: full fine-tuning is expensive

The memory cost

A 70B-parameter model in BF16 occupies $140$ GB just for the weights. Full fine-tuning needs to hold three more things alongside the weights. Gradients require another $140$ GB (BF16, one float per parameter). Optimizer state, AdamW’s first and second moments, typically kept in FP32 for stability, adds $4 \times 2 \times 70 \times 10^9 = 560$ GB. Activations for backpropagation depend on whether activation checkpointing is enabled and on batch size; figure tens to hundreds of additional GB.

The arithmetic is uncomfortable. Weights plus gradients plus optimizer state alone is $140 + 140 + 560 = 840$ GB. An A100 has $80$ GB of HBM; an H100 also has $80$ GB. Full fine-tuning of a $70$B model needs at least $10$ A100s or H100s before you even budget for activations and communication buffers. FSDP from Chapter 9 helps by sharding parameters, gradients, and optimizer state across GPUs, but it does not reduce the total memory budget. You still need ten GPUs’ worth of HBM, just split across the cluster. For a team with one or two GPUs, full fine-tuning of frontier-scale models is not an option you can engineer around.

The opportunity

The key observation is that most parameters do not need to change much during fine-tuning. Pre-training spent trillions of tokens learning useful features, syntax, world knowledge, the residual-stream representations explored in Chapter 5. Fine-tuning typically teaches the model to combine and select among those features differently, not to relearn them. The “combination knob” has fewer degrees of freedom than the underlying feature library.

If that observation is right, then the path forward is direct: freeze the base model entirely, and train only a small subset of additional parameters that captures the combination updates. Memory drops by roughly $10\times$, because the base weights need no gradient or optimizer state: they just sit in HBM as constants during the forward pass. That is the PEFT promise, and the rest of the chapter is about cashing it in.

The low-rank hypothesis

Intrinsic dimensionality

The empirical claim that fine-tuning updates are small in a useful sense predates LoRA. Aghajanyan et al. (2020) studied fine-tuning of language models by an unusual experiment: rather than train all $N$ parameters directly, they parameterized the update through a random projection from a much smaller $d$-dimensional space and trained only those $d$ values. They found that most fine-tuning effect was recoverable with $d$ on the order of $10^4$ to $10^5$, orders of magnitude smaller than $N$, which for a $70$B model is $7 \times 10^{10}$. The effective dimensionality of the fine-tuning update is tiny relative to the parameter count.

The intuition is that pre-training learned a language model, a richly structured object encoding syntax, world knowledge, and reasoning patterns into billions of parameters. Fine-tuning does not relearn that language model. It teaches the model to use the language model differently, to behave as the assistant side of a conversation, to refuse harmful requests, to prefer one phrasing over another. The space of those use policies is much smaller than the space of all possible parameter configurations. There is a lot of overparameterization in the base model relative to what fine-tuning needs to change.

This is a useful framing but not a theorem. It is an empirical regularity that holds for most fine-tuning tasks studied so far. Section 5 returns to the cases where it does not hold.

Low-rank decomposition

LoRA (Hu et al. 2021) turns the intrinsic-dimensionality observation into a concrete parameterization. For each weight matrix $W \in \mathbb{R}^{d \times k}$ in a transformer, hypothesize that the fine-tuning update $\Delta W$ has intrinsic low rank and approximate it as a product:

$\Delta W \approx B A, \quad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \quad r \ll \min(d, k)$

$A$ is a “down-projection” from the input dimension $k$ to a small bottleneck $r$; $B$ is an “up-projection” from $r$ back to the output dimension $d$. Their product $BA$ is a rank-$r$ matrix of the same shape as $W$, but parameterized by only $r(d+k)$ values instead of $d \cdot k$.

The savings are striking. For $d = k = 4096$ (a typical attention projection in a $7$B-class model) and $r = 16$:

• Full: $4096 \times 4096 = 16{,}777{,}216$ parameters per matrix.
• LoRA: $16 \times (4096 + 4096) = 131{,}072$ parameters per matrix.
• A $128\times$ reduction per weight matrix.

Aggregated across a $7$B transformer with LoRA on all attention projections, the trainable parameter count typically lands at $0.1\%$ to $1\%$ of base model parameters. For a $70$B model that is $50$–$500$ MB of trainable parameters against $140$ GB of frozen base. Gradients and optimizer state only need to cover the trainable subset, which is where the order-of-magnitude memory savings come from.

LoRA mechanics

The forward pass

For a weight matrix $W_0 \in \mathbb{R}^{d \times k}$ in the base model, LoRA modifies the forward pass to compute two parallel terms and sum them. The base path uses the frozen weight $W_0$; the LoRA path uses the trainable factors $A$ and $B$ together with a scaling factor $\alpha/r$:

$h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B A x$

$W_0$ is the frozen pre-trained weight matrix; it never receives a gradient and never enters the optimizer state. $A$ and $B$ are the trainable low-rank factors. $r$ is the rank (typically $8$ to $32$). $\alpha$ is a scaling factor (typically $\alpha = r$ or $\alpha = 2r$, discussed in the next section). The forward pass is two matrix multiplications in parallel; the LoRA path is cheap because $r$ is small.

Initialization

The two factors are initialized asymmetrically. $A$ is initialized with small random Gaussian noise (the original paper’s released implementation uses Kaiming-uniform). $B$ is initialized to zero. The asymmetry is the point. Because $B = 0$ at step $0$, the LoRA contribution $BAx$ is zero everywhere, and therefore $\Delta W = 0$ at the start of training. The model behaves identically to the base model on step zero. Training updates $B$ (and $A$) away from this neutral starting point.

The zero-init of $B$ is load-bearing. Without it, for instance, if both $A$ and $B$ were initialized with random Gaussians, the first forward pass would inject a non-trivial random perturbation into every weight matrix of the network at once. Pre-trained representations would be corrupted before a single gradient step. Starting from $\Delta W = 0$ guarantees that LoRA training begins exactly where the base model is, and that the optimization moves the model away from there along directions selected by the gradient on real fine-tuning data.

Merging vs adapter-mode

At inference, LoRA gives you two options for how to evaluate the model. The first is to merge the adapter into the base weight matrix once at the end of training: compute $W_{\text{merged}} = W_0 + (\alpha/r) B A$ and store the result. From then on, the forward pass uses $W_{\text{merged}}$ directly; there is no LoRA path to evaluate at inference, and the model is computationally indistinguishable from a base model of the same shape. Zero inference overhead. This is the right choice for production deployment when the adapter is fixed.

The second option is adapter-mode inference. Keep $W_0$, $A$, and $B$ separate in memory; evaluate both terms on every forward pass. The cost is a small constant overhead per layer, the LoRA matmul against $r$-dimensional bottlenecks is cheap, but it is not free. The benefit is that adapters are swappable per request. A serving stack can load one base model and many adapter checkpoints, then route incoming requests to whichever adapter the request needs. Modern inference engines (vLLM, TGI) support multi-LoRA serving in exactly this mode, and it is increasingly common when one base model powers dozens of fine-tuned personas or vertical specializations.

Most teams use merging for production performance and keep adapter-mode for experimentation and multi-tenant serving. The choice is operational, not algorithmic.

LoRA forward pass. $W_0$ is the frozen pre-trained weight; $A$ and $B$ are trainable low-rank factors. $r$ is the rank (typically $8$–$32$); $\alpha$ is a scaling factor (typically $\alpha = r$ or $\alpha = 2r$). At step zero, $B = 0$ so $\Delta W = 0$, the model is identical to the base.

The runnable cell below builds the LoRA forward pass and verifies that at step $0$, with $B = 0$, the LoRA output exactly equals the base model output. After training (simulated by drawing a non-zero $B$), the outputs diverge.

import numpy as np

def lora_forward(x, W_base, A, B, alpha=16, r=8):
    """
    LoRA forward pass.

    x:      (batch, d_in)
    W_base: (d_in, d_out), FROZEN base weight
    A:      (r, d_in)    , TRAINABLE down-projection
    B:      (d_out, r)   , TRAINABLE up-projection
    alpha:  scaling factor (typically alpha = r or 2r)
    """
    base_out = x @ W_base                         # (batch, d_out)
    lora_out = x @ A.T @ B.T * (alpha / r)        # (batch, d_out)
    return base_out + lora_out

# Setup
np.random.seed(0)
d_in, d_out, r = 512, 768, 8
batch = 4

W_base = np.random.normal(0, 0.02, (d_in, d_out))
A = np.random.normal(0, 0.01, (r, d_in))   # Gaussian init for A
B = np.zeros((d_out, r))                    # ZERO init for B

# At step 0: LoRA should contribute nothing
x = np.random.normal(0, 1, (batch, d_in))
out_step0 = lora_forward(x, W_base, A, B)
out_base = x @ W_base
print(f"At step 0, LoRA output == base output: {np.allclose(out_step0, out_base)}")
print("(This is why we zero-init B. Without it, the first forward pass")
print(" would corrupt pre-trained representations.)")

# After training: B is non-zero, LoRA contributes
B_trained = np.random.normal(0, 0.01, (d_out, r))
out_trained = lora_forward(x, W_base, A, B_trained)
print(f"\nAfter training (B != 0), LoRA != base: {not np.allclose(out_trained, out_base)}")

Implementing LoRA in PyTorch

The numpy version makes the mechanics explicit; production code wraps an existing nn.Linear instead of writing the forward pass out by hand. The pattern is a thin module that holds the frozen base layer plus its own trainable $A$ and $B$, and sums the two paths in forward, exactly as in (15.lora) :

import torch
import torch.nn as nn

torch.manual_seed(0)

class LoRALinear(nn.Module):
    """Wraps a frozen nn.Linear with a trainable low-rank adapter."""

    def __init__(self, base, r=4, alpha=8):
        super().__init__()
        self.base = base
        self.base.weight.requires_grad_(False)   # freeze the base layer
        if self.base.bias is not None:
            self.base.bias.requires_grad_(False)

        d_out, d_in = base.weight.shape
        self.A = nn.Parameter(torch.randn(r, d_in) * 0.01)   # Gaussian init
        self.B = nn.Parameter(torch.zeros(d_out, r))          # zero init
        self.scaling = alpha / r

    def forward(self, x):
        base_out = self.base(x)
        lora_out = (x @ self.A.T @ self.B.T) * self.scaling
        return base_out + lora_out

# A tiny two-layer "model", both linear layers wrapped in LoRA
d_model = 64
base = nn.Sequential(nn.Linear(d_model, d_model), nn.ReLU(), nn.Linear(d_model, d_model))
model = nn.Sequential(LoRALinear(base[0]), nn.ReLU(), LoRALinear(base[2]))

total = sum(p.numel() for p in model.parameters())
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total parameters:     {total}")
print(f"Trainable parameters: {trainable} ({100 * trainable / total:.1f}% of total)")

# The optimizer only ever sees the trainable adapter parameters
optimizer = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-2)

x = torch.randn(32, d_model)
target = torch.randn(32, d_model)
base_weight_before = base[0].weight.clone()

losses = []
for step in range(20):
    optimizer.zero_grad()
    loss = ((model(x) - target) ** 2).mean()
    loss.backward()
    optimizer.step()
    losses.append(loss.item())

print(f"\nLoss: {losses[0]:.4f} -> {losses[-1]:.4f}")
print(f"Base weight unchanged: {torch.equal(base_weight_before, base[0].weight)}")

Three lines carry the whole idea. self.base.weight.requires_grad_(False) removes the base weight from autograd entirely, so it accumulates no gradient and enters no optimizer state, exactly the memory argument from the start of this chapter, now enforced at the framework level rather than just asserted in prose. The optimizer is constructed from [p for p in model.parameters() if p.requires_grad], so it never even sees the base weight, no risk of an accidental update. And the loss drops over training (here from $1.15$ to $0.87$ on a random regression target, the actual numbers don’t matter) while base_weight_before and base.weight afterward stay bit-for-bit identical. In this toy two-layer network the trainable fraction is a double-digit percentage because the base layers are so small; on a real 7B-class transformer with the same r=4 on a handful of target modules, this fraction collapses to the sub-$1\%$ figures worked out numerically in the next section. Libraries like Hugging Face’s peft automate the wrapping across every target module in a real transformer, but the mechanism underneath is exactly this: swap nn.Linear for a wrapped version, freeze the base, hand the optimizer only the adapter parameters.

Practical LoRA: rank, alpha, target modules

Rank

The rank $r$ controls the capacity of each LoRA adapter. Typical values are $4$, $8$, $16$, $32$, and $64$; the most common starting point in modern recipes is $r = 8$ or $r = 16$. Higher rank means more trainable parameters per matrix, more memory for the adapter and its optimizer state, and slightly slower training. Lower rank means cheaper everything, at the cost of expressivity.

Empirically, returns diminish past $r = 16$ to $r = 32$ for most fine-tuning tasks. The original LoRA paper found that rank $1$ was often surprisingly effective on the GLUE and WikiSQL benchmarks they studied, a striking demonstration of how concentrated fine-tuning updates really are. Modern recipes use $r = 8$ to $32$ as the default range; ranks above $64$ rarely improve task performance enough to justify the cost. The rule of thumb: start at $r = 16$, and only push higher when you have evidence the task needs it.

Alpha and the scaling factor

The scaling factor $\alpha/r$ in $h = W_0 x + (\alpha/r) BAx$ controls the magnitude of the LoRA contribution at inference and training time. It is not a learning rate, gradients still flow into $A$ and $B$ at full magnitude. Rather, $\alpha/r$ acts as a constant multiplier on the LoRA path’s output. Higher $\alpha/r$ means the LoRA path contributes more strongly to the activations relative to the base path; lower $\alpha/r$ means the LoRA correction is more subtle.

Three conventions dominate. Setting $\alpha = r$ keeps the ratio at exactly $1$, no rescaling, the LoRA path is contributed at its raw magnitude. Setting $\alpha = 2r$ doubles the contribution; this is common in DPO and other preference-optimization recipes where the LoRA path is expected to drive a meaningful shift. Some implementations fix $\alpha = 16$ regardless of $r$, in which case the effective ratio varies (e.g., $\alpha/r = 2$ at $r = 8$, $\alpha/r = 1$ at $r = 16$, $\alpha/r = 0.25$ at $r = 64$). Mental model: think of $\alpha/r$ as a knob that controls how loud the LoRA correction is, relative to the frozen base.

Target modules

LoRA does not have to be applied to every weight matrix in the model. The three common patterns:

• Attention $W_Q$, $W_V$ only. The original LoRA paper’s setting. Minimum parameter overhead, roughly $0.1\%$ of base. Effective when the fine-tuning task is mild, light instruction following, modest style adaptation.
• Attention $W_Q$, $W_K$, $W_V$, $W_O$, all attention projections. The modern default. Roughly $0.3\%$ of base. The standard choice for most SFT and DPO LoRA recipes today.
• All linear layers, attention plus FFN up- and down-projections. Maximum coverage, roughly $0.5\%$–$1\%$ of base. The right choice when the task is complex and budget allows.

Empirically, covering more modules helps for harder fine-tuning tasks but eventually hits diminishing returns of its own. For most instruction tuning the attention-only or all-attention pattern is sufficient. For preference optimization and tasks requiring stronger behavioral shifts, covering FFN as well tends to help. As with rank, the default recommendation is the middle option: all attention projections, $r = 16$, $\alpha = 16$ or $32$. Tune from there.

The runnable cell below computes trainable parameter counts for a $7$B-class transformer under several LoRA configurations and compares against full fine-tuning. The reductions are large enough to change which GPUs you can use.

def full_ft_params(d_model=4096, n_layers=32, vocab=128000):
    """Approximate full transformer parameter count."""
    attn = 4 * d_model * d_model              # Q, K, V, O
    ffn = 8 * d_model * d_model               # FFN ~ 4x d_model expansion
    embed = 2 * vocab * d_model               # tied input/output
    return n_layers * (attn + ffn) + embed

def lora_params(d_model=4096, n_layers=32, rank=16, targets=4):
    """LoRA parameter count. targets = matrices per layer."""
    per_target = 2 * d_model * rank
    return n_layers * targets * per_target

# 7B-class model
d, L = 4096, 32

full = full_ft_params(d, L)
lora_qv = lora_params(d, L, rank=16, targets=2)
lora_qkvo = lora_params(d, L, rank=16, targets=4)
lora_all = lora_params(d, L, rank=16, targets=6)
lora_high_rank = lora_params(d, L, rank=64, targets=4)

print(f"7B-class transformer (d={d}, layers={L}):")
print(f"  Full fine-tuning:           {full/1e9:5.2f}B params")
print(f"  LoRA r=16, Q + V only:      {lora_qv/1e6:5.1f}M params ({100*lora_qv/full:.3f}% of full)")
print(f"  LoRA r=16, Q + K + V + O:   {lora_qkvo/1e6:5.1f}M params ({100*lora_qkvo/full:.3f}% of full)")
print(f"  LoRA r=16, all linear:      {lora_all/1e6:5.1f}M params ({100*lora_all/full:.3f}% of full)")
print(f"  LoRA r=64, Q + K + V + O:   {lora_high_rank/1e6:5.1f}M params ({100*lora_high_rank/full:.3f}% of full)")
print(f"\nLoRA reduces trainable params by 100-1000x.")
print(f"Combined with QLoRA's 4-bit base, fits 70B fine-tuning on a single 48GB GPU.")

LoRA is by far the dominant PEFT method in production today, but it is not the only one.

The PEFT family: beyond LoRA

Adapters

Adapters predate LoRA by two years. Houlsby et al. (2019) inserted a small bottleneck module, a down-projection $W_{\text{down}} \in \mathbb{R}^{d \times r}$, an activation function, an up-projection $W_{\text{up}} \in \mathbb{R}^{r \times d}$, and a residual connection, after each attention and FFN block, then trained only the adapters with the rest of the model frozen. The parameter savings are similar to LoRA: $2 d r$ per insertion point. The key difference is where the trainable parameters live. Adapters add new layers to the network; LoRA modifies existing weight matrices in parallel.

The architectural difference matters at inference. Adapters cannot be merged into the base weights, they are extra layers that must be evaluated separately on every forward pass. The inference overhead is small but non-zero. LoRA, by contrast, merges cleanly into the existing weight matrices and adds nothing at inference. That is the operational reason LoRA won the open-source default fight: same-quality adaptation, zero inference cost after merging. Adapters are still used, particularly in multi-task setups where their additive structure makes it easy to compose multiple adapters at once, but for single-task fine-tuning LoRA dominates.

Prefix tuning and prompt tuning

Two related methods sidestep weight updates entirely and instead inject trainable information through the model’s existing attention mechanism. Prefix tuning (Li & Liang 2021) prepends $L$ trainable continuous “soft prompt” vectors to the attention keys and values at every layer of the model. The model is otherwise frozen; only the prefixes train. Parameter count: roughly $2 L d \times \text{layers}$, typically about $1\%$ of base. Because prefix tuning never modifies model weights, it is the most “non-invasive” PEFT method, the base model is byte-for-byte identical before and after.

Prompt tuning (Lester et al. 2021) is even simpler. Train only the input embeddings of a short soft prompt prepended to the input sequence. The parameter count drops to just $L \cdot d$, tens to hundreds of thousands of parameters total, regardless of model depth. Prompt tuning is striking because of when it works: at very large scale ($10$B parameters and up), prompt tuning becomes competitive with full fine-tuning on many tasks; at smaller scale, it lags noticeably. The empirical claim from the paper is that the more capable the base model, the less prompt tuning has to do to specialize it.

(IA)³

(IA)³, Infused Adapter by Inhibiting and Amplifying Inner Activations, is the smallest PEFT method that still works well. Liu et al. (2022) showed that rather than adding a low-rank matrix to certain weights, you can multiply attention K, V and FFN intermediate activations by element-wise learned rescaling vectors. Three vectors of length $d$ per layer is enough. That is roughly $10\times$ fewer parameters than LoRA at the same rank, a few hundred kilobytes of trainable parameters for a $7$B model. The use case is few-shot fine-tuning, where you have very limited training data and want the smallest possible adapter that still adapts.

QLoRA: 4-bit base + LoRA

The memory problem persists

LoRA solves the gradient and optimizer-state problem. It does not solve the base model problem. The frozen weights still need to be in HBM during the forward pass, the model has to read them to compute anything. For a $70$B model in BF16, that is $140$ GB of base weights, sitting in memory unchanged across the entire training run. No current consumer or prosumer GPU has $140$ GB of HBM. The smallest configuration that fits is $2 \times$ A100 $80$GB. For a researcher with a single workstation GPU, even LoRA leaves you short.

QLoRA (Dettmers et al. 2023) targets the base-weight memory directly. The idea is to quantize the base model to $4$ bits while keeping the LoRA adapters in BF16. The trainable adapters stay in full precision so the gradient signal is clean; the frozen base shrinks $4\times$ because it does not need precision, it never updates. The result is that the dominant memory cost of fine-tuning collapses, and $70$B fine-tuning fits on a single $48$ GB GPU.

NF4 quantization

The specific $4$-bit format QLoRA uses is NF4 (NormalFloat-4), a non-uniform quantization scheme custom-designed for normally distributed weights. NF4 places its $16$ quantization levels at the quantiles of a normal distribution, closely spaced near zero, more widely spaced in the tails, rather than uniformly across the range like standard INT4. The motivation is empirical: pre-trained LLM weights are approximately normally distributed. A uniform $4$-bit quantizer wastes bits in the tails (where weights rarely land) and underrepresents the dense region near zero (where weights cluster). NF4 fits the distribution and loses less information per parameter than INT4 at the same bit width.

The memory impact is direct: a $70$B model in BF16 occupies $140$ GB; in NF4 it occupies roughly $35$ GB. That is a $4\times$ reduction from the dominant cost, and on its own it brings $70$B fine-tuning within reach of an $80$ GB A100.

This section treats NF4 as the format QLoRA needs; Chapter 18 opens it up as a quantization technique in its own right, deriving the equiprobable-normal-quantile construction and comparing it against INT4, GPTQ, and AWQ.

The full recipe

The QLoRA pipeline runs in four steps per training iteration. Step 1: load the base model and quantize all weights to NF4 at load time. The quantized representation is what lives in HBM for the entire run. Step 2: add LoRA adapters in BF16, a few hundred MB for a $70$B model, trivial relative to the base. Step 3: during the forward pass, dequantize each NF4 weight matrix to BF16 just before its matmul, run the matmul, then discard the dequantized version. The LoRA path runs entirely in BF16, no quantization involved. Step 4: during the backward pass, gradients flow only through the LoRA adapters, the base weights are frozen, so the quantization sits on the forward path only and never has to be made differentiable.

Two additional tricks make the headline numbers work. Double quantization quantizes the per-block quantization scalars themselves, saving an extra $\sim 0.4$ bits per parameter, small per-parameter but meaningful across $70$ billion of them. Paged optimizers use NVIDIA’s unified memory paging to spill optimizer state between GPU and CPU memory automatically, so optimizer state larger than HBM does not crash the training run. Together, these are not optional polish, they are the difference between QLoRA working on a single GPU and not.

The total memory budget for QLoRA fine-tuning a $70$B model lands at: NF4 base $\approx 35$ GB; LoRA adapters in BF16 $\approx 0.5$ GB; activations and (paged) optimizer state $\approx 5$–$10$ GB; total $\approx 40$–$45$ GB. A single $48$ GB A6000 fits the entire job. A single A100 or H100 $80$GB fits it with comfortable headroom. This is what unlocked open-source post-training in $2023$.

The runnable cell below works the QLoRA arithmetic explicitly for a $70$B model, comparing full FT, LoRA, and QLoRA memory budgets against A100 and A6000 GPU counts.

def memory_full_ft(n_params, precision_bytes=2):
    """Memory for full fine-tuning.
    Params + gradients (BF16) + optimizer (FP32 first + second moments)."""
    params = n_params * precision_bytes       # weights in BF16 = 2 bytes
    grads = n_params * precision_bytes        # gradients in BF16 = 2 bytes
    optim = n_params * 4 * 2                  # optimizer (FP32) = 4 bytes * 2 moments
    return params + grads + optim

def memory_lora(n_params_base, n_params_lora, precision_bytes=2):
    """Memory for LoRA fine-tuning.
    Base in BF16 (no gradient/optimizer) + LoRA in BF16 + LoRA optimizer in FP32."""
    base = n_params_base * precision_bytes
    lora_w = n_params_lora * precision_bytes
    lora_grad = n_params_lora * precision_bytes
    lora_optim = n_params_lora * 4 * 2
    return base + lora_w + lora_grad + lora_optim

def memory_qlora(n_params_base, n_params_lora):
    """Memory for QLoRA: base in NF4 (4 bits = 0.5 bytes/param) + LoRA in BF16."""
    base = n_params_base * 0.5                # 4-bit base
    lora_w = n_params_lora * 2
    lora_grad = n_params_lora * 2
    lora_optim = n_params_lora * 4 * 2
    return base + lora_w + lora_grad + lora_optim

# Compute for 70B model with LoRA on Q + K + V + O at r=16
n_base = 70e9
d_model, n_layers, rank, targets = 8192, 80, 16, 4   # 70B specs
n_lora = n_layers * targets * 2 * d_model * rank

mem_full = memory_full_ft(n_base) / 1e9
mem_lora = memory_lora(n_base, n_lora) / 1e9
mem_qlora = memory_qlora(n_base, n_lora) / 1e9

print(f"Fine-tuning a 70B model:")
print(f"  Full FT:  {mem_full:6.0f} GB  ({mem_full/80:.1f}x A100 80GB)")
print(f"  LoRA:     {mem_lora:6.0f} GB  ({mem_lora/80:.1f}x A100 80GB)")
print(f"  QLoRA:    {mem_qlora:6.0f} GB  ({mem_qlora/48:.1f}x A6000 48GB)")
print(f"\nQLoRA on a 70B model fits on a single 48GB GPU.")
print(f"This is what made open-source post-training accessible to small teams.")

LoRA and QLoRA cover the practical core, but the research frontier keeps moving.

Modern variants

DoRA

DoRA, Weight-Decomposed Low-Rank Adaptation (Liu et al. 2024), adds one piece of structure to LoRA. Decompose each base weight matrix $W$ into a magnitude vector $m$ and a direction matrix $V/\|V\|$ (with the norm taken per column). Apply LoRA only to the direction; learn the magnitude separately as a small additional trainable vector. The intuition is that the model can independently learn “how strongly does this column contribute” (magnitude) and “what direction does it point” (LoRA-updated direction), rather than collapsing both into a single low-rank update. Empirically DoRA matches or beats LoRA at the same trainable parameter count on a range of benchmarks. It is a refinement, not a paradigm shift, but it is gaining adoption in 2024 and 2025 PEFT recipes.

VeRA

VeRA, Vector-based Random Matrix Adaptation (Kopiczko et al. 2023), pushes the parameter count down further by asking a different question: does every layer need its own $A$ and $B$? VeRA’s answer is no. Freeze a single pair of random matrices $A_{\text{shared}}$, $B_{\text{shared}}$, sized to the largest layer and sliced as needed, and share them across every LoRA insertion point in the network. Neither matrix ever trains. What trains instead is a pair of small per-layer diagonal scaling matrices, $\Lambda_b$ and $\Lambda_d$, that rescale $B_{\text{shared}}$ and $A_{\text{shared}}$ row-by-row and column-by-column: $\Delta W = \Lambda_b B_{\text{shared}} \Lambda_d A_{\text{shared}}$.

Per layer, that is $r + d$ trainable scalars (the diagonal entries of $\Lambda_d \in \mathbb{R}^{r \times r}$ and $\Lambda_b \in \mathbb{R}^{d \times d}$) instead of LoRA’s $r(d+k)$, roughly $10\times$ fewer parameters than LoRA at the same rank. The random matrices carry no task-specific information at all; they are fixed, shared, frozen scaffolding, and the entire adaptation lives in the diagonal scaling. It is a more extreme version of the low-rank hypothesis: not just “the update is low rank” but “the update is a rescaling of a fixed random subspace.” VeRA trades away some expressivity for an order of magnitude fewer trainable parameters than even LoRA, which matters when you are storing thousands of small per-task adapters and every megabyte multiplies.

AdaLoRA

AdaLoRA (Zhang et al. 2023) attacks the rank-allocation question: not every weight matrix benefits equally from the same LoRA rank. Some layers, typically middle attention layers, gain more capacity from extra rank than others. AdaLoRA learns the rank allocation during training. Start with a uniform budget across modules, then prune the least-important rank components via an importance score derived from the singular values of the LoRA factors. At the end of training, capacity has been redistributed toward the modules that needed it. Useful when you have a fixed parameter budget and want to maximize per-parameter utility.

Mixture of LoRAs

A natural extension of LoRA is to maintain multiple adapters per weight matrix and route between them, essentially the MoE pattern from Chapter 11, applied at the adapter level rather than the FFN level. MoLoRA, X-LoRA, and related approaches train a small router that selects which LoRA expert to apply per token or per layer. The use case is multi-task fine-tuning: one base model serves many specializations, each carried by a small expert adapter, with the router learning when to invoke which. The serving story is favorable because the base model is shared and the experts are tiny; the training story is more involved because of the routing.

Exercises

Four exercises that build on the chapter’s machinery. Each is a self-contained problem with a starting template; hints are collapsed by default; try the problem first.

Exercise 1 (easy): LoRA forward pass + zero-init

Implement the LoRA forward pass and verify that with proper zero-initialization of $B$, the LoRA contribution is exactly zero at step 0, the model behaves identically to the base model.

import numpy as np

def lora_forward(x, W_base, A, B, alpha=16, r=8):
    return x @ W_base + (alpha / r) * (x @ A.T @ B.T)

np.random.seed(0)
d_in, d_out, r = 512, 768, 8
batch = 4

W_base = np.random.normal(0, 0.02, (d_in, d_out))
A = np.random.normal(0, 0.01, (r, d_in))
B = np.zeros((d_out, r))
x = np.random.normal(0, 1, (batch, d_in))

out_lora_step0 = lora_forward(x, W_base, A, B)
out_base = x @ W_base
print(f"At step 0: LoRA == base? {np.allclose(out_lora_step0, out_base)}")
# At step 0: LoRA == base? True

B_trained = np.random.normal(0, 0.01, (d_out, r))
out_lora_trained = lora_forward(x, W_base, A, B_trained)
print(f"After training: LoRA != base? {not np.allclose(out_lora_trained, out_base)}")
print(f"  Difference: max={np.abs(out_lora_trained - out_base).max():.4f}")
# After training: LoRA != base? True
#   Difference: max=0.0491

Exercise 2 (medium): Parameter count across configs

Compute trainable parameter counts for LoRA across multiple model sizes (7B, 13B, 70B), ranks (4, 8, 16, 32, 64), and target modules (Q+V / Q,K,V,O / all linear). Print a table and observe how the ratio (trainable / base) stays consistently small.

MODELS = {
    "7B":  {"d_model": 4096, "n_layers": 32, "total": 7e9},
    "13B": {"d_model": 5120, "n_layers": 40, "total": 13e9},
    "70B": {"d_model": 8192, "n_layers": 80, "total": 70e9},
}

TARGETS = {
    "Q+V":        2,
    "Q,K,V,O":    4,
    "all linear": 6,
}

def lora_params(d_model, n_layers, rank, targets_per_layer):
    return targets_per_layer * 2 * d_model * rank * n_layers

print(f"{'Model':<6} {'Target':<12} {'Rank':<6} {'Trainable':<12} {'% of base':<10}")
print("-" * 50)
for model_name, spec in MODELS.items():
    for target_name, target_count in TARGETS.items():
        for rank in [4, 8, 16, 32, 64]:
            count = lora_params(spec["d_model"], spec["n_layers"], rank, target_count)
            pct = 100 * count / spec["total"]
            print(f"{model_name:<6} {target_name:<12} r={rank:<4} {count/1e6:>6.1f}M {pct:>8.3f}%")
    print()

# Q,K,V,O, r=16 row across model sizes:
#   7B:  16.8M  0.240%
#   13B: 26.2M  0.202%
#   70B: 83.9M  0.120%
# -> ratio drops as model size grows, base scales faster than the adapter.

Exercise 3 (medium): LoRA merge equivalence

Verify that merging LoRA into the base weights produces exactly the same output as keeping them separate in adapter-mode. This is what enables zero-overhead inference.

import numpy as np

def lora_forward_adapter(x, W_base, A, B, alpha=16, r=8):
    return x @ W_base + x @ A.T @ B.T * (alpha / r)

def lora_merge(W_base, A, B, alpha=16, r=8):
    return W_base + (alpha / r) * (A.T @ B.T)

def lora_forward_merged(x, W_merged):
    return x @ W_merged

np.random.seed(2)
d_in, d_out, r = 256, 256, 16
W_base = np.random.normal(0, 0.02, (d_in, d_out))
A = np.random.normal(0, 0.01, (r, d_in))
B = np.random.normal(0, 0.01, (d_out, r))

x = np.random.normal(0, 1, (8, d_in))

out_adapter = lora_forward_adapter(x, W_base, A, B)
W_merged = lora_merge(W_base, A, B)
out_merged = lora_forward_merged(x, W_merged)

max_diff = np.abs(out_adapter - out_merged).max()
print(f"Max difference between adapter and merged: {max_diff:.2e}")
print(f"Within FP32 precision? {max_diff < 1e-5}")
# Max difference between adapter and merged: 1.55e-15
# Within FP32 precision? True

Exercise 4 (hard): QLoRA memory budget for 70B

Compute the full memory budget for QLoRA fine-tuning of a 70B-parameter model. Break down the components (NF4 base, BF16 LoRA, gradients, optimizer state, activations) and identify which GPUs the budget fits on.

def qlora_memory_70b(rank=16, target_count=4, d_model=8192, n_layers=80, activations_gb=8):
    total_params = 70e9

    base_nf4 = total_params * 0.5 / 1e9                      # 4 bits = 0.5 bytes/param
    lora_count = target_count * 2 * d_model * rank * n_layers  # trainable LoRA params
    lora_bf16 = lora_count * 2 / 1e9                          # 2 bytes/param
    gradients_bf16 = lora_bf16                                # same shape as adapters
    optimizer_fp32 = lora_count * 8 / 1e9                     # AdamW: 2 moments * 4 bytes

    total = base_nf4 + lora_bf16 + gradients_bf16 + optimizer_fp32 + activations_gb

    print(f"NF4 base:        {base_nf4:.2f} GB")
    print(f"LoRA count:      {lora_count/1e6:.1f}M")
    print(f"LoRA BF16:       {lora_bf16:.3f} GB")
    print(f"Gradients BF16:  {gradients_bf16:.3f} GB")
    print(f"Optimizer FP32:  {optimizer_fp32:.3f} GB")
    print(f"Activations:     {activations_gb} GB")
    print(f"Total:           {total:.2f} GB")
    return total

total = qlora_memory_70b()
# NF4 base:        35.00 GB
# LoRA count:      83.9M
# LoRA BF16:       0.168 GB
# Gradients BF16:  0.168 GB
# Optimizer FP32:  0.671 GB
# Activations:     8 GB
# Total:           44.01 GB

GPU_OPTIONS = [
    ("RTX 4090",     24),
    ("A6000",        48),
    ("A100 / H100",  80),
    ("2x A100 80GB", 160),
    ("4x A100 80GB", 320),
]
print(f"\nGPU fit check (total = {total:.1f} GB):")
for label, mem in GPU_OPTIONS:
    fits = mem >= total
    marker = "FITS" if fits else "does not fit"
    print(f"  {marker:<14} {label:<15} ({mem} GB)")
# does not fit   RTX 4090        (24 GB)
# FITS           A6000           (48 GB)
# FITS           A100 / H100     (80 GB)
# FITS           2x A100 80GB    (160 GB)
# FITS           4x A100 80GB    (320 GB)

When to use PEFT vs full fine-tuning

Use PEFT when

The default answer for most teams is use PEFT, and the conditions that push you toward it are the conditions most teams find themselves in. Compute budget is limited: full fine-tuning of a $70$B model needs at least $10$ A100s; LoRA needs one; QLoRA needs one prosumer GPU. The cost difference is often the entire decision. You need multiple fine-tuned versions of the same base model: LoRA adapters for a $70$B model are typically under $500$ MB, so storing dozens of them is operationally trivial and lets one base model serve many specializations. The fine-tuning task is a small departure from pre-training: instruction following, style adaptation, refusal-behavior tuning, and most preference optimization fall in this category, and these are the majority of production fine-tuning workloads. You want fast iteration: PEFT runs converge in hours where full FT takes days, which matters when you are sweeping hyperparameters or trying different data mixes.

Use full FT when

The cases where full fine-tuning earns its cost are narrower but real. The task requires deep capability shifts: adding a new language, a new modality, or substantial new knowledge that the base model genuinely does not have. Low-rank updates can fail here because the change being learned does not actually fit in a low-rank subspace. Or you are producing a foundation model that will itself be the starting point for many downstream tasks: frontier labs full-fine-tune their SFT base for this reason, since the SFT model becomes the reference for every later RLHF or DPO run and the small quality differences from full FT compound across the post-training stack. Or you have the compute and LoRA’s expressivity is genuinely limiting for your task, though this is the rarest case; most teams who think they need full FT actually have not exhausted LoRA’s capacity yet.

Hybrid recipes

In practice, production stacks mix the two. The most common pattern at well-resourced shops is full FT for SFT, run once at lab scale, expensive but tractable, and the resulting SFT model becomes the reference and the foundation for everything downstream. Then LoRA (or QLoRA) for downstream task adaptation, every product surface or vertical specialization gets its own small adapter on top of the shared SFT base. Storage costs scale with the number of adapters, not the number of models, and the base SFT investment is amortized across all of them.

For smaller teams, the analogous pattern is QLoRA for SFT and QLoRA for DPO, both runs sized to a single $48$ GB or $80$ GB GPU. The resulting adapters can be merged into the base at the end of training if a deployment target wants a single model file; otherwise they live as adapters on top of a shared base. This is how most of the high-quality open-source chat models from late $2023$ and through $2024$ were trained.

Parameter-efficient fine-tuning is what makes the methods of Chapter 13 and 14 practical at the scale of modern models. The recipe is short: freeze the base, train only $0.1\%$–$1\%$ of additional parameters as low-rank adapters, and combine with $4$-bit NF4 quantization (QLoRA) when memory is tight. That is what most production teams do, and it is the reason open-source post-training matured as fast as it did. LoRA is the dominant technique; the broader PEFT family (adapters, prefix tuning, (IA)³) is worth knowing but rarely necessary; modern variants (DoRA, AdaLoRA, MoLoRA) refine the recipe at the margin.

Chapter 16, the final chapter of Part V, covers distillation: how to compress a fully trained model into a smaller one for deployment. Where Chapters 13–15 covered how to train aligned models, Chapter 16 covers how to deploy them efficiently when even an inference-optimized $70$B is too large. After Chapter 16, Part V closes and Part VI (inference and serving, Ch 17–19) begins.