Fine-tuning a 7B model at 128K context with standard attention requires more than 80GB VRAM — and that's before the optimizer allocates a single byte. Long-context fine-tuning VRAM pressure isn't a marginal scaling concern; it's a hard wall that stops most teams before the first training step completes. As Gemini 1.5 Pro pushes to 2M tokens, Qwen2.5-Turbo reaches 1M, and Llama 3.1 establishes 128K as the new baseline expectation, teams that can't fine-tune at these lengths face a compounding disadvantage: either ship with generic base models that don't fit your data distribution, or pay API costs indefinitely with no path to ownership. This guide explains precisely why long-context fine-tuning is a structurally different VRAM problem than standard fine-tuning, and maps the specific PEFT combinations that make it tractable on hardware available today.

Why Long Context Is Not Just "More of the Same"

Standard transformer attention scales as O(n²) with sequence length — not linearly. Doubling context length quadruples activation memory. At 128K tokens, activations during the backward pass routinely exceed model weights by 2–5×, which means the quantization and adapter tricks that work beautifully for short-context fine-tuning only solve half the problem.

This distinction is where most practitioners lose time. There are two fundamentally different sources of memory pressure, and conflating them leads to misdiagnosed OOM errors:

Parameter memory covers weights and optimizer states. A 7B model in bf16 occupies ~14GB; with AdamW, the optimizer states add 2× the weight size, bringing it to ~42GB. This is the domain where PEFT and quantization operate — and they work well.

Activation memory scales with sequence length and batch size. At 128K context, this component dwarfs parameter memory. No amount of 4-bit quantization fixes activation explosion, because activations aren't quantized during the forward/backward pass under standard implementations.

Both must be addressed simultaneously. A QLoRA run on a 7B model that handles 4K context effortlessly will OOM at 64K context on the same GPU, even though the model weights fit — because the activations don't.

The hardware baseline most teams are working against: the H100 at 80GB remains the practical workhorse. The B200 at 192GB and AMD MI300X at 192GB are arriving in clusters but aren't universally accessible. Practitioners on 40GB A100s or consumer-grade hardware face tighter constraints that make the techniques below non-optional.

The VRAM Math: Concrete Numbers

Model Size	32K Context	128K Context	1M Context
7B	~24GB	~48GB	~320GB
70B	~160GB	~640GB	~4.5TB

Baseline inference estimates with standard attention. Training adds roughly 3–4× over inference due to gradients and optimizer states.

A 70B model at 128K context requires approximately 640GB at inference — roughly eight H100s — before any training overhead. At 1M tokens with a 7B model, you're looking at 320GB just to hold activations, which means a single H100 is irrelevant without aggressive optimization.

Gradient checkpointing partially addresses activation memory by discarding forward activations and recomputing them during the backward pass, recovering 30–60% of activation memory. The tradeoff is compute: at 1M tokens, the recompute cost becomes prohibitive without sequence chunking. Gradient checkpointing is necessary but insufficient on its own beyond ~64K context.

PEFT Techniques That Actually Move the Needle

QLoRA is the correct baseline, not the complete solution. 4-bit NF4 quantization of the frozen base model cuts weight memory approximately 75%, bringing a 70B model's parameters to around 35GB. Pair this with LoRA adapters at rank r=16–64. The layer selection matters more than most recipes acknowledge: for long-context tasks, Q, K, V, and output projection layers are all necessary targets. The common shortcut of adapting only Q and V leaves performance on the table for tasks requiring long-range retrieval, because key projections shape which tokens attend to what, and output projections determine how retrieved context is mixed into the residual stream.

QLoRA alone doesn't fix the activation problem. It buys you memory on the parameter side; you still need an attention-side solution.

LongLoRA is purpose-built for exactly this gap. The core mechanism is shifted sparse attention (S²-Attn): during fine-tuning, sequences are split into local groups, with half the attention heads shifted by half a group size to enable cross-group information flow. This approximates full attention behavior at a fraction of the activation cost. The critical insight from the LongLoRA paper is that full attention is only required during inference — the model can learn long-context representations through the shifted sparse pattern during training and generalize to full attention at test time. The concrete result: Llama 2 7B fine-tuned to 100K context on a single 80GB A100. That's the benchmark to calibrate against.

YaRN and RoPE scaling deserve precise positioning. These positional encoding extensions allow a model to extrapolate to context lengths beyond its training distribution by rescaling the rotary position embeddings. DeepSeek's approach to reaching 128K context used RoPE scaling to achieve roughly 40% VRAM savings versus retraining from scratch at full length. However, YaRN alone — without any fine-tuning on long-context data — degrades retrieval accuracy at the far end of extended ranges. Use it as a complement to reduce the volume of long-context training examples needed, not as a substitute for fine-tuning.

Chunked cross-entropy is systematically overlooked. At long sequence lengths, computing cross-entropy loss requires materializing the full logit matrix: [sequence_length × vocabulary_size]. For a 128K sequence with a 128K vocabulary, this single tensor can consume 4–8GB independently of everything else. Chunked cross-entropy computes the loss in blocks, avoiding materialization of the full matrix. Combined with gradient checkpointing, this pairing regularly saves 6–10GB on long-context runs — a meaningful margin when you're working at the edge of available VRAM.

Attention Architecture Choices

FlashAttention-2/3 is a prerequisite, not an optimization. By rewriting attention computation to use tiled SRAM operations, FlashAttention reduces attention memory from O(n²) to O(n) and delivers 1.5–2× throughput improvements on H100s (FlashAttention-3, 2024). Any long-context fine-tuning stack that omits it is leaving the single largest optimization on the table. Integration is straightforward: set attn_implementation="flash_attention_2" in HuggingFace Transformers. FlashAttention-3 with H100-specific optimizations is available via the flash-attn package. If your stack doesn't use this, fix that before evaluating anything else.

Ring Attention and sequence parallelism address the multi-GPU scaling case. Ring Attention distributes the sequence across GPUs arranged in a ring topology — each GPU computes attention over its local chunk while passing KV blocks around the ring in a pipelined fashion. This distributes activation memory proportionally to the number of GPUs, making sequence length a solvable parallelism problem rather than a hard wall. Meta's chunked attention implementation on 8×H100 handled 1M+ token training sequences as of July 2024. The contrast with tensor parallelism is important: tensor parallelism shreds model parameters across GPUs and addresses weight memory, but does nothing for sequence-length scaling. For long-context work, sequence parallelism is the relevant axis. DeepSpeed Ulysses and the ring-flash-attn library are the current standard implementations.

Hierarchical global attention — the pattern pioneered by Longformer and BigBird — is finding new relevance in fine-tuning contexts. The mechanism designates specific tokens as global attention nodes that attend to and are attended by all other tokens, while the bulk of the sequence uses local windowed attention. In a fine-tuning workflow, this translates to a practical optimization: freeze the local attention layers entirely and apply LoRA adapters only to the global attention projection layers. If your task has a structured instruction or summary token that should aggregate long-range context — a query token in a retrieval task, a system prompt token in a RAG application — this pattern reduces trainable parameters substantially while concentrating adaptation where it matters. Moonshot's Kimi Delta Attention in Kimi K3 applies a conceptually similar hybrid: maintaining global attention heads for cross-sequence synthesis while localizing the bulk of attention computation, extended across 93 transformer layers at the 2.8T parameter scale.

Practical Stacks for Real Hardware

Single 80GB H100, up to 100K context: QLoRA (4-bit NF4) + LongLoRA S²-Attn + FlashAttention-2 + gradient checkpointing. Trainable on a 7B model at batch size 1–2 with gradient accumulation. This is LongLoRA's demonstrated configuration and the correct ceiling to expect. Library stack: bitsandbytes for quantization, peft for LoRA adapters, transformers with flash attention enabled, and trl for training loop management. Don't attempt 128K on this configuration without LongLoRA — the activation memory will OOM regardless of quantization.

4×H100 (320GB aggregate), up to 256K context: Full bf16 LoRA fine-tuning without quantization, FlashAttention-3, and sequence parallelism via DeepSpeed Ulysses. Suitable for 7B–13B models. DeepSpeed ZeRO-3 handles optimizer state sharding across the four GPUs, so each holds only a quarter of the optimizer states. This configuration is well within reach for teams with cloud access and provides a clean debugging surface since you're not stacking quantization atop parallelism.

8×H100 (640GB aggregate), up to 1M context: Chunked attention + Ring Attention + LoRA on attention projection layers only + gradient checkpointing. Viable for 70B models at 128K–256K, or 7B models approaching 1M tokens. The ring-flash-attn package integrates with flash-attn and handles the KV communication pattern. B200 and MI300X clusters at 192GB per device will substantially ease these constraints through 2025–2026 — teams planning infrastructure investments should weight this.

For teams without multi-GPU cluster access, Lambda Labs, Vast.ai, and RunPod all offer on-demand 8×H100 instances. The economics favor renting for fine-tuning runs of days to weeks rather than owning, given how rapidly the hardware landscape is shifting toward B200 configurations. A single fine-tuning job at 256K context that takes 48 hours on 4×H100 at ~$12/hour per GPU totals under $2,400 — comparable to a few weeks of heavy API spend on a frontier model.

Decision Framework

The two variables that determine your toolchain are context length target and hardware envelope. Under 100K context on a single 80GB GPU: QLoRA + LongLoRA + FlashAttention-2 is the correct default. Between 100K and 256K on a 4-GPU cluster: bf16 LoRA + FlashAttention-3 + DeepSpeed Ulysses, no quantization required. Above 256K or targeting 1M: Ring Attention becomes necessary, and the architecture decisions around global versus local attention windows directly affect whether your adapted model generalizes well or overfits to the local context structure.

One underappreciated risk: recent research on fine-tuning-induced behavioral shifts suggests that adapting attention layers specifically — rather than MLP layers — can produce more stable long-context behavior, because MLP layers carry more of the factual and safety-relevant representations. When selecting LoRA target layers for long-context tasks, concentrating adapters on attention projections is both the memory-efficient choice and the behaviorally safer one.

The teams that will own long-context capabilities in 2025–2026 are not the ones waiting for B200 clusters to become commodity hardware. They're the ones who have already built the tooling to run 128K fine-tuning jobs on current 80GB hardware and are iterating on data quality and task formulation — the variables that matter once the VRAM problem is solved.