Local LLM Inference Hardware: Selecting GPUs for Open-Weight Model Deployment
Local LLM Inference Hardware: Selecting GPUs for Open-Weight Model Deployment — MasterNodeAI evergreen analysis covering local LLM inference GPU selection.
Local LLM inference GPU selection has moved from hobbyist experiment to serious infrastructure decision. The open-weight model landscape now spans 8B to 671B parameters — Llama 3.3, Qwen 2.5, DeepSeek R1, Mistral Small 3 — and the hardware choices available in early 2025 are meaningfully different from what existed 18 months ago. This brief covers the decision for founders, technical leads, and teams choosing a primary hardware path for on-premise inference: not data center gear, but consumer and prosumer hardware that can be ordered today.
The Decision
Three hardware paths dominate serious local deployment:
Single NVIDIA GPU (RTX 4090 or RTX 5090) — the mainstream CUDA-native path, covering 7B through 70B-class models depending on which card you choose.
Dual GPU configuration (2× RTX 4090) — the bridge option for teams that need 70B quantized inference without paying RTX 5090 prices.
Apple Silicon (Mac Studio M4 Max or M4 Ultra) — the unified memory alternative that removes the VRAM ceiling entirely.
AMD's RX 9070 XT (RDNA 4, launched early 2025) is a fourth option with improved ROCm support, but ROCm ecosystem maturity still lags meaningfully behind CUDA for production inference workloads. It belongs in a future brief, not a current purchase decision.
Context
Two things have changed to make this decision consequential now rather than six months from now.
First, quantization has become standard practice. A 32B model requires approximately 20 GB VRAM at 4-bit quantization via GGUF or ExLlamaV2, compared to roughly 64 GB in FP16. This compression has pulled serious model sizes down into consumer GPU range: Qwen 2.5 32B, Mistral Small 3 (24B), and Llama 3.3 70B (at Q4, approximately 40 GB) are all practically deployable on consumer hardware. A hardware choice from 2023 — optimized for 13B models — is now mismatched to the workload distribution most teams actually run.
Second, the RTX 50-series Blackwell launch in January 2025 and the Apple M4 Ultra Mac Studio reset the price-to-capability map. The RTX 5090 at $1,999 MSRP carries 32 GB VRAM — enough to run Llama 3.3 70B at Q3/Q4 quantization on a single card, which previously required two RTX 4090s. Apple's M4 Ultra supports up to 192 GB unified memory, enabling unquantized 70B inference and quantized models exceeding 100B parameters. These are not incremental updates; they change the single-card ceiling.
Privacy, latency, and API cost pressure are pushing more teams toward on-premise deployment. For teams running sustained inference workloads — internal tools, RAG pipelines, code assistants — the monthly API cost comparison against a one-time hardware purchase closes within three to six months at moderate usage.
Option Analysis
| Option | VRAM | 70B Q4 Capable? | Approx. Cost | Best For |
|---|---|---|---|---|
| RTX 4090 (single) | 24 GB | No (40 GB needed) | $700–900 used / $1,600–1,800 new | 7B–32B inference |
| RTX 5090 (single) | 32 GB | Yes (Q3/Q4 tight fit) | ~$1,999 new | 7B–70B single-card |
| 2× RTX 4090 | 48 GB combined | Yes (comfortably) | $1,400–1,800 used pair | 70B on a budget |
| Mac Studio M4 Max | 128 GB unified | Yes (unquantized) | $2,999–3,199 | High-memory, low-complexity |
| Mac Studio M4 Ultra | 192 GB unified | Yes (easily) | $4,999+ | 100B+ models, macOS teams |
RTX 4090 (24 GB VRAM). The most benchmarked consumer GPU for LLM inference. Runs 7B through 32B models at 4-bit quantization without issue — Qwen 2.5 32B at Q4 occupies roughly 20 GB, leaving headroom for KV cache. Used units in the $700–900 range make this the highest value-per-VRAM-GB option currently available. The ceiling is real: 70B quantized models at ~40 GB exceed single-card capacity, full stop. Blackwell kernel optimizations in llama.cpp and ExLlamaV2 are written for the new architecture; the 4090 will not benefit from them. For teams whose target models fit within 32B, this limitation is irrelevant.
RTX 5090 (32 GB VRAM). The 8 GB VRAM increase over the 4090 sounds incremental but is structurally significant: it crosses the threshold for single-card 70B quantized inference. Llama 3.3 70B at Q3 quantization fits within 32 GB; Q4 is tight and context-window dependent. Blackwell tensor core improvements deliver real throughput gains for batch inference workloads. At $1,999 new versus $700–900 for a used 4090, buyers targeting sub-32B models are paying a significant premium for capability they will not use.
Dual RTX 4090 (48 GB combined via PCIe tensor parallelism). NVLink is not required for single-model inference across two consumer GPUs — llama.cpp handles PCIe tensor parallelism adequately. The 48 GB combined capacity comfortably covers 70B at Q4. Total cost using used cards ($1,400–1,800 for the pair) can undercut a new RTX 5090. The tradeoffs are real: throughput efficiency across two PCIe-connected cards runs at roughly 60–70% of theoretical linear scaling, not 2×. The setup requires a board with dual PCIe x16 slots, a PSU rated for 600W or more under GPU load, and more involved thermal management. This is the right path for budget-constrained teams with 70B requirements who have the systems knowledge to manage it.
Mac Studio M4 Max / M4 Ultra. Unified memory architecture eliminates the VRAM ceiling as a constraint entirely. The M4 Ultra's 192 GB configuration enables unquantized 70B inference and quantized models well above 100B. Apple's MLX framework has demonstrated up to 1.5× throughput versus llama.cpp on M-series chips for certain model families. The practical constraints are ecosystem lock-in (no CUDA, no vLLM without significant workarounds), non-upgradeability, and the entry price — $2,999 minimum for the M4 Max configuration. For macOS-native teams or organizations running mixed document-processing and inference pipelines on Apple hardware, the simplicity argument is genuine. For teams already running Linux inference infrastructure, it introduces a divergent toolchain.
Decision Framework
Step 1 — Identify your target model size. Be specific, not aspirational. VRAM requirements at 4-bit quantization: 7B–14B models require 6–10 GB; 32B models require approximately 20 GB; 70B models require approximately 40 GB. FP16 inference multiplies those figures by roughly 3–4×. Add 20% headroom for context window and KV cache growth — a 70B Q4 model at 8K context will use more VRAM than the same model at 2K context.
Step 2 — Identify your binding constraint. Budget under $1,000: used RTX 4090, full stop. Need 70B on a single card: RTX 5090 or Mac Studio M4 Max. Running macOS-first infrastructure with no Linux ops capacity: Apple Silicon. Primary constraint is throughput for multi-user serving: NVIDIA CUDA with vLLM — vLLM's PagedAttention mechanism substantially improves memory utilization on 24 GB cards by eliminating fragmented KV cache allocation (see the original vLLM paper, Kwon et al., 2023, for the underlying implementation and efficiency analysis).
Step 3 — Match throughput to use case. Single-user interactive inference at 1–5 tokens per second is tolerable on Apple Silicon or CPU offloading for smaller models. A team of five to ten running batched requests against an internal inference endpoint needs NVIDIA CUDA with ExLlamaV2 or vLLM — Apple Silicon's memory bandwidth, while impressive for its thermal envelope, does not compete with discrete GPU bandwidth for sustained throughput under concurrent load.
NVIDIA held approximately 88% discrete GPU market share in Q4 2024 (Jon Peddie Research — verify against published JPR report before citing externally). The CUDA software ecosystem reflects that dominance: llama.cpp, ExLlamaV2, vLLM, and Ollama all have deeper CUDA optimization paths than any alternative.
Recommendation
For most teams: buy a used RTX 4090 at $700–900. It handles the full range of practical open-weight workloads through 32B parameters — Mistral Small 3, Qwen 2.5 32B, Llama 3.1 8B at any quantization level — at the lowest cost-per-capability point in the current market. The model families most teams actually deploy in 2025 fit comfortably within 24 GB at Q4 quantization.
If your workload specifically requires 70B-class models and budget is primary: two used RTX 4090s at approximately $1,500 total beats a new RTX 5090 on cost while providing more headroom. Accept the multi-GPU complexity as the cost of that savings.
If your workload requires 70B inference on a single card with minimal setup overhead and you can absorb the price: RTX 5090 at $1,999 is the cleaner engineering choice — one card, one driver, Blackwell kernels, and a VRAM ceiling that will cover the next two to three years of open-weight model releases.
If your team operates entirely within the Apple ecosystem and memory capacity matters more than raw token throughput: Mac Studio M4 Max (128 GB) is the right call. The MLX framework is maturing quickly, and the operational simplicity of a single silent device handling 100B+ models is a genuine advantage for certain deployment contexts.
The worst outcome is buying for the model size you think you might want in 18 months rather than the one you're deploying today. Buy for your current workload. The used GPU market for the 4090 is liquid; the upgrade path is clear.