ByteDance is pre-training a model with up to 10 trillion parameters. GPT-4 — estimated at roughly 1.8 trillion parameters across a mixture-of-experts architecture — serves millions of requests daily. Moonshot AI's Kimi K3 has crossed into the 3-trillion-parameter class as the largest open-weight model available. Nvidia is reportedly building a 1-trillion-parameter Nemotron model targeting a late-2026 release. Serving trillion parameter models is no longer a research curiosity; it is an operational challenge that engineering teams are encountering without a clear playbook.

The core tension is this: MoE architectures make trillion-parameter models computationally tractable during inference, because only a fraction of parameters activate per token. DeepSeek-V3, at 671 billion total parameters, activates roughly 37 billion per token. That ratio is what makes large-scale serving economically conceivable. But the memory requirement does not shrink with sparsity — you still need to store all 671 billion weights on live hardware. At 1 trillion parameters in FP16, that is approximately 2 terabytes of GPU memory before you account for KV cache, activation buffers, or framework overhead. No single GPU holds that. No single node holds that cleanly. Everything else in this article is about what you do with that foundational constraint.

Why MoE Doesn't Solve the Memory Problem

The confusion worth clearing up immediately: MoE reduces compute per token, not memory per replica. When a token passes through DeepSeek-V3, only a subset of expert subnetworks execute. But all experts must reside in memory waiting to be dispatched to. This is the difference between training infrastructure and serving infrastructure, and it matters enormously for cost modeling.

Training is batch-oriented and can tolerate high latency. A trillion-parameter training run costs over $100 million per run by Sam Altman's 2023 estimate, runs for weeks across thousands of GPUs, and the economics are amortized across the model's lifetime. Serving is latency-sensitive, load-variable, and billed by the token. OpenAI's GPT-4-class API pricing has sat in the $0.06–0.12 per 1,000 output tokens range — a figure that represents the compressed output of extraordinary infrastructure investment. The infrastructure decisions you make at serving time directly determine whether that price point is a margin or a loss.

Hardware Tier Requirements

The minimum viable hardware floor for serving a 1-trillion-parameter model in FP16 is roughly 25 high-memory GPUs with high-bandwidth interconnect — and that figure assumes no KV cache headroom, which is not a realistic production assumption. In practice, you are looking at 32–48 GPUs as a functional floor for a single serving replica.

Three hardware options are realistic today:

NVIDIA GB200 NVL72 is the purpose-built solution. The rack-scale system pairs 72 Blackwell GPUs with approximately 13.5TB of shared HBM3e memory and GPU-to-GPU bandwidth around 900 GB/s via NVLink/NVSwitch. NVIDIA positions this configuration as supporting models up to 27 trillion parameters. A single NVL72 rack can hold a 1-trillion-parameter FP16 model with substantial headroom for KV cache. This is the current best-in-class option — and it is scarce. Allocation queues are measured in quarters, not weeks, and the capital cost per rack runs into the millions of dollars.

NVIDIA H100/H200 (8× node clusters) represents the realistic near-term option for most teams. Each H100 carries 80GB HBM3; an 8-GPU node gives you 640GB. Fitting 2TB of FP16 weights requires three to four nodes minimum, connected via InfiniBand at 400–800 Gb/s. That inter-node bandwidth is the critical problem: NVLink intra-node bandwidth runs at roughly 900 GB/s, while InfiniBand at 800 Gb/s converts to approximately 100 GB/s effective throughput — an order of magnitude slower. Every cross-node communication call is a latency event.

AMD MI300X offers the highest per-chip memory available at 192GB HBM3, which changes the calculus for medium-scale trillion-parameter configurations. A single 8-GPU node gives you 1.5TB — enough for a 671B FP16 model with some headroom. The MI300X's memory advantage is real and measurable. The deployment friction is also real: the CUDA ecosystem, TensorRT-LLM optimizations, and most production inference tooling is built around NVIDIA hardware. Teams choosing MI300X for trillion-parameter serving should budget for engineering time that NVIDIA deployments would not require.

The interconnect bottleneck deserves emphasis because it drives architectural decisions more than raw FLOP counts. When your model must span multiple nodes, every all-reduce operation during tensor parallelism crosses InfiniBand. This is why fitting within a single NVL72 rack — when FP8 or INT4 quantization makes it feasible — is strongly preferable to a multi-node H100 cluster from a latency standpoint.

Parallelism Strategies

Three parallelism axes apply to trillion-parameter MoE serving, and the right combination depends on your hardware topology.

Tensor parallelism splits individual weight matrices across GPUs. It is low-latency when confined to NVLink-connected GPUs within a node, making it well-suited for attention layers and dense feed-forward blocks. The practical ceiling is 8 GPUs — beyond that, the all-reduce communication overhead begins consuming a larger fraction of the compute budget than the parallelism saves.

Pipeline parallelism splits the model layer-by-layer across nodes. It enables multi-node deployment but introduces pipeline bubbles — idle GPU time between microbatches — that reduce effective utilization by 30–50% without careful microbatch scheduling. For H100 multi-node deployments where you have no alternative, pipeline parallelism is unavoidable; budget for the efficiency loss explicitly in your cost model rather than assuming theoretical peak throughput.

Expert parallelism is the MoE-specific strategy, and it is where the architecture's inference advantages actually materialize. Different expert subnetworks reside on different GPUs, and the router dispatches tokens to the appropriate device at inference time. DeepSeek's inference engine uses expert parallelism to achieve roughly 2× throughput improvement for MoE models compared to naive tensor parallelism. The mechanism is straightforward: because only a subset of experts activate per token, expert parallelism avoids the all-reduce overhead of tensor parallelism for the expert layers.

For a 1-trillion-parameter MoE model on GB200 NVL72 hardware, the current best-practice combination is expert parallelism across expert groups with tensor parallelism within each expert group. For H100 multi-node clusters, pipeline parallelism becomes a third required dimension — layer groups on different nodes, with expert parallelism within each pipeline stage.

One underappreciated problem that parallelism strategy does not solve: MoE routing creates hot experts. Some expert subnetworks receive disproportionate token traffic based on input distribution — a pattern that causes GPU underutilization on cold experts and latency spikes on overloaded ones. Training-time auxiliary load-balancing losses, used in DeepSeek-V3's training procedure, reduce routing imbalance but do not eliminate serving-time variance. Production deployments should implement expert utilization monitoring and have a strategy for rebalancing expert placement when traffic patterns shift.

Software Stack and Optimization

Framework	MoE Support	Quantization	Best For	Production Maturity
vLLM	Mixtral-class; 1T+ maturing	FP8, INT4 via bitsandbytes	Flexible OSS deployments	High for <671B
TensorRT-LLM	Strong, Blackwell-optimized	FP8 native, INT4	NVIDIA hardware, throughput-critical	High, NVIDIA-only
SGLang	Growing	FP8	Structured generation workloads	Medium
NVIDIA NIM	Via TensorRT-LLM backend	FP8	Enterprise ops, reduced overhead	Medium-high

TensorRT-LLM's tight integration with Blackwell hardware produces measurably better throughput than framework-agnostic options — 2–4× over naive HuggingFace inference on equivalent hardware. The tradeoff is higher setup complexity and lock-in to NVIDIA's toolchain. For teams running NVL72 hardware at scale, that tradeoff is straightforward. For teams evaluating AMD MI300X or building hardware-portable systems, vLLM's broader hardware support and PagedAttention-based KV cache management may justify the throughput gap.

Quantization is the single highest-ROI optimization available. FP16 to FP8 halves memory footprint with minimal quality degradation on most generation tasks — this is the first optimization to deploy, and it is largely free on Blackwell hardware where FP8 is a native compute type. FP16 to INT4 reduces memory by 4×, bringing a 1-trillion-parameter model from ~2TB to approximately 500GB — within reach of a single NVL72 rack without spanning multiple nodes. INT4 requires careful calibration datasets to maintain output quality; the quality-memory tradeoff is task-dependent and should be validated against your specific workload before production deployment.

KV cache becomes the dominant memory consumer at high concurrency. A 128K-context request for a large MoE model can accumulate KV cache that rivals or exceeds the weight memory footprint. DeepSeek's optimized attention implementation reports approximately 60% KV cache reduction versus standard implementations. PagedAttention (vLLM) and continuous batching are non-negotiable for any production deployment handling more than a handful of concurrent long-context requests — without them, memory fragmentation causes effective throughput to collapse under load.

Speculative decoding — using a small draft model to propose tokens that the large model verifies in parallel — delivers 2–3× latency improvements for certain workloads. The technique is less straightforward for MoE architectures because the draft model and verification model have different routing behaviors, complicating the verification step. It remains viable for MoE serving but requires architecture-specific implementation work; treat it as a second-phase optimization after the parallelism and quantization stack is stable.

Cost Levers and Decision Framework

The infrastructure cost for serving trillion-parameter models breaks into three components: hardware acquisition or cloud rental, power and cooling, and engineering overhead.

A single GB200 NVL72 rack running at capacity represents millions of dollars in hardware and requires data center power density that most colocation facilities cannot support. Cloud alternatives — renting H100 nodes from major providers — run roughly $2–4 per GPU-hour for H100 80GB nodes. A 32-GPU H100 deployment for a single 1T FP16 model replica costs approximately $64–128 per hour before software and networking. At current GPT-4 API price points of $0.06–0.12 per 1,000 output tokens, sustaining a profitable service requires throughput rates that only high-batch, well-optimized deployments achieve.

The levers that actually move the cost-per-token figure: quantization (INT4 cuts hardware requirements by 4×), continuous batching (amortizes fixed memory cost across more concurrent requests), and routing efficiency (hot expert mitigation reduces tail latency, which caps achievable batch sizes). Teams that treat these as optional optimizations rather than table stakes will find trillion-parameter serving economically unviable at any meaningful traffic level.

The decision framework for whether to serve at this scale: if your use case requires frontier-quality outputs and can sustain the throughput volume needed to amortize hardware costs, the path is NVL72 hardware with TensorRT-LLM, FP8 quantization from day one, and expert parallelism as the primary distribution strategy. If your volume is insufficient to fill a dedicated deployment, the economics favor API access to hosted frontier models until traffic justifies owned infrastructure. The crossover point depends on your output token volume, but most teams reach it somewhere between 50 million and 200 million output tokens per month — below that, hosted APIs are cheaper including all overhead; above it, owned infrastructure begins to win.

The capital markets have registered the infrastructure opportunity clearly — NVIDIA has secured preliminary agreements with Apollo, Blackstone, BlackRock, and others to mobilize over $500 billion in AI infrastructure financing. That capital is flowing toward exactly the hardware described here. The teams that will extract value from it are those who understand that serving trillion parameter models is a systems engineering problem first, and a model selection problem second.