NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a heavily compressed variant of its hybrid MoE model, Nemotron-3-Super, that delivers up to a 2.14x boost in server throughput on an 8x B200 node while preserving the same block layout. By slashing total parameters from 120.7B to 75.3B and active parameters from 12.8B to 9.3B, the compressed model achieves a 2.03x throughput improvement at a matched user throughput of 100 tokens per second in the decode-heavy 8K/64K regime. For teams deploying large language models in production, this represents a meaningful step toward serving more concurrent users on the same hardware without sacrificing the architectural advantages of hybrid Mamba-Transformer MoE designs.
The Architecture of Compression: Preserving the Block Layout
Nemotron-3-Super is a hybrid model that combines Mamba state-space model layers, MoE layers, and attention layers. Its architecture consists of 88 blocks: 40 Mamba blocks, 40 MoE blocks, and 8 attention blocks. Puzzle-75B-A9B preserves this exact layout. What changes is the capacity inside those blocks. The Mamba SSM state size drops from 128 to 96 channels, the MoE routed expert intermediate size is pruned to a mean of 59.9% of the original, and the number of activated routed experts per token is reduced from 22 to a mean of 4-18. The active routed expert capacity, relative to the parent, drops to a mean of 30.9%. The number of routed experts, shared expert size, and MoE latent size remain unchanged. Attention layers are left untouched because the team states Nemotron-3-Super is already KV-cache efficient. Mamba layers are pruned uniformly, as inference frameworks do not support a different SSM state size per layer.
Targeting 2x Server Throughput at 100 Tokens Per Second
The deployment target was fixed before the architecture search began. The first goal was 2x server throughput at 100 tokens per second per user. The second was 8 concurrent 1M-token requests on a single H100. The research was carried out using the Puzzletron framework, a decomposed neural architecture search system. Three pruning techniques define the search space: intermediate channel pruning within each routed expert to a uniform size across an MoE layer, top-k reduction that varies the number of experts per token per layer, and Mamba SSM pruning from 128 to 96 channels. The SSM speedup from this pruning is 1.2x to 1.3x during decode at batch sizes between 8 and 512. Channels were ranked by estimated contribution to the Mamba layer output, averaged over 67M tokens of validation data.
Iterative Puzzle Outperforms Single-Step Compression
The standard single-step Puzzle approach assumes the quality impact of replacements is additive. That ignores higher-order interactions between replaced components. Iterative Puzzle alternates bounded compression with short knowledge distillation recovery in a sequence of stages, rather than jumping directly to the target. Scores are recomputed against the current compressed model, not the original parent. The three-stage process used here first compressed MoE weights to 75% of teacher capacity and Mamba SSM state to 75%, healed for 24B tokens. Stage two compressed MoE weights to 60%, healed for 43.2B tokens. Stage three reduced the activated routed-expert budget to 50%, allocated heterogeneously, and healed for 52.8B tokens. The three-step procedure averages 69.05 across ten benchmarks, against 68.48 for the single-step approach. Gains appear on MMLU-Pro, GPQA, HLE, AA-LCR, LiveCodeBench, SciCode, and RULER-256K. IFBench-Instruct fell 0.2 points and IFBench-Prompt fell 0.5.
Benchmark Results: Where Throughput Gains Are Largest
On a single 8xB200 node with NVFP4 weights, FP8 KV cache, and FP16 Mamba state, the boost ranges from 1.60x to 2.14x depending on the scenario and user throughput floor. In the 50K input / 2K output regime at a UT floor of 100, the boost is 1.60x. In the 8K / 64K regime at a UT floor of 100, the boost is 2.03x. At higher user throughput floors, the boost increases, reaching 2.14x for the 8K/64K regime at 125 tok/s. The prefill-heavy 50K/2K regime gains least, while the decode-heavy 8K/64K regime gains most. On a single 8xH100 node, using FP8 weights and FP8 KV cache, the gains are smaller: 1.91x on 50K/2K and 1.82x on 8K/64K.
Single-H100 1M-Token Concurrency: From 1 to 8
The memory constraint changes on a single H100 with 80 GB of HBM. Super’s NVFP4 weights occupy about 70 GB. Each 1M-token request adds about 4 GB of KV cache. Effective concurrency is therefore 1. Puzzle-75B-A9B’s NVFP4 weights occupy about 44.5 GB. The attention layout is unchanged, so per-request KV cost is the same. Concurrency at 1M rises to 8. Aggregate decode throughput at that concurrency is roughly 4x Super’s single-request throughput. Prefill of a 990K-token prompt is about 1.2x faster.
Accuracy Costs: Instruction-Following and Agentic Benchmarks
The real costs of compression show up on specific benchmarks. Arena-Hard-V2 drops by 4.2 points, SWE-Bench by 2.6 points. AIME25 (no tools) falls by 2.5 points. GPQA (no tools) drops by 1.9 points. RULER-256K and AA-LCR barely move. At NVFP4 quantization, HLE drops from 16.5 to 15.7. The team notes that the instruction-following and agentic evaluations lose most. Three BF16 results do not regress: AA-LCR gains 0.1, Scale AI Multi-Challenge ties at 56.6, and TauBench Telecom gains 0.4.
Recovery Pipeline: Distillation, RL, and Verbosity Control
Knowledge distillation ran on 30% pretraining data and 70% SFT data from Nemotron-3-Nano. During the Puzzle phase, KD used a 32K sequence length. Recovery then trained at 128K and scaled to 512K, with a budget of up to 100B tokens. RL post-training adopted Stage 2 of the Nemotron-3-Super RL pipeline, focused on software engineering, using a KL penalty of 0. The team swept learning rates and averaged the resulting weights. Short-context KD recovers most categories to over 97% of Nemotron-3-Super. Long-context KD lifts long-input and long-generation benchmarks specifically. Verbosity is a key detail: after the last Puzzle iteration, the model generated 132% of Super’s token count. That fell to 99% after the full recovery pipeline.
Deployment Quantization and Multi-Token Prediction
Two post-training quantization recipes are provided: FP8 W8A8 targeting Hopper, and NVFP4 W4A4 targeting Blackwell. The NVFP4 recipe uses max calibration, not the AutoQuantize sensitivity search used for Super. The resulting checkpoint is slightly more aggressively quantized and performed similarly. NVFP4 is not natively supported on Hopper but is used for the 1M-context H100 target because HBM capacity binds there. Puzzle-75B-A9B inherits a shared MTP head from Super. Transferring Super’s trained head directly gave similar acceptance lengths. Continued training on the transferred head addresses a training-inference mismatch. On SPEED-Bench at draft length 7, average acceptance length rose from 3.45 to 4.34, roughly 25% to 30%, concentrated at later draft positions.
Who Benefits From This Model
The strongest use case is ultra-long-context RAG on a single GPU, where a document analysis service goes from 1 concurrent request to 8, with aggregate decode throughput roughly 4x. Interactive coding assistants also benefit significantly. At 100 tok/s in the 8K/64K regime, one node serves 2.03x the tokens. Adjusted for verbosity, that is 2.16x the completed requests per minute. Prefill-heavy document pipelines gain only 1.60x, so the model is less suited for applications where prompt processing dominates compute. For agentic SWE loops, the 2.6-point SWE-Bench gap should be checked against the task mix, as the RL recovery targeted this capability and only partly restored it.
What This Means for Developers
If you are currently serving a large hybrid MoE model and hitting either a throughput ceiling or a memory bottleneck on long-context requests, Nemotron-Labs-3-Puzzle-75B-A9B offers a drop-in replacement with the same block architecture. The throughput gains are verified at specific user throughput floors and hardware configurations, and the accuracy regressions are concentrated in instruction-following and agentic tasks rather than long-context retrieval or coding. The model is available in three checkpoints on Hugging Face: BF16, FP8, and NVFP4. Teams should benchmark against their own task mix, particularly if instruction-following quality is critical, but the memory and throughput improvements on long-context workloads are substantial.
