Tencent has released AngelSpec, an open-source, torch-native training framework for speculative-decoding draft models, covering both autoregressive multi-token prediction (MTP) and the block-parallel DFlash family. The release addresses a fundamental limitation in existing speculative decoding systems: the assumption that a single universal drafter can perform well across all real-world workloads. AngelSpec treats workload heterogeneity as a first-class design constraint, specializing architecture, training data, and verification depth around the specific characteristics of different traffic types.
Why speculative decoding needs more than one drafter
Speculative decoding is a lossless inference acceleration technique. A lightweight draft model proposes several future tokens, and the target model verifies them together in a single forward pass using rejection sampling. The acceleration achieved depends on two factors: how many draft tokens are accepted, and the total time per draft-verify cycle. These two quantities move in opposite directions across different domains. In high-entropy open-ended conversation, many continuations are semantically valid, so acceptance decays quickly with proposal depth. Generating and verifying a long block in such cases wastes compute. Autoregressive MTP drafting fits this regime because it proposes a shorter candidate sequence. Code and mathematical reasoning behave differently—programming syntax, repeated identifiers, formal expressions, and step-by-step derivations constrain future tokens more strongly, creating longer predictable spans that block-parallel drafting amortizes effectively.
AngelSpec’s two complementary drafters
AngelSpec provides two distinct drafters rather than a single compromise. The MTP model is trained on rich, diverse conversation-oriented data, while the block-diffusion model (DFlash family) is strengthened with code- and mathematics-focused samples. This specialization allows each drafter to excel in its intended domain without sacrificing performance elsewhere.
The MTP path: Training-Time Test and target-model rollout
The original Hy3 model used a single MTP layer with no recurrent self-conditioned unrolling. At inference, the block could be reused recurrently, but the training objective never prepared it for a long self-generated chain. Errors accumulated with depth, causing the second and third draft positions to accept at substantially lower rates than the first. AngelSpec addresses this train-inference mismatch with a shared-parameter, multi-depth scheme. It retains D logical prediction depths but reuses one physical MTP block. During training, that block is autoregressively unrolled for D steps, with each prediction fed into the next invocation. Following the Training-Time Test principle of EAGLE-3, depth k+1 receives the arg max prediction from depth k instead of the ground-truth token. Parameters are shared, but supervision stays depth-specific: the teacher target advances one future position at every depth.
Two further choices carry most of the gain. First, the target backbone and target language-model head are frozen, and MTP inputs from the backbone are detached. The drafter improves its proposal distribution without touching the distribution that verification must preserve. Second, training uses target-model rollout: responses are generated by the frozen target rather than taken from the original reference corpus. That produces the exact token choices, hidden-state trajectories, and local uncertainty patterns the MTP must approximate at serving time.
The measured effect is concentrated where it should be. At T = 0, mean acceptance moves from 52.8% to 66.4%, and mean accepted length from 2.58 to 2.99. The first-position rate is nearly preserved at 0.799 to 0.814. The deeper positions carry the delta: p3 climbs from 0.290 to 0.706 on GSM8K, and from 0.387 to 0.757 on HumanEval.
Block-parallel DFlash family: DFlash, DSpark, and DFly
For code and math workloads, AngelSpec provides block-parallel drafters in the DFlash family. DFlash is the baseline block-parallel drafter, while DSpark and DFly introduce successive improvements. DFly, the most advanced variant, incorporates a Markov head, hidden state correction, and a code/math enriched training dataset. The cumulative ablation on Hy3-A21B shows the progression: DFlash backbone achieves a mean accepted length of 3.77, adding the DFly backbone raises it to 4.40, the Markov head pushes it to 4.56, hidden correction to 4.60, and the code/math data expansion to 4.75. The domain data expansion adds 700K prompts: 500K code from OpenCodeInstruct and OpenCodeReasoning, 200K math from Big-Math. MT-Bench slips from 3.06 to 3.02, which the report calls the intended specialization trade.
Benchmark results across Qwen3-8B and Hy3-A21B
AngelSpec’s benchmarks cover six datasets: Math500, GSM8K, HumanEval, MBPP, LiveCodeBench, and MT-Bench. On Qwen3-8B, DFly achieves the highest average mean accepted length (MAL) at 5.41, compared to 5.32 for DSpark, 4.57 for DFlash, and 3.24 for MTP. On Hy3-A21B, DFly’s average MAL is 4.79, versus 3.69 for DFlash and 3.00 for MTP. DSpark keeps the lead on MT-Bench for Qwen3-8B (3.77 vs 3.67), consistent with the report positioning DFly for code and math workloads.
Throughput performance on Hy3-295B-A21B with TP=8
Output-token throughput tests on HY3-295B-A21B with tensor parallelism 8 show DFly-8 posting the highest average speedup at every tested concurrency from 4 to 64. At concurrency 16, DFly-8 achieves 2.22x speedup over autoregressive decoding, compared to 1.99x for DFlash-8 and 1.64x for MTP-3. At concurrency 64, the picture is mixed by dataset: MTP-3 reaches 2.53x on GSM8K against 2.44x for DFly-8, and leads MT-Bench at 1.80x against 1.48x. DFly-8 still holds the highest six-dataset average.
D-cut: dynamic verification budget control
AngelSpec also introduces D-cut, a runtime mechanism that restricts the verification budget to ratios 0.25, 0.50, 0.75, and 1.00, then picks the ratio that maximizes estimated batch progress divided by profiled step latency. Replayed against Hunyuan production traffic on Hy3-295B-A21B with TP=8 and 8x H20, D-cut at concurrency 64 achieves 981 tok/s versus 848 tok/s without D-cut, a 15.7% increase. Mean accepted length only falls from 2.50 to 2.43, while throughput rises significantly. At lower concurrency, gains are smaller but still positive.
What is the break-even committed length for AngelSpec drafters?
The break-even committed length is the minimum number of tokens per iteration a drafter must commit to match the predicted throughput of the baseline Hy3-MTP3 at the same request concurrency. Lower is easier to beat. The Hy3-MTP3 baseline break-even is 3.00 tokens. DFlash L3-B5 requires 3.80, DFlash L5-B8 requires 4.61, DFlash L7-B8 requires 4.56. The DFly backbone requires 3.67, DFly with Markov head requires 5.21, and the final DFly requires 5.14. Target-model execution dominates device-side latency, so proposer depth is not the main cost driver. Hidden correction lands at a lower average latency proxy than the Markov head, which is why it ships as the default in DFly.
Interactive benchmark console for exploring AngelSpec results
To make the benchmark data accessible, Tencent provides an interactive console embedded in the technical report. The console allows users to run queries such as “bench –draft-quality”, “bench –throughput”, “ablate –arch dfly”, “mtp –ttt –rollout”, “dcut –replay live”, and “profile –break-even”. Each command displays per-benchmark mean accepted lengths, throughput tables, ablation steps, and acceptance rates. The console is designed to help practitioners understand the trade-offs and choose the right drafter for their specific workload. All figures are sourced from the AngelSpec technical report.
Implications for production inference serving
AngelSpec’s open-source release gives developers and researchers a torch-native framework to train and deploy speculative-decoding draft models that are specialized for heterogeneous traffic. The combination of MTP for conversational contexts and block-parallel DFlash family for code and math workloads addresses a real pain point in production serving where traffic is not a uniform mixture. The Training-Time Test and target-model rollout techniques improve MTP acceptance rates without altering the target distribution. D-cut provides a lightweight runtime mechanism to adapt verification depth to dynamic batch conditions. As inference workloads continue to diversify, frameworks like AngelSpec that embrace workload heterogeneity rather than averaging across benchmarks will become essential for achieving optimal latency and throughput at scale.
