A Stanford research team has introduced a new system called TRACE that systematically identifies why agentic LLMs repeatedly fail and then builds targeted, synthetic training environments to close those specific capability gaps. The system, released open-source under an MIT license, represents a shift from broad, untargeted reinforcement learning to a structured, diagnostic approach that treats each recurring failure mode as a distinct, trainable problem. By analyzing the contrast between successful and failed trajectories, TRACE uncovers the precise capabilities an agent lacks—rather than relying on generic error labels or sparse rewards from full-task rollouts—and then generates verifiable environments to train each deficit in isolation.
What Problem Does TRACE Solve?
Agentic LLMs often fail repeatedly in the same ways, yet most training approaches cannot pinpoint the missing skill that causes those failures. Direct reinforcement learning (RL) or supervised fine-tuning (SFT) on full-task trajectories provides rewards that are too sparse to indicate which specific capability was absent. Broad synthetic data generation is similarly inefficient, because it distributes training budget evenly across skills the model already possesses and those it lacks. TRACE addresses this by observing that failures are not random: a small set of recurring deficits accounts for the majority of failed trajectories. Each of these deficits can be isolated and turned into its own dense, verifiable training signal.
How Does TRACE Work?
TRACE runs an automated four-step pipeline, with each step driven by an LLM agent following a markdown prompt. The process is designed to be benchmark-agnostic and requires no human labels or LLM judge for reward computation.
Step 1: Contrastive Capability Analysis
The base agent generates rollouts in the target environment. An analysis agent splits these into successful and failed sets and labels every trajectory-capability pair as NAcodecodecode, PRESENTcodecodecode, or LACKINGcodecodecode. A capability is retained only when it is both contrastive and high-coverage—specifically, its contrastive gap must exceed δ = 0.20, and its coverage must exceed ρ = 0.10. This filter ensures the pipeline focuses on skills whose absence is concentrated in failures, not on capabilities that make no meaningful difference.
Step 2: Targeted Environment Synthesis
For each retained capability, a generation agent builds one synthetic environment that isolates that single skill while preserving the target environment’s tool schemas and output format. Task instances are procedurally generated from random seeds, and because generation and verification are algorithmic, rewards require no human labels or LLM-based judge.
Step 3: Capability Adapter Training
Each capability receives its own LoRA (Low-Rank Adaptation) adapter, trained on the synthetic environment using GRPO (Group Relative Policy Optimization). The base model remains frozen throughout. GRPO groups rollouts by shared seed, ensuring scenarios within a group are identical, and rewards are normalized within each group to isolate the policy’s contribution.
Step 4: MoE Composition With Token-Level Routing
TRACE composes the trained adapters into a Mixture-of-Experts (MoE) model. The backbone and adapters remain frozen, and only lightweight token-level gates are trained. At inference, each token is routed top-1 to a single capability adapter, enabling the model to switch experts mid-trajectory based on the current token’s needs.
Which Capabilities Did It Find?
On τ²-Bench, a customer-service benchmark, contrastive analysis recovered four key deficits: structured data reasoning, multi-step task completion, precondition verification, and tool calling precision. These findings remained stable across ten independent runs. Structured data reasoning alone covered the largest share of failed tasks, with multi-step task completion following closely behind.
Use Cases With Examples
To ground these capabilities, consider three concrete tasks. For a customer-service airline agent, a user asks to cancel a basic economy flight booked 14 days ago. Precondition verification checks policy eligibility before the agent calls cancel_reservationcodecodecode. For a compound retail request, a user asks to cancel two reservations and modify a third; multi-step task completion prevents the agent from quitting after completing the first sub-task. For a coding agent on SWE-bench Verified, correctly locating the relevant function or file is a capability necessary for fixing a bug or updating an API call.
How Do the Results Compare?
TRACE was tested on two backbone models and two benchmarks. τ²-Bench measures customer-service pass rate across 50 airline and 114 retail tasks. SWE-bench Verified measures Pass@1 on 500 real GitHub issues.
Performance Summary
On Qwen3-30B-A3B, TRACE improved τ²-Bench by +15.3 points and SWE-bench Verified by +15 points Pass@1, beating the strongest external baselines—GEPA and SWE-RL—by +8.6 and +8.4 points respectively. The system is also sample-efficient: using under one-fourth the rollouts of standard GRPO, it exceeded that method’s final scores by +10.4 points on τ²-Bench. On Qwen3.6-27B, TRACE reached 73.2% Pass@1 on SWE-bench Verified, surpassing GPT-5.2-Codex (72.8%), GLM 5, and Claude 4.5 Sonnet on the public leaderboard, despite being an open-weight model with only 27 billion parameters.
| Backbone | Method | τ²-Bench Overall (%) | SWE-bench Verified Pass@1 (%) |
|---|---|---|---|
| Qwen3-30B-A3B | Base | 32.9 | 26.0 |
| GEPA (prompt optimization) | 39.6 | 31.0 | |
| SWE-RL | — | 32.6 | |
| Single Capability GRPO | 40.3 | 36.6 | |
| TRACE | 48.2 | 41.0 | |
| Qwen3.6-27B | Base | 50.0 | 68.0 |
| GEPA (prompt optimization) | 53.0 | 69.6 | |
| SWE-RL | — | 70.4 | |
| TRACE | 59.1 | 73.2 |
figurefigurefigure
Implementation for Developers
For AI professionals, the pipeline is straightforward to set up. After environments are generated, each capability adapter is trained against a running vLLM server. Each LoRA adapter adds approximately 1.6 billion trainable parameters—5.3% of the backbone—while the trained MoE gate adds just 491,760 parameters in total. Default thresholds are ρ = 0.10, δ = 0.20, and 8-of-10 cross-run consistency.
Key Takeaways
- Failure contrast is actionable. Comparing successful and failed trajectories surfaces trainable deficits, not generic error labels.
- Targeted environments are sample-efficient. Each verifiable environment rewards one capability, so every rollout carries dense signal.
- Composition beats collapse. Separate experts with token-level routing beat the best single-adapter alternative by over seven points.
- Training beats prompting. Trained adapters scale monotonically, while prompt-only optimization plateaus after four capabilities.
- A 27B model topped the leaderboard. Qwen3.6-27B hit 73.2% Pass@1 on SWE-bench Verified, above GPT-5.2-Codex.
What This Means for Developers
TRACE’s approach—diagnose, isolate, train, and compose—offers a practical path for improving agentic LLMs without the cost of full-model retraining or the inefficiency of broad synthetic data. For teams working on task-oriented agents, the open-source release under MIT license means the pipeline can be adapted to custom benchmarks and domains. The immediate next step is to apply TRACE to your own agent: run the contrastive analysis on your rollouts, identify the capabilities your model lacks, and generate targeted environments for those specific deficits. The sample efficiency and open-weight results suggest this diagnostic approach may become a standard component in agent training pipelines.
