Prime Intellect has released prime-rl version 0.6.0, an open framework designed for reinforcement learning on trillion-parameter Mixture-of-Experts (MoE) models. The update targets heavy agentic workloads such as long-horizon software-engineering tasks, where rollouts can span hundreds of turns and tool calls. The research team demonstrated the framework by training GLM-5 on SWE tasks at sequence lengths up to 131,000 tokens, with step times staying under five minutes, a batch size of 256 rollouts, and a footprint of just 28 H200 nodes.
What is prime-rl 0.6.0?
prime-rl is an open framework for asynchronous reinforcement learning that post-trains large open-source models on agentic tasks. Version 0.6.0 extends this capability to trillion-parameter MoE scale. The example model used in the announcement is zai-org/GLM-5.1, but the optimizations apply equally to other large MoE models, including moonshotai/Kimi-K2.7-Code and nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16. A full GLM-5.1 run starts with a single command on a Slurm cluster, making the system accessible for teams already working with distributed compute infrastructure.
Why Asynchronous RL Matters for Agentic Tasks
Agentic tasks produce long-tail outliers. Some coding rollouts take hours to complete. Waiting for every rollout to finish before each policy update would leave GPUs idle for extended periods. Asynchronous RL avoids this bottleneck by disaggregating the trainer and inference systems. They run and scale independently. The inference policy updates as soon as the optimizer step finishes, with only one synchronization point: the policy update itself.
prime-rl pushes new weights the moment they exist. Already-dispatched rollouts keep their active prefix cache, so a single rollout may mix tokens from several policy versions. New rollouts repopulate their own KV cache, even when prefixes match, using a KV-cache salt to force this. Requests from policies that are too old are dropped, controlled by the max_off_policy_steps parameter. This design minimizes idle GPU time while maintaining training stability.
Inference Optimizations for Throughput and Latency
Inference is usually the throughput bottleneck in an RL system. prime-rl 0.6.0 applies several optimizations to maximize throughput while keeping latency bounded.
FP8 inference uses lower precision to speed up both prefill and decode, leveraging DeepEP and DeepGEMM kernels. Wide Expert Parallelism (Wide EP) spreads experts across 32 or more GPUs paired with a large data-parallel rank. Each GPU holds separate experts and serves as an endpoint, with synchronization happening per-layer through dispatch and combine operations.
Prefill and decode disaggregation (P/D disaggregation) separates prefill and decode workers. Some model-env pairs hit a 4:1 prefill-to-decode token ratio. Shared workers would inflate end-to-end latency. Separating them ensures long tool outputs do not throttle decode workers.
KV cache management uses tiered offloading to CPU and disk. vLLM native offloading creates one pool per worker, while Mooncake Store pools RAM and disk across all nodes centrally, raising concurrency. Request routing is handled by a forked vllm-router by default, with the NVIDIA Dynamo router available as a drop-in. Routers score workers based on KV cache reuse, queue depth, and live load.
Router replay (R3) captures inference routing decisions and replays them directly on the trainer. This cuts the KL mismatch between trainer and inference by roughly an order of magnitude. The routed expert payload, shaped as [num_layers, top_k, seq_len], can grow to hundreds of gigabytes, reaching data rates of tens of Gbps. prime-rl treats it as an opaque payload and uses optimized PyTorch operations for processing.
Training Optimizations at Trillion-Parameter Scale
The trainer builds on torchtitan, a PyTorch-native training codebase, and relies on three-dimensional parallelism: Fully Sharded Data Parallelism (FSDP), Expert Parallelism (EP), and Context Parallelism (CP). The GLM-5 case study uses all three.
FSDP (FSDP2) shards parameters, gradients, and optimizer states across devices, gathering weights on demand per layer via fully_shard. This provides baseline memory amortization. Expert Parallelism (EP) shards experts within a layer, shrinking active layer memory. With 78 layers and 800 billion parameters in float32, one layer’s all-gather requires roughly 40 GB. Setting EP=8 dispatches tokens instead of gathering full experts. torch-native all2all is slightly faster within one node, while DeepEP wins when EP spans multiple nodes.
Context Parallelism (CP) shards the sequence dimension and is critical at 131,000-plus sequence length, where activations dominate memory rather than parameters. GLM-5 uses a custom attention mechanism (DSA) that neither Ulysses nor Ring Attention parallelizes directly, so prime-rl ships a custom context-parallel implementation for it.
FP8 training uses DeepGEMM block-scaled FP8, as proposed by DeepSeek V3. While this rarely increases throughput due to quantization overhead, its real value is in matching trainer and inference precision, which reduces KL mismatch and stabilizes training.
Use Cases: Who Benefits from prime-rl 0.6.0
Long-horizon SWE agents: Training models on real repository issues with hundreds of turns and tool calls. P/D disaggregation keeps decode latency predictable despite long tool outputs.
1T-scale post-training on fewer nodes: The GLM-5 run fit on 28 H200 nodes, demonstrating that Wide EP and KV offloading can raise concurrency and throughput without requiring massive clusters.
Stable agentic RL at scale: Router replay and FP8 training both reduce trainer-inference KL mismatch, leading to steadier training dynamics and fewer silent failures.
What This Means for Developers and Researchers
For teams working on agentic AI systems, prime-rl 0.6.0 offers a practical path to training trillion-parameter MoE models on reinforcement learning tasks without requiring a datacenter-scale cluster. The framework is open source and available now. Developers can start by cloning the repository, configuring a Slurm cluster, and running a single command to launch a GLM-5.1 training run. The key takeaway is that asynchronous RL, combined with disaggregated inference and targeted parallelism strategies, makes trillion-parameter agentic training feasible on a modest number of nodes. Teams evaluating this approach should consider their own sequence-length requirements, the degree of off-policy drift they can tolerate, and whether their workload benefits from the router replay mechanism to maintain training stability.
