Moonshot AI has officially launched Kimi K3, a 2.8-trillion-parameter open model that the company is calling the world’s first 3T-class open-weight artificial intelligence system. The model introduces a sparse Mixture-of-Experts (MoE) architecture with native vision capabilities and a 1-million-token context window, representing a significant new upper bound in publicly available model scale. Kimi K3 is built on two proprietary architectural innovations: Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), both of which fundamentally alter how information flows across sequence length and model depth. The release targets demanding long-horizon coding, complex knowledge work, and multi-step reasoning tasks where context retention and computational efficiency are critical.
What Is the Kimi K3 Model?
Kimi K3 is a sparse MoE model that effectively activates 16 of its 896 expert modules for each token it processes. This level of sparsity requires sophisticated routing and optimization, which Moonshot addressed with several supporting techniques including Quantile Balancing for expert allocation, Per-Head Muon for attention head optimization, Sigmoid Tanh Unit (SiTU) for activation control, and Gated MLA for fine-grained attention selectivity. The model follows nine months of continuous scaling leadership from Moonshot, where previous Kimi models had already set the size benchmark for open models in eight of the prior twelve months.
How Does Kimi Delta Attention Work?
Kimi Delta Attention (KDA) is a hybrid linear attention mechanism designed to improve decoding efficiency in extremely long contexts. Moonshot states that KDA enables up to 6.3 times faster decoding when operating within million-token sequences compared to standard quadratic attention mechanisms. This improvement is particularly relevant for use cases such as codebase analysis, document processing, and agentic workflows that require maintaining coherence across very long inputs. The implementation also posed new challenges for prefix caching, which Moonshot addressed by contributing a dedicated implementation to the vLLM serving framework.
Attention Residuals and the Architecture of Depth
Attention Residuals (AttnRes) operate along the depth axis of the model rather than the sequence axis. Instead of accumulating representations uniformly across every layer, AttnRes selectively retrieves representations from earlier layers based on their relevance to the current computation. Moonshot reports that this method delivers roughly 25 percent higher training efficiency at under 2 percent additional computational cost. When combined with refined training and data recipes that accompany the structural changes, the overall scaling efficiency is approximately 2.5 times better than that of Kimi K2, the previous generation model.
How Does Kimi K3 Perform Against Leading Models?
Moonshot has published a comprehensive evaluation table positioning Kimi K3 against proprietary models such as Claude Fable 5, GPT 5.6 Sol, Claude Opus 4.8, GPT 5.5, and GLM-5.2. Across the company’s own evaluation suite, Kimi K3 consistently outperformed all tested models on most benchmarks. However, Moonshot is direct about where K3 sits in the broader landscape: overall performance still trails the most powerful proprietary systems on certain high-difficulty reasoning and agentic benchmarks. Specific results include a score of 81.2 percent on FrontierSWE, 91.2 percent on BrowseComp (with context compaction at 300K tokens), 93.5 percent on GPQA-Diamond, and 84.8 percent on CharXiv RQ for vision tasks. These figures were measured with reasoning effort set to maximum, and the harnesses used varied by benchmark between KimiCode, Claude Code, and Codex.
What Makes the K3 Architecture Distinct?
The model uses Stable LatentMoE as the core mixture of experts framework, with a sparsity level that activates only 16 experts out of 896 available. Quantile Balancing derives expert allocation directly from router-score quantiles, eliminating the need for heuristic update rules and removing a sensitive balancing hyperparameter that often plagues MoE training. Per-Head Muon extends the Muon optimizer to treat attention heads independently, while SiTU and Gated MLA improve activation control and attention selectivity respectively. For serving, K3 applies quantization-aware training beginning at the supervised fine-tuning (SFT) stage. Inference uses MXFP4 weights with MXFP8 activations, a combination chosen for broad hardware compatibility across different accelerator architectures. Moonshot recommends supernode configurations with 64 or more accelerators for running K3 in production environments.
Who Should Explore Kimi K3 Now
For AI researchers, developers building long-context applications, and engineering teams working on agentic coding systems, Kimi K3 represents an immediately available open model that pushes the known frontier of scale and efficiency. The 1-million-token native context window removes the need for tiered pricing or sliding-window workarounds in many professional use cases. Developers should review the vLLM integration for KDA prefix caching, evaluate whether the MXFP4/MXFP8 quantization path aligns with their hardware stack, and consider whether K3’s benchmark profile suits their specific task requirements. Because the model is open and the architectural innovations are documented, teams can also study the KDA and AttnRes mechanisms as potential patterns for their own model development work. The full set of model weights, technical documentation, and serving guidelines are available through the Kimi blog and API platform.
