Two Development Vectors for Chinese Labs
A clear divide in strategies has emerged in the Chinese large language model market. While some developers are trying to solve the problem of the exorbitant inference costs of ultra-heavy MoE (Mixture of Experts) networks, others are scaling up the physical size of their computations to previously unseen levels. A series of September leaks sheds light on upcoming releases from Moonshot AI, Zhipu AI, and ByteDance.
The Architectural Foundation of Kimi K3 and the Prerequisites for K4
To understand the purpose of the upcoming Kimi K4, it is necessary to assess the engineering trade-offs of the current K3 version. The model, with a nominal size of 2.8 trillion parameters, uses a complex sparse-attention scheme:
- Activation coefficient: only 104 billion parameters are active when processing a single token (approximately 3.7% of the network’s total weight). The architecture includes 93 layers (1 dense and 92 sparse), and each token is routed to 16 of 896 experts plus two global experts.
- Hybrid attention: 69 layers use Kimi Delta Attention (a linear-attention mechanism for keeping the cost of a 148 576-token context window manageable), while only 24 layers use resource-intensive Gated MLA.
- Quantization during training: the use of Quantization-Aware Training made it possible to build in reduced-precision operation at the pretraining stage, rather than applying post-quantization.
Despite its high scores on independent benchmarks, K3 suffers from heavy inference costs: $3 per million input tokens and $15 per million output tokens, along with low throughput and a tendency toward long-winded phrasing. If the Kimi K4 developers really do reduce the number of simultaneously active parameters below 104 billion, this would allow resources to be reallocated toward reducing latency and lowering API costs for product use cases.
Oddities in the Zhipu AI and DeepSeek Indexes
The leak also mentions GLM-5.5 / GLM-5.4 from Zhipu AI and the V4.1P suffix from DeepSeek. In Zhipu’s case, the appearance of the 5.5 index before 5.4 looks atypical relative to the official lineup (where the latest major model is GLM-5.3 with 743 billion parameters). Most likely, these refer to specialized enterprise variants or internal service-level code names.
Meanwhile, DeepSeek V4.1P may be a Pro variant of the recently released lightweight Flash version, filling the niche between low-cost inference and complex analytical tasks.
ByteDance’s Bet: 10 Trillion Parameters
In contrast to Moonshot’s attempts to make models more compact is ByteDance’s project. According to industry sources, the company is training a model with up to 10 trillion parameters, more than three times the size of Kimi K3. Until now, the largest open Chinese architectures have clustered around 1.6–2.8 trillion parameters.
At the same time, increasing the number of weights inevitably runs up against the law of diminishing returns:
| Parameter | Kimi K3 (Moonshot AI) | ByteDance Project (10T) |
|---|---|---|
| Total number of parameters | 2.8 trillion | Approximately 10 trillion |
| Architecture type | MoE (896 experts, 16 active) | Ultra-high-density MoE |
| Primary objective | Long context and code generation | Generalization and multimodality |
| Expected readiness date | In operation since July 2026 | Winter 2026/2027 (preliminary weights) |
The ultimate value of such giants is determined not by the number of parameters on paper, but by whether it is possible to rein in response latency and make the system operate within reasonable server power consumption limits.
Compare models before you start
The service sets its plans, limits and model catalog. If they differ from this article, contact us so we can update it and record a new review date.
Browse modelsAffiliate link: your price stays the same and the project earns a commission.