Baidu releases ERNIE-5.5-Flash-Base weights

Baidu has released the weights for ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts (MoE) language model with 280 billion total parameters, according to the official release notes on the ERNIE platform. The model activates only 14 billion parameters per token, a design choice intended to keep inference costs low while maintaining high capability. It was pre-trained from scratch on 33.5 trillion tokens with a training context of up to 128K tokens, and it supports a 1 million-token context for both fine-tuning and inference. This release provides a checkpoint at the end of pre-training, before any mid- or post-training, allowing developers to use it as a starting point for domain adaptation, instruction tuning, and reinforcement learning research.

Architectural innovations for efficiency

The release highlights a specific focus on efficiency, particularly in how the model handles long contexts and memory usage. The architecture features 43 Transformer layers with a hidden size of 4096 and 64 attention heads. A key component is the hybrid attention mechanism combining Multi-head Latent Attention (MLA) and Heavily Compressed Attention (HCA). In this setup, only one layer in seven uses MLA to maintain token-level access to the full context, while the remaining six layers use HCA. HCA compresses every 128 tokens into a single cached entry, allowing these layers to see the whole sequence while caching only a small fraction of the memory required by full-context layers.

To optimize compute, the model uses layer-specific head dimensions. HCA layers, which attend over a sequence compressed 128 times, use wide 512-dimensional heads because their attention is computationally cheap. Conversely, MLA layers, which attend at the token level and are more expensive, use narrower 256-dimensional heads. The release notes state that in pre-training comparisons at a similar KV-cache budget, the combination of HCA and MLA outperformed HCA combined with Compressed Sparse Attention (CSA).

Routing and output head improvements

The MoE layers in ERNIE-5.5-Flash-Base replace the standard single-router selection with a Multi-Router system. Instead of one router with a hard top-k selection, several independently initialized sigmoid routers score the experts jointly. The release notes explain that this approach allows experts to be chosen because one router strongly favors them or because several favor them together, making wrong choices easier to correct. This reduces sensitivity to noisy features and distribution shift while keeping the number of activated experts and communication volume the same.

The output head also undergoes a significant change with the introduction of MultiMax. This replaces the single softmax temperature with a learnable piecewise temperature. This allows the model to suppress the long tail of implausible tokens while keeping several plausible next tokens, a trade-off that a single temperature cannot achieve. The release notes indicate that this change significantly lowers pre-training loss with negligible training overhead.

Training and tokenization details

ERNIE-5.5-Flash-Base was pre-trained using the Muon optimizer with FP8 mixed precision. The training context was gradually extended to 128K tokens over the course of pre-training. The model uses a new BPE tokenizer with a vocabulary of about 200,000 entries. A notable feature of this tokenizer is that numbers are tokenized right-to-left in groups of up to three digits. For example, the number 1,234,567 is tokenized as 1 234 567. This alignment of digits with the same place value across numbers of different lengths is designed to help with arithmetic and number-heavy text.

The model also incorporates ECHO, a multi-token prediction mechanism that reuses the backbone's own final layer to predict future tokens. Unlike conventional multi-token prediction which attaches extra modules, ECHO ensures that the loss on future tokens directly trains the same parameters that produce the next token. The release notes state that this encourages the backbone to anticipate where the sequence is heading, with clear gains on reasoning and code. The same layer also serves as the draft model for speculative decoding, drafting several tokens ahead with a higher acceptance rate than an auxiliary module trained the same way.

What it means for developers

For developers, the primary value of ERNIE-5.5-Flash-Base lies in its role as a base model for further training. By releasing the weights at the end of pre-training, Baidu allows teams to build on the general capabilities already in place and choose their own direction for domain adaptation. The 1M-token context support is particularly relevant for agentic applications and repository-scale code understanding, where long-context costs are a major bottleneck. The efficiency gains from the hybrid attention and multi-router MoE design should make fine-tuning and inference more affordable compared to dense models of similar capability.

Developers should be aware that this is a base model, not an instruction-tuned chat model. It requires further training for conversational or task-specific use. The novel attention mechanisms and routing strategies may also require specific implementation details for optimal performance, which are detailed in the release notes. The right-to-left number tokenization is a specific design choice that may affect how the model handles numerical data compared to other models.

What to watch

• Fine-tuning results: How well the model performs in domain-specific tasks after instruction tuning and reinforcement learning.
• Inference costs: Real-world benchmarks of the KV-cache and compute efficiency compared to other MoE models.
• Ecosystem integration: Whether the model is integrated into popular frameworks like Hugging Face or vLLM for easier deployment.
• Competitor responses: How other model providers respond to the specific architectural innovations like HCA and Multi-Router.