Haidass1.5-143M

English | 中文

Timeline

🎉 [2026.8.29] We achieved 4th place on the public leaderboard OpenSLM Leaderboard! 🚀🚀🚀 This is the first multilingual model on the leaderboard whose entire training process was completed on the Ascend ecosystem.

🎉 [2026.8.28] We released the Haidass1.5-143M ! Building on our previous work Haidass-143M-v1, we expanded data sources and further increased the training volume to approximately 400B tokens. Additionally, we adopted a multi-stage training strategy, adjusting data mixture strategies across different stages. Through these approaches, the model showed improvements across multiple metrics.

🎉 [2026.8.12] We achieved 2nd place on the public leaderboard Tiny-ML-Leaderboard!

🎉 [2026.8.10] We released Haidass-143M-v1! The model was trained on the Ascend ecosystem, using 8 Atlas A2 servers (64*910B). The pretraining phase reached approximately 100B tokens, with training data including Chinese, English, and mathematics.

Model Overview

A bilingual (English/Chinese) small language model trained entirely on the Huawei Ascend NPU ecosystem.

Haidass1.5-143M is a 143M-parameter bilingual language model trained from scratch on approximately 400B tokens of English and Chinese data. The entire training pipeline runs on the Huawei Ascend ecosystem, using the MindSpeed-LLM framework on Atlas A2 servers (910B). A custom 64,000-token bilingual vocabulary was trained alongside the model. This model is competitive among multilingual models under 150M parameters and ranks favorably across multiple evaluation benchmarks.

Model Architecture

Parameter	Value
Architecture	Qwen3
Layers	30
Hidden size	576
Attention heads	9
KV heads (GQA)	3
Head dim	64
FFN intermediate size	1,536
Vocabulary size	64,000
Max sequence length	4,096
Tie word embeddings	Yes
Activation	SwiGLU (SiLU)
Normalization	RMSNorm (eps=1e-6)
Position encoding	RoPE (θ=100,000)
Precision	BF16
Total parameters	~143M

Training Data

Pretraining Phase (~400B tokens)

Primary data sources:

• openbmb/Ultra-FineWeb（ultrafineweb-en + ultrafineweb-zh）
• mlfoundations/dclm-baseline-1.0-parquet（dclm）
• HuggingFaceTB/finemath（finemath-4plus）
• openbmb/Ultra-FineWeb-L3
• HuggingFaceTB/cosmopedia

Training Configuration

Parameter	Value
Framework	MindSpeed-LLM (v2.3.0)
NPU model	Huawei Ascend 910B
Global batch size	128
Sequence length	4,096

Optimizer

Parameter	Value
Optimizer	AdamW
Peak learning rate	1.5e-3
Min learning rate	0
Weight decay	1e-5
Gradient clipping	2.0
Adam β1	0.9
Adam β2	0.95

Tokenizer

Property	Value
Vocabulary size	64,000
Language coverage	English + Chinese

Evaluation

Evaluated using the lm-evaluation-harness framework, zero-shot.

Benchmark	Score
ARC-Easy	59.09
ARC-Challenge	28.33
PIQA	68.72
HellaSwag	40.54
OpenBookQA	31.2
Winogrande	51.78
agi_eval	25.93

Key Features

• Fully Ascend-native: Trained entirely on Huawei Ascend 910B NPUs using the MindSpeed-LLM framework
• Bilingual: Trained on a mixture of English and Chinese data

Intended Use

This is a research model, suitable for:

• Studying training dynamics of small models on Ascend NPUs
• English/Chinese language modeling research
• Serving as a base model for fine-tuning or annealing experiments

Limitations

• Small model scale; reasoning and generation capabilities are limited
• No instruction tuning — raw pretrained model only
• No RLHF or alignment training

Comming Soon

We will soon release the complete training pipeline along with accompanying experiments and related work, including:

• Training pipeline design
• Synthetic data pipeline
• Training data selection methodology and data mixture strategies
• Training pipeline adaptation for the Ascend ecosystem and cluster environment configuration
• MindEval evaluation framework
• Issues encountered during training and their solutions ...

Stay tuned!

Citation

@misc{Haidass1.5-143M,
  title={Haidass1.5-143M: A Bilingual Small Language Model Trained on Ascend 910B},
  year={2026},
  note={Trained from scratch on 400B tokens using MindSpeed-LLM on Ascend 910B NPUs}
}

License

Apache 2.0