Kolibri: Aleph Alpha's Apache-2.0 MoE That Puts German First

On the Day of German Reunification — a date chosen for symbolism — Aleph Alpha released Kolibri-1, an English–German mixture-of-experts transformer with 78.1B total parameters of which only 3.46B (4.4%) are active per token, under an Apache 2.0 licence. The weights are on Hugging Face, and the accompanying 189-page tech report is unusually candid: it documents the abort experiments, load-balancing failures, and the tokeniser that went back to the drawing board because "German is not English". What makes Kolibri interesting is not that a European lab shipped a big model — it is that every design decision serves one client: a German federal agency or regulated enterprise that needs the model on its own infrastructure, in German, under EU law.

One constraint, three design choices

The report is explicit about the target customer: public administration, industry, and aerospace — domains where a language model "has to run on infrastructure that the organisation controls, rest on training in line with European values, and serve German as well as English". Everything else falls out of that. Serving cost limits deployment scale, so the model must be cheap to run; German quality needs dedicated data and a tokeniser, not a translation layer; and regulation (EU AI Act, GPAI Code of Practice, GDPR) has to be designed in, not bolted on afterwards. Aleph Alpha trained the model end-to-end in Germany and Finland on 768 NVIDIA B200s and signed the EU GPAI Code of Practice.

Hybrid attention and 4.4% sparsity

Kolibri is a 50-layer decoder with two ideas that keep per-token compute low:

• Hybrid sliding-window attention. 40 of 50 blocks use sliding-window attention (SWA) over the 512 preceding tokens with RoPE (θ = 10000); every fifth block uses full attention without any positional encoding. The report attributes the NoPE-on-full-attention choice to better long-context performance, echoing Yang et al. QK normalisation per head (RMSNorm before RoPE) not only stabilises training but keeps every query and key within the FP8 range — so the FP8 KV cache cannot overflow, a detail that matters when you serve 256k-token sequences.
• Deep sparsity. Each MoE layer routes a token to 6 of 384 routed experts plus one shared expert (SiLU-gated FFNs, hidden dimension 512), scored by sigmoid router logits without renormalisation. Expert load is balanced with Exact Quantile Balancing and Load-Error Injection, avoiding the collapse that plagued earlier MoE training runs.

Trained context is 256k tokens; the model is tested up to 1M — the pitch being RAG over full contract archives or regulatory filings without chunking.

Figure: one token through Kolibri — the router picks 6 of 384 experts per layer while a shared expert is always on; 10 of 50 blocks attend globally without positional encoding.

UniBPE: a tokeniser built on German morphology

German compounds mean a byte-pair encoding trained on English data fragments German into the wrong units. Aleph Alpha's answer is UniBPE: BPE's bottom-up merging, but each merge is scored by the reduction in Unigram cross-entropy rather than raw pair frequency. The closed-form loss update needs no new statistics over BPE.

The effect is measurable. On German web text the Kolibri tokeniser places 65% of split points on morpheme boundaries (GPT-5: 47%) and needs 11.2% fewer tokens than GPT-5 — the best of the nine external tokenisers compared — while reaching 4.90 bytes/token, the highest compression of all twelve tokenisers evaluated on FineWeb-2 German. On English it stays close to GPT-5 (4.58 vs 4.67 bytes/token). In proxy ablations matched in language-modelling quality, the tokeniser's morphological alignment translated into better long-context retrieval recall at every length. For anyone serving German documents, fewer tokens means cheaper inference and longer effective context — a real cost asymmetry, not a benchmark trophy.

Closing the German data gap

21.3% of pre-training tokens are German — across a roughly 24T-token run. Getting there was the hard part: after deduplication the open German corpora yielded only ~390B tokens. Aleph Alpha closed the gap in-house rather than by machine-translation:

• a German-specific Common Crawl curation pipeline (they had to retune English filters — German administrative prose routinely exceeds English mean-word-length bounds, so the standard filters silently delete the register public administration actually writes in), yielding ~1.3T unique tokens of organic German;
• synthetic German data and rephrasing from organic sources;
• translation used sparingly (6%), because translated text "carries the cultural fingerprint of its source language".

The result is a 2.4T-token German pool, 80% of it curated or generated by Aleph Alpha itself — and German lands as 21.3% of pre-training tokens. The goal is a model bilingual by design, not an English model that has read some German.

Post-training: cold-start German reasoning, then RL

SFT runs on 537B tokens and deliberately cold-starts reasoning in German across all domains — the model reasons in German rather than merely answering in it. Then reinforcement learning on more than 1.2M internally curated tasks (reasoning, tool use, instruction following, code, retrieval) with verifiable rewards: advantage normalisation skipped (Dr. GRPO), DAPO-style decoupled clipping and dynamic sampling, plus targeted On-Policy Self-Distillation (OPSD) bootstrapping failing groups with hints. One notable contribution is a per-token logit-gradient norm bound (Gmax = 2): instead of capping the importance-weight ratio like CISPO, they cap the actual gradient a single token can put on the logits, which accounts for the softmax factor that ratio caps ignore.

Two evaluation behaviours stand out. On "I don't know" questions Kolibri Base sends probability mass towards abstention and correctness second only to Nemotron 3 Super Base (120B-A12B); post-trained, it abstains instead of answering wrongly and hallucinates markedly less than its predecessor. And a reasoning-effort level per request (none/low/medium/high) lets an operator trade latency and cost against quality, with thresholds per dataset and difficulty from the 30th/70th percentiles of reasoning length.

On the Pareto frontier

On the report's Overall aggregates, Kolibri scores 0.8 points above Qwen3.5 35B-A3B in English and 1.0 in German at a near-identical active count, with the best Overall among all compared MoEs in both languages. It beats every compared MoE — including Nemotron 3 Super and GLM-4.5 Air at ~4× the active parameters — on AIME 2025/2026 in English; only the dense Qwen3.8 27B scores higher overall, while activating roughly eight times as many parameters per token.

Model	Total / active	Type	Languages	Context	Overall (EN/DE)
Kolibri 78.1B	78.1B / 3.46B	MoE	DE + EN	256k → 1M	best MoE, Pareto frontier
Qwen3.5 35B-A3B	35B / 3B	MoE	multilingual	—	+0.8/+1.0 below Kolibri
Qwen3.8 27B	27B / 27B	dense	multilingual	—	highest overall, ~8× active
Nemotron 3 Super	120B / 12B	MoE	EN	—	strong base; below on AIME
Mistral Small 4	119B / 6B	MoE	EN/FR/etc.	—	mid-table

The honest caveats: Qwen3.8 27B is better overall (at ~8× the serving cost per token); the German benchmark coverage is thinner than the English one; and every number comes from Aleph Alpha's own evaluation suite. Independent replication would strengthen the claims. But the Pareto-frontier framing is the correct one — Kolibri's bet is that a 3.5B-active model is enough for regulated German workloads at a fraction of the compute.

Savanna: the model factory is the real product

Buried in Part 4 is arguably the most reusable output: Savanna, their "Model Training as Code" factory. A training run is a declarative TOML file pinned to immutable versions of code, tokeniser and data. Jobs run on Kueue with topology-aware scheduling within a single InfiniBand domain; orchestration is Flyte; state and queue load are observed via Grafana; the artifact registry never mutates. They practice trunk-based development with a nightly end-to-end mini-training that catches regressions. Agents increasingly run in the loop — reviewing experiments, triaging failed runs, marking limitations with # XXX comments for their successors. That Kolibri climbed faster than Kolibri Origin on the same wall-clock window is the factory's sales pitch: the pipeline, not the checkpoint, is the moat.

Apache 2.0 weights, closed factory

The licence nuance matters for sovereign deployments. Apache 2.0 covers the weights and configuration: you may run, modify, fine-tune and sell services built on them. You do not get the training code, data curation recipes, or the Model Factory — Aleph Alpha explicitly retains those. "Open weights" is a deployment licence, not reproducible research. For a public-sector buyer that is close to ideal — no royalties, no data egress, and a vendor that signed the EU GPAI Code of Practice. For a researcher, the closed data pipeline is the frustrating part.

Deploying Kolibri

The model card lists ~78GB of FP8 weights. Serving uses a vLLM plugin — note the custom reasoning and tool-call parsers:

pip install "aleph-alpha-inference>=1.0"

vllm serve Aleph-Alpha/Kolibri-1 \
  --kv-cache-dtype fp8 \
  --reasoning-parser kolibri1 \
  --tool-call-parser kolibri1 \
  --enable-auto-tool-choice

Hardware: where it runs, and how fast

Kolibri is small enough to run locally and sparse enough that serving is memory-bound, not compute-bound. The FP8 weights are ~78GB, and every decoded token re-reads the active ~3.5GB of them from memory — so single-stream speed is roughly memory bandwidth divided by that read, at a realistic ~70% vLLM efficiency:

Hardware	Memory	Bandwidth	FP8 fit	Est. single-stream decode
B200	180GB HBM3e	7.7 TB/s	comfortable, + 1M context	~1,500 tok/s
H200	141GB HBM3e	4.8 TB/s	comfortable	~1,000 tok/s
RTX PRO 6000 Blackwell	96GB GDDR7	1.8 TB/s	fits, little slack	~360 tok/s
RTX 6000 Ada	48GB GDDR7	0.96 TB/s	no — two cards via NVLink	~400 tok/s (2×)
DGX Spark (GB10)	128GB unified	273 GB/s	fits (78GB weights)	~55 tok/s

The model card's "minimum two A100 80GB" is effectively a BF16-era instruction — 156GB of BF16 weights across 160GB of HBM. In FP8, a single 96GB+ device holds the model alone. Prefill and massively batched throughput scale with FP8 FLOPs instead (B200: 4.5 PFLOPS) — that is where datacentre cards pull ahead of the Spark by an order of magnitude. For interactive chat the Spark's ~55 tok/s already sits above the roughly 30-50 tok/s that reads as fluid — which is why a 140W desk box is a credible sovereign serving node for this model.

KV cache: the part worth calculating

Hybrid attention pays off exactly where deployments feel it. Only the 10 full-attention blocks accumulate KV state; the 40 sliding-window blocks cap theirs at 512 tokens. With 4 KV heads × 128 dims, one token costs 1KB per full block in FP8 (the --kv-cache-dtype fp8 in the deploy command, made safe by QK-RMSNorm), and all SWA blocks together add a constant ~21MB:

Context	Full-attention KV	SWA (constant)	Total FP8 KV
128k	1.34 GB	21 MB	~1.3 GB
256k	2.68 GB	21 MB	~2.7 GB
1M	10.74 GB	21 MB	~11 GB

A full million-token context is an 11GB cache, not the ~54GB a dense 50-full-block equivalent at the same KV width would need — KV pressure rarely limits concurrency the way it does elsewhere. The report's own estimate (Table 31, ~80B/3.4B-active model, 256k-token requests, FP8): 31 simultaneous users on one B200, 18 on two H100s, 67 on four. A 96GB workstation card still supports a handful of 256k sessions, a DGX Spark holds weights plus a full 1M context with room to spare — and with BF16 KV, every number above doubles.

Figure: the same cache at 128k, 256k and 1M tokens in FP8. Only the 10 full-attention blocks grow with context; the 40 sliding-window blocks stay at a constant ~21MB — so a million-token context is an 11GB cache, against ~54GB for a dense 50-block equivalent.

Can you cluster Sparks? The sweet spot is small

DGX Sparks link over ConnectX-7 at 200GbE (25 GB/s per node) — and that is the trap: local memory runs at 273 GB/s, so the fabric delivers under a tenth of it. Kolibri already fits on one Spark, so a second only earns its keep for BF16 weights (156GB across two nodes) or extra long-context headroom — not for speed. Sharding the model routes that ~3.5GB-per-token read over the fabric, and decode collapses to roughly the interconnect rate, 15-25 tok/s. One B200's ~1,500 tok/s is worth about 28 Sparks. Beyond that, a GB10 rack only pays off as many independent sovereign replicas (one per desk) or for training-scale parallelism, where the 200GbE fabric is again the bind. Sweet spot: one Spark per desk; two at most, for BF16; beyond that, buy a B200.

The takeaway: Kolibri is a European-origin open-weight model whose whole design — tokeniser, data mix, sparsity, abstention, licence — is shaped around one buyer, the regulated German enterprise, and it makes a coherent engineering argument for why "small active, big router, German-first" is the right shape for that buyer. If you serve German text at scale, the 11.2% tokeniser saving alone is worth measuring on your own corpus.