The Allen Institute for AI has a habit of doing the unglamorous work nobody else will fund. On October 1, via a Hugging Face blog post (with an HF update roundup following on October 2), Ai2 released Olmo-core 3 — an open-source training framework built for one of the hardest problems in modern AI: training mixture-of-experts models at the trillion-parameter scale without a hyperscaler’s private infrastructure team.
This is infrastructure, not a model. You cannot download Olmo-core 3 and chat with it. What you can do — in principle — is train the next generation of open MoE models with it. Ai2 says the framework will underpin its own next-gen Olmo MoE, which has not been released yet. The release is a bet that the open-source community’s bottleneck is no longer model weights but model training recipes: the distributed-systems engineering that turns a cluster of GPUs into a trillion-parameter model instead of a trillion-parameter paperweight.
Why MoE training is the hard problem
A quick primer on why a framework like this needs to exist. In a mixture-of-experts model, each token is routed to only a few specialist sub-networks (“experts”). That routing decision — made billions of times per training run — creates a distributed-systems headache: experts must be spread across GPUs, tokens shipped to the right expert and results shipped back, and the whole thing kept load-balanced so no GPU sits idle while another drowns. Get the communication patterns wrong and your trillion-parameter model trains slower than a dense model a tenth its size. The techniques Olmo-core 3 assembles — rowwise expert parallelism, GPU-resident routing, grouped GEMMs — are each answers to a specific way this goes wrong. Individually documented in papers; rarely available together in one open codebase.
The headline numbers
Ai2’s reported figures are striking, and — as always with vendor benchmarks — should be read as “what the authors measured on their hardware” rather than universal law:
- Scale: up to 1.2 trillion total parameters (58.36 billion active) trained across 512 GPUs, sustaining 858 TFLOP/s per GPU.
- Throughput: 52,000 tokens per second per GPU on a 47B-parameter MoE, versus 19,400 for the prior FSDP-based approach — about a 2.7× improvement over Ai2’s own previous stack.
- Expert scaling: experts scale from 8 to 128 with 4 active per token, while active parameters hold near 3.2B as the total grows from 4.6B to 47B — with under 5% throughput loss. That near-flat scaling curve is the whole point of MoE: capacity grows while cost per token barely moves.
- Precision: MXFP8 microscaling delivers +21% over BF16, with peak memory dropping from 103 to 95 GiB.
The timing is pointed. Just this week, Mistral previewed Large 4, a 1.05T-parameter MoE trained on 3,800 GPUs behind closed doors. Olmo-core 3 is the open counterweight: the tooling that lets the rest of the world attempt the same class of model.
How it gets there
The engineering details read like a greatest-hits of distributed MoE training. Rowwise expert parallelism spreads experts across devices in a way that keeps communication patterns sane. GPU-resident routing avoids the host-device round trips that quietly murder MoE throughput. Grouped GEMMs batch the many small matrix multiplications that expert layers generate into operations GPUs actually like. On top of that: expert and pipeline parallelism composed together, plus a distributed optimizer so the optimizer state itself — often the silent memory hog in large runs — doesn’t cap your scale.
None of these techniques is individually new. The contribution is composition and openness: a single framework where they work together, with the code public, so a university lab or startup can stand on it instead of re-deriving three years of systems lore. (One honest data-quality note: sources disagree on whether the benchmark GPUs were B300 or B3000-series NVIDIA parts, so treat the hardware attribution as “B300-series” until Ai2 clarifies.)
A word on MXFP8, since precision formats are where modern training efficiency increasingly lives. Microscaling formats like MXFP8 share scaling factors across small blocks of values, preserving dynamic range where it matters while spending fewer bits where it doesn’t. The reported +21% throughput over BF16 with lower peak memory (103 down to 95 GiB) is the kind of win that compounds across a 512-GPU run: faster steps, cheaper steps, and headroom for larger batches. As with everything in this release, independent replication will determine how much of that +21% is format and how much is Ai2’s kernel engineering — but the direction matches the industry’s broader move below BF16.
Why this matters more than another model drop
Open-weights releases get the headlines, but training frameworks decide who gets to produce the next open-weights release. Every closed lab already has its internal equivalent of Olmo-core 3 — that is part of why their models keep scaling. The open ecosystem has been catching up asymmetrically: great inference engines, decent fine-tuning tools, and a conspicuous gap at the “train a frontier-scale MoE from scratch” layer. Olmo-core 3 aims directly at that gap.
The caveats are real. These are Ai2’s own benchmarks with no independent validation yet, and trillion-parameter training remains punishingly expensive no matter how good the framework — 512 GPUs at 858 TFLOP/s each is not a hobbyist setup. But cost curves bend, and today’s 512-GPU run is tomorrow’s routine job. When the next-gen Olmo MoE lands trained on this stack, we will get the fullest test of the claim. Until then, Olmo-core 3 is the most significant open-source infrastructure release of the month — the kind of thing that compounds.
There is also a strategic reading. Ai2 is not a commercial lab; it does not need Olmo-core 3 to defend a product margin. Its incentive is a thriving open ecosystem, and open training infrastructure is the highest-leverage place to invest — every efficiency gain multiplies across every team that builds on it. Contrast that with the closed labs, where equivalent systems are competitive moats. The MoE era will be shaped as much by who shares the tooling as by who trains the biggest model. On that score, October 2026 is off to a good start.
Featured image: the Frontier exascale supercomputer at Oak Ridge National Laboratory (via Wikimedia Commons — CC BY 2.0).
