Compression · Shipped · open weights
Expert pruning
Cutting experts out of large mixture-of-experts models until they fit the hardware people actually own.
The idea
A mixture-of-experts model is mostly idle on any given token. The router picks a few experts and the rest sit in memory, paid for in VRAM and in the price of the machine that holds them. For anyone running locally or inside an air-gapped network, that idle weight is the whole obstacle.
So we remove experts rather than shrink everything at once. The architecture and training survive; what changes is how many experts remain. The result drops into most inference stacks unchanged — same interface, lower latency, smaller footprint, more concurrent users per GPU.
Where we are
Our first release removes roughly a quarter of the experts from a frontier coding model, and it is the most-downloaded original work we have published. A deeper cut is in development, along with the same treatment for another frontier reasoning model.
We are expanding the evaluation set before claiming how much capability each cut preserves. Today the fair statement is that these models are good enough for most tasks at a much better cost profile — with the numbers to follow rather than asserted now.
Models from this thread