Models / MiMo-V2.6-Flash 309B-A15B
Can I run MiMo-V2.6-Flash 309B-A15B locally?
Xiaomi · 311B parameters · mixture of experts, 15B active per token · up to 1024K context · license mit · Hugging Face
At Q4_K_M with a 32K context, MiMo-V2.6-Flash 309B-A15B needs about 198 GB of GPU or unified memory by our estimate. The official checkpoint (FP8) is 173 GB on disk.
9 full-attention layers (4 KV heads, 192-dim keys and 128-dim values, averaged to 160) and 39 sliding-window layers with a 128-token window, from config.json. The sliding layers use 8 KV heads; with a 128-token window that cache stays tiny.
Memory by quantization and context
| Quantization | Weights | Total at 8K | Total at 32K | Total at 128K |
|---|---|---|---|---|
| Q3_K_M (~4.0 bit) | 155 GB | 161 GB | 162 GB | 164 GB |
| Q4_K_M (~4.9 bit, popular default) | 190 GB | 197 GB | 198 GB | 200 GB |
| Q5_K_M (~5.7 bit) | 222 GB | 229 GB | 230 GB | 232 GB |
| Q6_K (~6.6 bit) | 255 GB | 264 GB | 264 GB | 267 GB |
| Q8_0 (~8.5 bit, near-lossless) | 330 GB | 341 GB | 342 GB | 344 GB |
| F16 / BF16 (16 bit) | 622 GB | 641 GB | 642 GB | 644 GB |
Bits per weight from llama.cpp's quantize docs. F16 KV cache. Totals include our runtime-overhead estimate.
Cheapest hardware that fits (Q4_K_M, 32K)
- 3× NVIDIA A100 80GB PCIe: about €44,997 (used, ebay.nl, 237 GB usable)
- NVIDIA DGX Station (GB300): about €83,738 (new, configurator.exxactcorp.com, 740 GB usable)
Runs in
| Tool | Platforms | How | Source |
|---|---|---|---|
| colibri Disk-streaming MoE engine | Linux, Windows, macOS | coli setup --model mimo-v2.6-flash (experts stream from SSD; 32/52 GB RAM) | source |
| llama.cpp Inference engine (C/C++) | Linux, Windows, macOS, Android (build) | GGUF: ggml-org/MiMo-V2.6-Flash-RL-GGUF | source |
| vLLM Serving engine | Linux | named in the official model card | source |
| SGLang Serving engine | Linux | named in the official model card | source |
Only tools with a public source are listed: an official build (ggml-org, lmstudio-community, mlx-community, bartowski or Unsloth on Hugging Face), the model card, ollama.com or the tool's own README. Apps built on llama.cpp, such as Jan and PocketPal AI, can usually load the same GGUF files. All local apps.
Try other settings
9 full-attention layers (4 KV heads, 192-dim keys and 128-dim values, averaged to 160) and 39 sliding-window layers with a 128-token window, from config.json. The sliding layers use 8 KV heads; with a 128-token window that cache stays tiny.
2 hardware options fit
Cheapest real offer per device from AI Hardware Deals, converted to EUR at approximate rates. Multi-GPU rows multiply the card price.
| Setup | Headroom | ||||
|---|---|---|---|---|---|
| NVIDIA DGX Station (GB300) unified memory | 1 device | 740 GB | 542 GB | €83,738 new, configurator.exxactcorp.com | ≤ 774 tok/s |
| Apple Mac Studio M5 Ultra 512GB (2026) unified memory | 1 device | 384 GB | 186 GB | no verified price | ≤ 131 tok/s |
Speed ceiling is memory bandwidth divided by the bytes of (active) weights read per token, shown only where we have a sourced bandwidth figure. Real decode speed is lower, especially with long contexts or when splitting across cards. All memory numbers are estimates; see the methodology.
FAQ
- How much VRAM does MiMo-V2.6-Flash 309B-A15B need?
- About 198 GB at Q4_K_M with a 32K-token context (190 GB weights, 0.8 GB KV cache, 6.7 GB overhead). These are estimates.
- Can I run MiMo-V2.6-Flash 309B-A15B on a single GPU?
- Not at Q4_K_M with 32K context on any single GPU in our catalog. Use several GPUs, a high-memory unified-memory machine, a smaller quantization or a shorter context.
- Which apps can run MiMo-V2.6-Flash 309B-A15B locally?
- Verified so far: colibri, llama.cpp, vLLM, SGLang. Each entry links to its source on this page.