Models / Kimi K3
Can I run Kimi K3 locally?
Moonshot AI · 2.78T parameters · mixture of experts, 104B active per token · up to 1024K context · license other · Hugging Face
At Q4_K_M with a 32K context, Kimi K3 needs about 1754 GB of GPU or unified memory by our estimate. The official checkpoint (COMPRESSED-TENSORS) is 1561 GB on disk.
Hybrid model: only the latent-attention (MLA) layers keep a growing KV cache; the linear-attention layers keep a small fixed state (not counted).
Memory by quantization and context
| Quantization | Weights | Total at 8K | Total at 32K | Total at 128K |
|---|---|---|---|---|
| Q3_K_M (~4.0 bit) | 1389 GB | 1431 GB | 1432 GB | 1435 GB |
| Q4_K_M (~4.9 bit, popular default) | 1701 GB | 1753 GB | 1754 GB | 1756 GB |
| Q5_K_M (~5.7 bit) | 1982 GB | 2043 GB | 2043 GB | 2046 GB |
| Q6_K (~6.6 bit) | 2281 GB | 2350 GB | 2351 GB | 2354 GB |
| Q8_0 (~8.5 bit, near-lossless) | 2954 GB | 3044 GB | 3044 GB | 3047 GB |
| F16 / BF16 (16 bit) | 5560 GB | 5728 GB | 5729 GB | 5731 GB |
Bits per weight from llama.cpp's quantize docs. F16 KV cache. Totals include our runtime-overhead estimate.
Cheapest hardware that fits (Q4_K_M, 32K)
No device or multi-GPU setup (up to 4 cards) in our catalog with a current price fits this. Try a lower quantization in the calculator below.
Runs in
| Tool | Platforms | How | Source |
|---|---|---|---|
| colibri Disk-streaming MoE engine | Linux, Windows, macOS | coli setup --model kimi-k3 (experts stream from SSD; 32/64 GB RAM, about 9 s per token) | source |
| llama.cpp fork only Inference engine (C/C++) | Linux, Windows, macOS, Android (build) | Unsloth GGUF needs Unsloth's llama.cpp fork (PR #48), not mainline yet | source |
| vLLM Serving engine | Linux | named in the official model card | source |
| SGLang Serving engine | Linux | named in the official model card | source |
| TokenSpeed Serving engine | Linux | named in the official model card | source |
Ollama: cloud only, it does not download the model to your machine.
Only tools with a public source are listed: an official build (ggml-org, lmstudio-community, mlx-community, bartowski or Unsloth on Hugging Face), the model card, ollama.com or the tool's own README. Apps built on llama.cpp, such as Jan and PocketPal AI, can usually load the same GGUF files. All local apps.
Try other settings
Hybrid model: only the latent-attention (MLA) layers keep a growing KV cache; the linear-attention layers keep a small fixed state (not counted).
Nothing in our catalog fits this setup
Cheapest real offer per device from AI Hardware Deals, converted to EUR at approximate rates. Multi-GPU rows multiply the card price. Try a smaller quantization, shorter context or more GPUs.
| Setup | Headroom |
|---|
Speed ceiling is memory bandwidth divided by the bytes of (active) weights read per token, shown only where we have a sourced bandwidth figure. Real decode speed is lower, especially with long contexts or when splitting across cards. All memory numbers are estimates; see the methodology.
FAQ
- How much VRAM does Kimi K3 need?
- About 1754 GB at Q4_K_M with a 32K-token context (1701 GB weights, 0.9 GB KV cache, 52.0 GB overhead). These are estimates.
- Can I run Kimi K3 on a single GPU?
- Not at Q4_K_M with 32K context on any single GPU in our catalog. Use several GPUs, a high-memory unified-memory machine, a smaller quantization or a shorter context.
- Which apps can run Kimi K3 locally?
- Verified so far: colibri, vLLM, SGLang, TokenSpeed. Each entry links to its source on this page.