Local LLM apps and engines

Strata Single-model engine + one-click installer

An inference engine written for one model, Qwen3.8-Flash-Next (125B MoE, 6B active). It keeps the most-used experts on a 12 GB+ graphics card, all 24,576 experts in system RAM and the 29 GB n-gram lookup table on the SSD, and uses the model's own MTP layer to guess and check several tokens per pass. Version 0.1.45 also runs Qwen3.6-35B-A3B.

Platforms

Windows 10/11, Linux, macOS (Apple Silicon, experimental)

Hardware

NVIDIA RTX 20-50 series or AMD RX 6800 / 7000 / 9000 with 12 GB+ VRAM, 32-64 GB RAM, about 80 GB disk. Community-tested: older Tesla/GTX cards, Intel Arc, Strix Halo.

Formats

GGUF quants of Qwen3.8-Flash-Next (ISTA-DASLab GSQ-RCO, Unsloth, UkisAI Swift 1.5); uses parts of llama.cpp/ggml

API

OpenAI-compatible at http://127.0.0.1:8080/v1, Anthropic /v1/messages, Responses API, browser chat

Good for

Running Qwen3.8-Flash-Next fast on a gaming PC: the project measured 53-94 tokens/s on an RTX 5070 12 GB with 64 GB RAM.

License

MIT

Checked

v0.1.45, released 17 Oct 2026

Models on this site it runs: Qwen3.6 35B-A3B, Qwen3.8-Flash-Next 125B-A6B

Sources: README · Model sizes · v0.1.45 release notes · Paper (PDF)

colibri Disk-streaming MoE engine

A dependency-free C engine that runs very large mixture-of-experts models on ordinary machines. It keeps the dense part in RAM and streams the experts from the disk when they are needed, so a fast NVMe SSD sets the speed. One engine per model family: GLM-5.2/5.3, GLM-5.3-Flash, Kimi K3, DeepSeek V4 / V4.1 Flash, MiMo-V2.6, Qwen3.8-Flash-Next, Qwen3.6 (also Qwen3-Coder and Qwen3.8-27B), Inkling and OLMoE.

Platforms

Linux, Windows, macOS

Hardware

CPU only works; 8 GB RAM minimum (16 GB+ better) and 22 GB disk for the smallest model. Optional GPU: CUDA on Linux, Vulkan on Linux/Windows (any vendor, integrated too), Metal for some engines on Apple Silicon.

Formats

Its own int4/int8 containers, converted once from the Hugging Face safetensors (pre-converted copies on Hugging Face)

API

OpenAI-compatible (coli serve, /v1/chat/completions), web dashboard, terminal chat

Good for

Trying huge MoE models with little RAM, if you accept slow output: in its own tests small models run at 6-30 tokens/s and the 300B-750B ones at roughly 0.05-4 tokens/s on ordinary machines.

License

Apache-2.0

Checked

v2.0.0, released 6 Oct 2026

Models on this site it runs: Qwen3.8 27B, Qwen3-Coder 30B-A3B, Qwen3.6 35B-A3B, Qwen3.8-Flash-Next 125B-A6B, DeepSeek V4 Flash (0731), MiMo-V2.6-Flash 309B-A15B, GLM-5.3-Flash 320B-A18B, GLM-5.3, DeepSeek V4.1 Flash, Kimi K3

Sources: README and model tables · Project site · Quick start

Ollama Desktop app + CLI

The easiest way to download and run a model with one command (ollama run model:tag). It has its own model library; some large models there are cloud-only and do not run locally.

Platforms

macOS, Windows, Linux

Hardware

CPU, NVIDIA, AMD and Apple Silicon

Formats

Models from the Ollama library (GGUF-based; MLX builds on Mac for some models)

API

Local REST API on port 11434, OpenAI-compatible endpoint

Good for

Beginners and quick local APIs.

License

MIT

Checked

Library tags checked 17 Oct 2026

Models on this site it runs: Gemma 4 E2B, Gemma 4 E4B, Llama 3.1 8B, Qwen3 8B, Qwen3.5 9B, Gemma 4 12B, Phi-4 14B, gpt-oss-20b, LFM2.5 24B-A2B, Devstral Small 2 24B, Gemma 4 26B-A4B, Qwen3.8 27B, Qwen3.6 27B, Qwen3-Coder 30B-A3B, GLM-4.7-Flash 30B-A3B, Gemma 4 31B, Nemotron 3 Nano 30B-A3B, Qwen3 32B, Qwen3.6 35B-A3B, Llama 3.3 70B, Llama 4 Scout 17B-16E, gpt-oss-120b, Nemotron 3 Super 120B-A12B, Qwen3.5 122B-A10B, Mistral Medium 3.5 128B, Qwen3.8-Flash-Next 125B-A6B, Qwen3-Coder 480B-A35B, DeepSeek V3.1

Sources: GitHub · Model library

LM Studio Desktop app

A desktop app to find, download and chat with models, with a local server. It uses llama.cpp and, on Apple Silicon, MLX under the hood. Ready builds are published in the lmstudio-community Hugging Face org.

Platforms

macOS, Windows (x64, arm64), Linux (x64, arm64)

Hardware

CPU, NVIDIA, AMD, Apple Silicon

Formats

GGUF and MLX

API

OpenAI-compatible local server; lms CLI

Good for

A polished GUI on Mac and Windows.

License

Free app (closed source); lms CLI is MIT

Checked

Site checked 17 Oct 2026

Models on this site it runs: Gemma 4 E2B, Gemma 4 E4B, Llama 3.1 8B, Qwen3.5 9B, Gemma 4 12B, Phi-4 14B, gpt-oss-20b, Devstral Small 2 24B, Gemma 4 26B-A4B, Qwen3.8 27B, Qwen3.6 27B, Qwen3-Coder 30B-A3B, GLM-4.7-Flash 30B-A3B, Gemma 4 31B, Nemotron 3 Nano 30B-A3B, Qwen3 32B, Qwen3.6 35B-A3B, Llama 3.3 70B, Llama 4 Scout 17B-16E, GLM-4.5-Air, gpt-oss-120b, Mistral Small 4 119B, Nemotron 3 Super 120B-A12B, Qwen3.5 122B-A10B, Qwen3.8-Flash-Next 125B-A6B, MiniMax M2.7, DeepSeek V4 Flash (0731), Qwen3-Coder 480B-A35B

Sources: lmstudio.ai · lmstudio-community models

llama.cpp Inference engine (C/C++)

The engine behind GGUF and most local apps (Ollama, LM Studio, Jan, PocketPal and parts of Strata build on it). llama-server gives a local OpenAI-compatible API; llama-cli a terminal chat.

Platforms

Linux, Windows, macOS, Android (build)

Hardware

CPU, NVIDIA CUDA, AMD, Apple Metal, Vulkan and more

Formats

GGUF

API

llama-server: OpenAI-compatible

Good for

Maximum control and the widest hardware support.

License

MIT

Checked

17 Oct 2026

Models on this site it runs: Gemma 4 E2B, Ling-3.0-tiny 7.9B-A1.3B, Gemma 4 E4B, Llama 3.1 8B, Qwen3 8B, Qwen3.5 9B, Gemma 4 12B, Phi-4 14B, gpt-oss-20b, LFM2.5 24B-A2B, Devstral Small 2 24B, Gemma 4 26B-A4B, Qwen3.8 27B, Qwen3.6 27B, Qwen3-Coder 30B-A3B, GLM-4.7-Flash 30B-A3B, Gemma 4 31B, Nemotron 3 Nano 30B-A3B, Qwen3 32B, Qwen3.6 35B-A3B, Llama 3.3 70B, Llama 4 Scout 17B-16E, GLM-4.5-Air, gpt-oss-120b, Mistral Small 4 119B, Nemotron 3 Super 120B-A12B, Qwen3.5 122B-A10B, Ling-3.0-flash 124B-A5.1B, Mistral Medium 3.5 128B, Qwen3.8-Flash-Next 125B-A6B, Step 3.5 Flash 196B-A11B, MiniMax M2.7, DeepSeek V4 Flash (0731), MiMo-V2.6-Flash 309B-A15B, GLM-5.3-Flash 320B-A18B, Qwen3-Coder 480B-A35B, DeepSeek V3.1, GLM-5.3, Kimi K2 Instruct, Kimi K3

Sources: GitHub · Quantization types

MLX (mlx-lm) Apple framework + CLI

Apple's machine-learning framework; mlx-lm runs and serves LLMs on Apple Silicon. Converted models live in the mlx-community Hugging Face org.

Platforms

macOS (Apple Silicon only)

Hardware

M1-M5 Macs with unified memory

Formats

MLX safetensors (4/6/8-bit, mxfp4)

API

mlx_lm.server: OpenAI-compatible

Good for

The fastest option on many Macs.

License

MIT

Checked

17 Oct 2026

Models on this site it runs: Gemma 4 E2B, Gemma 4 E4B, Llama 3.1 8B, Qwen3 8B, Qwen3.5 9B, Gemma 4 12B, Phi-4 14B, gpt-oss-20b, LFM2.5 24B-A2B, Devstral Small 2 24B, Gemma 4 26B-A4B, Qwen3.8 27B, Qwen3.6 27B, Qwen3-Coder 30B-A3B, GLM-4.7-Flash 30B-A3B, Gemma 4 31B, Nemotron 3 Nano 30B-A3B, Qwen3 32B, Qwen3.6 35B-A3B, Llama 3.3 70B, Llama 4 Scout 17B-16E, GLM-4.5-Air, gpt-oss-120b, Mistral Small 4 119B, Nemotron 3 Super 120B-A12B, Qwen3.5 122B-A10B, Mistral Medium 3.5 128B, Qwen3.8-Flash-Next 125B-A6B, Step 3.5 Flash 196B-A11B, MiniMax M2.7, GLM-5.3-Flash 320B-A18B, Qwen3-Coder 480B-A35B, DeepSeek V3.1, GLM-5.3, Kimi K2 Instruct, DeepSeek V4 Pro

Sources: GitHub · mlx-community models

KTransformers CPU+GPU hybrid engine

A research project for CPU-GPU heterogeneous inference: big MoE models run with the experts on the CPU (INT4/INT8) and the rest on a GPU.

Platforms

Linux, Windows

Hardware

x86 CPU (AMX/AVX-512, AVX2) plus NVIDIA or AMD (ROCm) GPU

Formats

Hugging Face weights, CPU-side INT4/INT8, GPTQ

API

OpenAI-compatible server

Good for

One GPU plus lots of system RAM for large MoE models.

License

Apache-2.0

Checked

17 Oct 2026

Models on this site it runs: Qwen3 8B, Qwen3.5 9B, Qwen3.6 27B, Qwen3-Coder 30B-A3B, Qwen3 32B, Qwen3.6 35B-A3B, Qwen3.5 122B-A10B, Qwen3.8-Flash-Next 125B-A6B, GLM-5.3-Flash 320B-A18B, Qwen3-Coder 480B-A35B, GLM-5.3, Kimi K2 Instruct

Sources: GitHub

vLLM Serving engine

A high-throughput serving engine for many users at once. Most new open models ship vLLM recipes on day one.

Platforms

Linux

Hardware

Mainly datacenter and workstation GPUs (NVIDIA, AMD and others)

Formats

Hugging Face safetensors, FP8, AWQ/GPTQ, NVFP4

API

OpenAI-compatible

Good for

Serving a model to a team when it fits in GPU memory.

License

Apache-2.0

Checked

17 Oct 2026

Models on this site it runs: Ling-3.0-tiny 7.9B-A1.3B, Qwen3 8B, Qwen3.5 9B, gpt-oss-20b, LFM2.5 24B-A2B, Devstral Small 2 24B, Qwen3.8 27B, Qwen3.6 27B, GLM-4.7-Flash 30B-A3B, Nemotron 3 Nano 30B-A3B, Qwen3 32B, Qwen3.6 35B-A3B, Llama 3.3 70B, GLM-4.5-Air, gpt-oss-120b, Mistral Small 4 119B, Nemotron 3 Super 120B-A12B, Qwen3.5 122B-A10B, Ling-3.0-flash 124B-A5.1B, Mistral Medium 3.5 128B, Qwen3.8-Flash-Next 125B-A6B, Step 3.5 Flash 196B-A11B, MiniMax M2.7, DeepSeek V4 Flash (0731), MiMo-V2.6-Flash 309B-A15B, GLM-5.3-Flash 320B-A18B, GLM-5.3, Kimi K2 Instruct, Kimi K3

Sources: GitHub · Recipes

SGLang Serving engine

A fast serving framework for LLMs and multimodal models, with per-model cookbooks.

Platforms

Linux

Hardware

Mainly datacenter and workstation GPUs

Formats

Hugging Face safetensors, FP8 and more

API

OpenAI-compatible

Good for

High-throughput serving, agent workloads.

License

Apache-2.0

Checked

17 Oct 2026

Models on this site it runs: Ling-3.0-tiny 7.9B-A1.3B, Qwen3 8B, Qwen3.5 9B, LFM2.5 24B-A2B, Devstral Small 2 24B, Qwen3.8 27B, Qwen3.6 27B, GLM-4.7-Flash 30B-A3B, Nemotron 3 Nano 30B-A3B, Qwen3 32B, Qwen3.6 35B-A3B, GLM-4.5-Air, Mistral Small 4 119B, Nemotron 3 Super 120B-A12B, Qwen3.5 122B-A10B, Ling-3.0-flash 124B-A5.1B, Mistral Medium 3.5 128B, Qwen3.8-Flash-Next 125B-A6B, Step 3.5 Flash 196B-A11B, MiniMax M2.7, DeepSeek V4 Flash (0731), MiMo-V2.6-Flash 309B-A15B, GLM-5.3-Flash 320B-A18B, GLM-5.3, Kimi K2 Instruct, Kimi K3

Sources: GitHub

TokenSpeed Serving engine

An inference engine from the non-profit LightSeek Foundation, focused on datacenter GPUs (NVIDIA Blackwell, AMD Instinct).

Platforms

Linux

Hardware

Datacenter GPUs

Formats

Hugging Face weights, FP8/FP4

API

OpenAI-compatible

Good for

Datacenter serving; not a home-PC tool.

License

MIT

Checked

17 Oct 2026

Models on this site it runs: Qwen3.8 27B, Qwen3.8-Flash-Next 125B-A6B, GLM-5.3-Flash 320B-A18B, GLM-5.3, Kimi K3

Sources: GitHub

Jan Desktop app

An open-source ChatGPT-style app that runs models offline with a bundled llama.cpp engine. Any GGUF that mainline llama.cpp supports should work.

Platforms

Windows, macOS, Linux

Hardware

CPU, NVIDIA, AMD, Apple Silicon

Formats

GGUF

API

Local OpenAI-compatible server

Good for

An open-source alternative to LM Studio.

License

Open source (see repo)

Checked

17 Oct 2026

Sources: GitHub

PocketPal AI Phone app

Runs small GGUF models fully offline on a phone, using llama.cpp. You pick and download models from Hugging Face inside the app.

Platforms

iOS / iPadOS, Android

Hardware

Phone CPU, GPU (Metal on iOS, OpenCL on Adreno) and Qualcomm Hexagon NPU

Formats

GGUF

Good for

Small models (about 1-8B, such as Gemma 4 E2B/E4B or Ling-3.0-tiny GGUFs) on a phone.

License

MIT

Checked

17 Oct 2026

Sources: GitHub · App Store · Google Play