Strata / Qwen3.8-Flash-Next
A 125B Qwen3.8-Flash-Next running on one RTX 5060 Ti: 16 GB VRAM, 64K context, 74–110 tok/s decode in a real session.
Started the engine from scratch and measured it: picked the quant, configured the expert cache and context, then verified speed with real conversations and long prompts — recording what this machine actually does.
- 01LOAD
Resident in 25.4 s
64 GB of weights sit on the SSD and are mapped on demand; cold start to answering took 25.4 s, then the engine stays resident.
- 02CACHE
6,496 experts stay in VRAM
An 8.7 GB expert cache hits 82.5–91.4%; the rest stream in per request over PCIe Gen5 x8, a few hundred MB to about 1 GB a second.
- 03MEASURE
Numbers from real requests
74.3 / 75.2 / 81.8 tok/s on three short answers, 92.5–110.5 tok/s on longer turns — read off the Monitor panel, not estimated by a benchmark script.
Strata serves Qwen3.8-Flash-Next from a single RTX 5060 Ti (16 GB VRAM, 47 GB RAM). The model is split into two IQ2_XS GGUF files (about 64 GB) on the SSD; once the engine is resident, 6,496 experts (8.7 GB) stay in VRAM and each request streams only about 8–17% of its experts over PCIe Gen5 x8 — which is how a 16 GB card keeps producing tokens.
Numbers from these real sessions: 25.4 s from cold start to ready; a short Chinese answer at 149 tokens, 75.2 tok/s; an English one at 148 tokens, 81.8 tok/s; longer turns between 92.5 and 110.5 tok/s; prompts around 4,000 tokens prefilled at 1,091–1,449 tok/s. These are measurements from this machine at this quantization, not a guarantee for other hardware — the numbers move with the card, the quant and the context length.
Experiments describe their actual environment and scope. Check the linked source for implementation details, license, current support and reproduction steps.
语音助手 · 具身交互JARVIS / Reachy Mini
A local JARVIS assistant bringing XTTS speech synthesis and Reachy Mini camera and motion controls together.
↗
边缘设备 · 本地推理A local LLM on a Pi 5
A Raspberry Pi 5 on Ubuntu with an LLM-8850 M.2 accelerator running AXERA's quantized Qwen2.5-0.5B: 18.15–18.42 tok/s across three runs of one prompt, and a model that introduced itself as created by Anthropic.
↗
公开仓库pipesay
Linux live dictation for Wayland — speak, see text, auto-copy. MIT.
↗