Benchmarks & Testing
Measured numbers with their date, quantisation and setup beside them, the poor results included.
A benchmark number is only useful when you know how it was made. The same model can score differently with another quantisation, another runtime or another prompt, and most published tables leave these details out.
This matters for anyone who has to choose. A leaderboard can say a model is strong while it fails on the work you actually have, and a vendor's own test is rarely the test you need. Speed has the same problem: the time a model takes to read a long input (prefill) and the time it takes to write (decode) answer different questions, yet they are often reported as one figure.
I measure in my own studio, on my own hardware. Published rows use llama.cpp only, with the date, quantisation and setup beside each number, and prefill and decode are reported separately. The suites include LiveCodeBench (30 coding tasks), real software issues from SWE-bench Verified, an extraction golden set and quality gates for video.
Poor results are published too. The best local coder solved 22 of 30 LiveCodeBench tasks, and on 150 SWE-bench Verified issues a local model resolved 16. One model, Nemotron 3.5 Lightning, was removed after it solved 15 of 30 tasks and, on one task, wrote ten times as much as a faster model to reach an answer.
The practice follows one rule: a local model can fail work, but it can never approve it. A plain test decides.
Evidence
- Qwen3.8-27B decodes at 38.9 tokens per second with a Q4_K_M build and an MTP Q4_0 sidecar (llama.cpp d454a20, local workstation, three prompts, temperature 0). The older Q4_K_M build reached 25.9 and IQ3_S reached 26.2.
- On LiveCodeBench v6 (30 tasks, llama.cpp, local workstation) the three Qwen3.8-27B variants solved 17 (IQ3_S), 16 (MTP) and 15 (older Q4_K_M) tasks, a tie within noise.
- On a golden set of 24 documents and 120 fields, qwen3.8-flash-next-coder IQ1_M (29.6 GB) extracted 120 of 120 fields, with a median of 9.84 s and a 95th percentile of 11.27 s per document (llama.cpp 0.6.0, local workstation).
- On the SWE-bench Verified sympy subset, Qwen3-Coder-Next Q4_K_M under llama.cpp on local hardware resolved 24 of 63 tasks (native replay harness).
- A local model may fail work but never approve it.
- A CUDA decode smoke test of Qwen2.5-0.5B-Instruct Q8_0 reached 231.27 tokens per second over 95 tokens (llama.cpp b11429, CUDA 12.4, own hardware).
- Runs an extraction golden set of 24 documents and 120 fields.
- Runs LiveCodeBench v6 on 30 tasks.
- Runs a SWE-bench Verified sympy subset through a native harness that validates 63 of 67 gold patches.
- Reports prefill and decode speed separately for each build and quantisation.
- Checks generated videos against rules and human verdicts.
- Runs open models locally with llama.cpp, with cloud models beside them; removed Nemotron 3.5 Lightning (30B) after it solved 15 of 30 LiveCodeBench tasks against 18 for gemma-4-26b-a4b, and used 12,650 tokens against 1,236 on one 14k-token task; fine-tunes Kev checkpoints on Jared Palmer's open-source recipe; a shell check decides pass or fail; a local model may fail work but never approve it.
- Across 150 SWE-bench Verified tasks graded by the official harness on Modal, Qwen3-Coder-Next under llama.cpp resolved 16. Codex resolved 58 that the local model missed. Neither resolved the other 76.