Rigorous, reproducible benchmarking — one variable at a time.
Every test uses the identical prompt on the same hardware rig. No moving targets — only one variable changes per run.
We isolate each factor — batch size, context length, quantization level — and change exactly one thing at a time so results are comparable.
The model is freshly reloaded into VRAM before every benchmark, eliminating cache effects and ensuring clean, consistent measurements.
Every prompt and every result file is published. You can verify the methodology, reproduce it yourself, or fork the work.
One machine, one GPU — real local inference.
24 GB VRAM — dedicated to local model inference
LM Studio runs locally on the same machine, serving models through an OpenAI-compatible API for consistent benchmarking.
A YouTube series dedicated to running open-source large language models on real local hardware — no cloud APIs, no managed services.
The goal is straightforward: measure how these models behave under controlled conditions on a single consumer GPU, publish every prompt and result file openly, and let the community verify everything for themselves.
Watch episodes, join the discussion, or explore the raw data.