Run 1
2:35 min:stime
17,251output tokens
10.0%thinking
111.41tok/s
479lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-26b-a4b-qat_bare_r1
Gemma 4 26B A4B: 110 tokens per second. Does a rules file change the page?
google/gemma-4-26b-a4b-qat
Same model, same prompt, same rig. 3 runs with nothing added, 3 with one Karpathy rules file.
google/gemma-4-26b-a4b-qat
Ranges overlap (counted effects)Effects count CSS/JS constructs, not visual quality.
Repeated ranges
21–36
KARPATHY · n=329–39
0:36–2:35 min:s
KARPATHY1:03–1:27 min:s
4,109–17,251
KARPATHY6,799–9,461
109.51–115.72 tok/s
KARPATHY106.55–109.16 tok/s
10.0%–20.8%
KARPATHY15.6%–29.9%
311–479
KARPATHY413–528
BARENO RULES, NO EXTRAS
1:45 min:stime
11,531output tokens
20.8%thinking
311lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
KARPATHYONE RULES FILE
1:15 min:stime
8,040output tokens
29.9%thinking
454lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
Every run. Every result.
BARE
NO RULES, NO EXTRAS · n=3
Run 2
1:45 min:stime
11,531output tokens
20.8%thinking
109.51tok/s
311lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-26b-a4b-qat_bare_r2
Run 3
0:36 min:stime
4,109output tokens
16.4%thinking
115.72tok/s
311lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-26b-a4b-qat_bare_r3
KARPATHY
ONE RULES FILE · n=3Run 1
35effects
1:15 min:stime
8,040output tokens
29.9%thinking
106.55tok/s
454lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-26b-a4b-qat_karpathy_r1
Run 2
39effects
1:03 min:stime
6,799output tokens
15.6%thinking
108.52tok/s
528lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-26b-a4b-qat_karpathy_r2
Run 3
29effects
1:27 min:stime
9,461output tokens
16.2%thinking
109.16tok/s
413lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-26b-a4b-qat_karpathy_r3
output = thinking + final code. Lines of code show the page size the model chose, not the cost of equivalent work.
How to reproduce
Settings recorded for these runs. Multiple values show differences between runs. Effort is requested, not necessarily received; it is not comparable across models. Hover over quantization, K/V cache or engine values for their sources.
google/gemma-4-26b-a4b-qat
| Model | Quantization | Context window | K/V cache | max_tokens | Engine | Harness | Effort requested |
|---|---|---|---|---|---|---|---|
| google/gemma-4-26b-a4b-qat | Q4_0 | 262,144 | K f16 / V f16 (LM Studio default) | 131,072 | [email protected] | pi 0.85.1 | medium |
Watch the experiment.
REPRODUCE THE EXPERIMENT
One prompt.
All the evidence.
Compare runs within the same model. Inspect the shared prompt, settings and raw measurements in the repository.