Run 1
6:26 min:stime
11,279output tokens
9.5%thinking
29.24tok/s
403lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-31b-qat_bare_r1
Gemma 4 31B needs 7 minutes for one web page. Do rules speed it up?
google/gemma-4-31b-qat
Same model, same prompt, same rig. 3 runs with nothing added, 3 with one Karpathy rules file.
google/gemma-4-31b-qat
Ranges overlap (counted effects)Effects count CSS/JS constructs, not visual quality.
Repeated ranges
29–39
KARPATHY · n=331–44
6:26–22:25 min:s
KARPATHY5:05–21:35 min:s
11,279–39,349
KARPATHY8,910–37,583
28.96–29.27 tok/s
KARPATHY28.52–29.25 tok/s
2.8%–9.5%
KARPATHY2.9%–7.8%
385–418
KARPATHY369–447
BARENO RULES, NO EXTRAS
6:38 min:stime
11,515output tokens
8.7%thinking
385lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
KARPATHYONE RULES FILE
7:30 min:stime
12,835output tokens
3.4%thinking
447lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
Every run. Every result.
BARE
NO RULES, NO EXTRAS · n=3
Run 2
22:25 min:stime
39,349output tokens
2.8%thinking
29.27tok/s
418lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-31b-qat_bare_r2
Run 3
6:38 min:stime
11,515output tokens
8.7%thinking
28.96tok/s
385lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-31b-qat_bare_r3
KARPATHY
ONE RULES FILE · n=3Run 1
31effects
5:05 min:stime
8,910output tokens
7.8%thinking
29.25tok/s
369lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-31b-qat_karpathy_r1
Run 2
44effects
7:30 min:stime
12,835output tokens
3.4%thinking
28.52tok/s
447lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-31b-qat_karpathy_r2
Run 3
32effects
21:35 min:stime
37,583output tokens
2.9%thinking
29.03tok/s
431lines of code
Effort requested: medium; received: on · differs from requested effort · source: metrics.json. Effort is not comparable across models.
gemma-4-31b-qat_karpathy_r3
output = thinking + final code. Lines of code show the page size the model chose, not the cost of equivalent work.
How to reproduce
Settings recorded for these runs. Multiple values show differences between runs. Effort is requested, not necessarily received; it is not comparable across models. Hover over quantization, K/V cache or engine values for their sources.
google/gemma-4-31b-qat
| Model | Quantization | Context window | K/V cache | max_tokens | Engine | Harness | Effort requested |
|---|---|---|---|---|---|---|---|
| google/gemma-4-31b-qat | Q4_0 | 65,536 | K q8_0 / V q8_0 | 32,768 | [email protected] | pi 0.85.1 | medium |
Watch the experiment.
REPRODUCE THE EXPERIMENT
One prompt.
All the evidence.
Compare runs within the same model. Inspect the shared prompt, settings and raw measurements in the repository.