Run 1
2:44 min:stime
6,334output tokens
14.8%thinking
38.59tok/s
376lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
muse-glimmer_bare_r1
Muse Glimmer builds a web page in 2.5 minutes. What changes with Karpathy rules?
meta/muse-glimmer
Same model, same prompt, same rig. 3 runs with nothing added, 3 with one Karpathy rules file.
meta/muse-glimmer
Ranges overlap (counted effects)Effects count CSS/JS constructs, not visual quality.
Repeated ranges
34–42
KARPATHY · n=338–41
2:29–2:44 min:s
KARPATHY2:19–2:30 min:s
5,733–6,334
KARPATHY5,342–5,778
38.48–38.62 tok/s
KARPATHY38.30–38.64 tok/s
13.5%–20.6%
KARPATHY9.2%–16.4%
313–376
KARPATHY268–307
BARENO RULES, NO EXTRAS
2:29 min:stime
5,733output tokens
20.6%thinking
313lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
KARPATHYONE RULES FILE
2:20 min:stime
5,382output tokens
9.2%thinking
307lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
Every run. Every result.
BARE
NO RULES, NO EXTRAS · n=3
Run 2
2:29 min:stime
5,752output tokens
13.5%thinking
38.62tok/s
330lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
muse-glimmer_bare_r2
Run 3
2:29 min:stime
5,733output tokens
20.6%thinking
38.48tok/s
313lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
muse-glimmer_bare_r3
KARPATHY
ONE RULES FILE · n=3Run 1
41effects
2:20 min:stime
5,382output tokens
9.2%thinking
38.41tok/s
307lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
muse-glimmer_karpathy_r1
Run 2
40effects
2:19 min:stime
5,342output tokens
16.4%thinking
38.3tok/s
268lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
muse-glimmer_karpathy_r2
Run 3
38effects
2:30 min:stime
5,778output tokens
11.7%thinking
38.64tok/s
272lines of code
Effort requested: medium; received: medium · source: metrics.json. Effort is not comparable across models.
muse-glimmer_karpathy_r3
output = thinking + final code. Lines of code show the page size the model chose, not the cost of equivalent work.
How to reproduce
Settings recorded for these runs. Multiple values show differences between runs. Effort is requested, not necessarily received; it is not comparable across models. Hover over quantization, K/V cache or engine values for their sources.
meta/muse-glimmer
| Model | Quantization | Context window | K/V cache | max_tokens | Engine | Harness | Effort requested |
|---|---|---|---|---|---|---|---|
| meta/muse-glimmer | Q4_K_M | 131,072 | K f16 / V f16 (LM Studio default) | 125,550 124,962 | [email protected] | pi 0.85.1 | medium |
Watch the experiment.
REPRODUCE THE EXPERIMENT
One prompt.
All the evidence.
Compare runs within the same model. Inspect the shared prompt, settings and raw measurements in the repository.