Run 1
7:00 min:stime
22,049output tokens
5.8%thinking
52.5tok/s
1,084lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_bare_r1
Qwen 3.8 27B with Karpathy skills: what you gain, what you lose
qwen/qwen3.8-27b
On Qwen3.8 27B, one Karpathy rules file: no clear difference in time or tokens, 22% fewer visual effects, fewer in each of the 5 runs.
qwen/qwen3.8-27b
Karpathy below every bare run (counted effects) · −21.6% mean changeEffects count CSS/JS constructs, not visual quality.
Repeated ranges
129–155
KARPATHY · n=5101–121
5:38–7:00 min:s
KARPATHY4:08–7:59 min:s
18,858–22,049
KARPATHY14,333–24,268
52.50–56.30 tok/s
KARPATHY50.69–57.84 tok/s
1.6%–5.8%
KARPATHY1.8%–11.2%
978–1,122
KARPATHY902–1,122
BARENO RULES, NO EXTRAS
5:49 min:stime
19,321output tokens
1.7%thinking
1,059lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
KARPATHYONE RULES FILE
5:06 min:stime
15,659output tokens
2.5%thinking
902lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
Every run. Every result.
BARE
NO RULES, NO EXTRAS · n=5
Run 2
5:49 min:stime
19,321output tokens
1.7%thinking
55.33tok/s
1,059lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_bare_r2
Run 3
5:38 min:stime
19,052output tokens
3.2%thinking
56.3tok/s
1,033lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_bare_r3
Run 4
5:38 min:stime
18,858output tokens
1.6%thinking
55.83tok/s
1,122lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_bare_r4
Run 5
6:04 min:stime
20,397output tokens
2.3%thinking
56.03tok/s
978lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_bare_r5
KARPATHY
ONE RULES FILE · n=5Run 1
101effects
4:08 min:stime
14,333output tokens
1.8%thinking
57.84tok/s
920lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_karpathy_r1
Run 2
121effects
7:59 min:stime
24,268output tokens
11.2%thinking
50.69tok/s
1,122lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_karpathy_r2
Run 3
114effects
5:06 min:stime
15,659output tokens
2.5%thinking
51.26tok/s
902lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_karpathy_r3
Run 4
112effects
7:08 min:stime
23,174output tokens
7.2%thinking
54.13tok/s
934lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_karpathy_r4
Run 5
110effects
4:40 min:stime
15,810output tokens
2.9%thinking
56.57tok/s
973lines of code
Effort requested: medium; received: unknown. Effort is not comparable across models.
qwen3.8-27b_karpathy_r5
output = thinking + final code. Lines of code show the page size the model chose, not the cost of equivalent work.
How to reproduce
Settings recorded for these runs. Multiple values show differences between runs. Effort is requested, not necessarily received; it is not comparable across models. Hover over quantization, K/V cache or engine values for their sources.
qwen/qwen3.8-27b
| Model | Quantization | Context window | K/V cache | max_tokens | Engine | Harness | Effort requested |
|---|---|---|---|---|---|---|---|
| qwen/qwen3.8-27b | Q4_K_M | 262,144 | K q4_0 / V q4_0 | 131,072 | [email protected] | pi 0.85.1 | medium |
Watch the experiment.
REPRODUCE THE EXPERIMENT
One prompt.
All the evidence.
Compare runs within the same model. Inspect the shared prompt, settings and raw measurements in the repository.