Local models are good enough now that "can I run this on my laptop?" is a real question. But the answers people give are usually anecdotes: it "runs", it "feels fast". We wanted numbers, for every model size we could fit, and for the part that is easy to overlook: what the model does when you let it think.
So we took one 16 GB Apple M5 MacBook Pro and ran the Qwen3.5 family (0.8B, 2B, 4B, and a 4-bit 9B) through a text benchmark (GSM8K grade-school math) and a vision benchmark (AI2D science diagrams), each with thinking switched off and on. Every single request was instrumented: accuracy, time to first token, decode speed, tokens generated, GPU memory, system RAM, swap, CPU and GPU load and thermal state. Then we gave thinking mode 8x more tokens, and tried to rescue the runs where the model got stuck repeating itself.
This post walks through what we found, in the order we found it, and ends with what a MacBook like this can and cannot do.
The short answers
Bigger is better, and 4-bit is a bargain. GSM8K accuracy without thinking rises 54% → 77% → 85% → 88%. The 9B at 4-bit beats the 4B at full precision and is almost twice as fast.
Thinking is limited by its token budget, not by the model. With the standard caps it often hurts on math because answers never finish; with 8x the budget the 4B and 9B reach 95% on the same questions.
The MacBook is bandwidth-bound, not compute-bound. Speed tracks weight size almost perfectly (≈125 GB/s effective); a long context costs under 1 GB.
The practical ceiling is about 8 GB of weights. That is a 4B model in bf16, or a 9B-class model at 4-bit. Time, not memory, is what limits thinking-mode evaluation.
88%best GSM8K: 9B 4-bit, thinking off (26 tok/s)
85%best AI2D: 4B bf16, thinking on (14 tok/s)
≈125 GB/seffective memory bandwidth reached while decoding
0 throttlingCPU speed limit stayed at 100% in all 27 hours of generation
The machine is an Apple M5 (10-core CPU/GPU) with 16 GB of unified memory, on AC power, with other apps closed, running MLX 0.32 and mlx-vlm 0.7.12. All four models are the same multimodal Qwen3.5 checkpoints, so one set of weights does both tasks. The 0.8B, 2B and 4B run in bf16. The 9B is 19 GB in bf16 and simply does not fit, so we converted it to 4-bit (5.5 GB).
GSM8K uses the first 100 test questions, scored by exact match on the final number. AI2D uses the first 100 test items with options numbered 1-4 (25% is chance). Thinking off is greedy decoding; thinking on samples at temperature 0.6, top-p 0.95. Token caps are 512 / 32 (off) and 3,072 / 2,048 (on) for GSM8K / AI2D, and a truncated answer counts as wrong. With n=100 per cell, results are accurate to about ±10 points, so we lean on large gaps and paired tests.
A caveat on AI2D with thinking off. That mode has a 32-token budget. The 4B likes to explain before answering and gets cut off 49% of the time, so its 45% reflects the format limit, not weak vision. The 9B answers directly (82%); the 0.8B and 2B sit near chance.
2. Does a bigger model help?
Yes, almost everywhere, with one surprise: at full thinking caps the 9B does worse than the 4B.
Accuracy by model and mode; error bars are 95% Wilson intervals (n=100). The dashed line is chance level on AI2D (25%).
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
Without thinking, GSM8K accuracy climbs steadily with size: 54%, 77%, 85%, 88%. Paired sign tests (appendix B) confirm the big steps: 0.8B to 2B is significant on GSM8K in both modes and on AI2D with thinking; 2B to 4B is significant with thinking on. From 4B to 9B-4bit, GSM8K without thinking is within noise.
The surprise is thinking mode. With thinking on, the 4B is significantly better than the larger 9B-4bit on both tasks (GSM8K 84% vs 70%, AI2D 85% vs 76%). Not because it reasons better, as the next chapter shows, but because the 9B runs out of tokens more often (45% vs 28% on GSM8K).
Takeaway. Model size buys accuracy, but only if the answer actually finishes. Whether the answer finishes depends on the token budget.
3. Thinking mode: powerful, until it runs out of tokens
Thinking helps vision, looks harmful on math, and the difference is almost entirely about truncation.
For the smaller models on AI2D, thinking is a clear win: 0.8B 41% to 48%, 2B 46% to 73%, 4B 45% to 85%. On GSM8K it goes the other way: the 0.8B falls from 54% to 22% and the 9B from 88% to 70%. Look at how often the models hit the cap:
Share of requests that hit the token cap. Up to 94% of the 0.8B's thinking runs on GSM8K never finish.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
Thinking is expensive in tokens, from roughly 6x more on GSM8K to several hundred times more on AI2D, where the non-thinking answer is just a few tokens:
Mean generated tokens (log scale).
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
And the tokens are not evenly spent. Wrong answers are long: in thinking mode they use about 1.9x the tokens of correct ones. A model that is going to get it right usually does so in about 1362 tokens; one that is struggling keeps going until it hits the cap.
Mean tokens for correct vs wrong answers (thinking on, 1x caps).
correct answerswrong answers
What happens with 8x the budget?
To separate capability from budget we re-ran the first 20 questions of each task with caps 8x higher (24,576 tokens for GSM8K). If the cap were the whole story, accuracy would jump:
GSM8K, thinking on, same 20 questions at 1x and 8x caps.
1x cap (3072 tok)8x cap (24,576 tok)
AI2D, thinking on, same 20 questions at 1x and 8x caps.
1x cap (2048 tok)8x cap (16,384 tok)
It does, for every model: the 4B goes from 70% to 95% on GSM8K and the 9B from 65% to 95%, now with none of its samples truncated. With a big enough budget the 9B catches up with the 4B, which resolves the surprise in chapter 2. (n=20, so each number is good to about ±20 points; the direction is the same for all four models.) The price is time: the 4B's average answer on GSM8K takes ~131 s at 1x and ~410 s at 8x.
Share of requests hitting the cap, 1x vs 8x.
GSM8K, 1xGSM8K, 8xAI2D, 1xAI2D, 8x
Takeaway. "Thinking hurts math" was a budget artefact. Give it room and thinking becomes the best mode; but budget is wall-clock time on a laptop, so it is a real trade-off.
4. The ones that never finish: repetition loops, and a rescue attempt
More tokens fix most truncations, but not the loops. Those just loop for longer.
Not all truncated answers are alike. We classified each truncated thinking trace by whether its last 3,000 characters compress to under 15% of their size (a sure sign the model is repeating itself). At the standard caps, most truncations are genuinely long reasoning that would finish given more room:
Truncated thinking-on outputs at 1x caps (counts, GSM8K + AI2D).
repetition looplong but not looping
At 8x the picture flips: of the 39 truncated outputs from the 0.8B, 2B and 4B, 34 end in a loop. The extra budget rescued most of the answers that were merely slow, and what remains is mostly the model stuck in a cycle. The 9B almost never loops.
Truncated thinking-on outputs at 8x caps: the remainder is loops.
repetition looplong but not looping
Can sampling settings pull a model out of a loop?
Qwen recommends top-k 20 and a presence penalty for thinking mode. Our first runs used neither, so we re-ran the truncated-and-wrong samples (up to 10 per task) with top-k 20 and presence penalty 1.5:
Share of re-run samples that became correct.
1x sets: rescued % of re-run8x sets: rescued % of re-run
It rescued roughly 20-35% of the 1x cases and 50-70% of the 8x cases. This is encouraging but weak evidence: there is no control arm (a re-run with the original settings), and the sets are tiny (2-20 samples per cell), so part of the rescue could be plain re-sampling luck. Details are in appendix D.
Takeaway. Budget fixes slowness; sampling settings may help with loops; bigger models loop less. Look at why answers are truncated before deciding the fix.
5. How fast is it, really?
Decode speed is set by one number: how many gigabytes of weights the chip must read per token.
Decode throughput (thinking on). It depends on the model, not the task.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
Generating a token means streaming essentially all the weights through the GPU once, so speed halves when weights double: 71 tok/s for the 0.8B, 30 for the 2B, 14 for the 4B. Multiply decode speed by weight size and you get the effective memory bandwidth, which lands between 112 and 143 GB/s for every model, most of what the chip can deliver. That is also why the 4-bit 9B (5.5 GB) generates at 26 tok/s, nearly twice the 4B in bf16 (8.5 GB).
Implied memory bandwidth = decode tok/s × weight size in GB, against Apple's quoted peak for the M5.
decode tok/s × weight GB
Prefill (reading the prompt) is compute-bound instead and runs at 400-2,600 tokens/s depending on the model; vision prompts carry about 200 extra image tokens, so they cost more to read.
Prefill throughput.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
Mean end-to-end latency per request (log scale): sub-second for direct answers, minutes for thinking on the larger models.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
6. What does a correct answer cost?
Direct answers are 6-20x cheaper than thinking on math, and the small models are not the cheap ones you would expect.
Wall-clock seconds per correct answer (total time / number correct), log scale.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
Dividing total time by the number of correct answers puts accuracy and speed on one axis. On GSM8K without thinking the 0.8B is cheapest per correct answer (about 9 s) but only because it is fast; the larger models waste fewer tokens (381 tokens per correct answer for the 9B against 662 for the 0.8B). With thinking on, every model pays about 120-190 s per correct GSM8K answer.
Generated tokens per correct answer, log scale.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
Total generation time (26.7 h across 1,760 inferences): the 4B was the slowest model to benchmark at 10.1 h.
main benchmark (400 inferences)8x long-thinking (40 inferences)
7. How much of the 16 GB do we use?
The 0.8B and 2B are comfortable. The 4B in bf16 is right at the edge, and the 4-bit 9B has more headroom.
Peak MLX memory, peak system RAM and peak swap per model, against the 16 GB limit.
MLX peak GPU memorySystem RAM used (max)Swap used (max)
macOS and background services hold about 6.5 GB even with apps closed, leaving ≈9.5 GB for a model. The 4B in bf16 peaks at 9.4 GB of MLX memory and macOS used up to ~2.4 GB of swap, yet decode speed stayed flat at 14 tok/s, so the swap held idle pages, not the model. (The 9B runs showed ~2.3 GB of swap with only 6.5 GB of MLX memory: system-wide pressure.) The 0.8B and 2B never touched swap.
GPU utilisation was 97-99% for every thinking-on run and CPU utilisation only 6-12% (the CPU just feeds the GPU), so generation is fully GPU-bound. The thermal speed limit never dropped below 100%: the M5 did not throttle in 27 hours of generation, including single runs of about 6 hours.
8. So what can this MacBook do?
What fits, how long things take, and where the walls are.
≈9.5 GBusable for a model: macOS and background services keep ≈6.5 GB in use even with apps closed
One of the pleasant surprises: a 25,000-token thinking trace adds under 1 GB. Qwen3.5 mostly uses gated-delta (linear) attention layers, whose state does not grow with length, so memory is decided by weights, not context.
Peak MLX memory above the weights for the longest request in each set.
longest 1x run (~3k tokens)longest 8x run (16-25k tokens)
What fits
Rows marked "(est.)" are extrapolations: weights = parameters × bytes per parameter (bf16 2, 8-bit ≈1.06, 4-bit ≈0.6), plus ~1 GB working memory against the usable ≈9.5 GB; decode speed = measured bandwidth ÷ weight size (this rule predicts the four measured models within about 15%).
Model size
Precision
Weights GB
On this 16 GB Mac
Decode tok/s (thinking on)
0.8B
bf16
1.6
fits comfortably
71 (measured)
2B
bf16
4.1
fits comfortably
30 (measured)
4B
bf16
8.5
tight (swap likely)
14 (measured)
9B
4-bit
5.5
fits comfortably
26 (measured)
9B
bf16
18.9 (est.)
does not fit
–
8B-class
8-bit
8.5 (est.)
tight (swap likely)
≈15
14B-class
4-bit
8.4 (est.)
tight (swap likely)
≈15
14B-class
8-bit
14.8 (est.)
does not fit
–
32B-class
4-bit
19.2 (est.)
does not fit
–
What an experiment costs in time
Time, not memory, is the practical limit for thinking-mode evaluation. The full GSM8K test split (1,319 questions) with the 4B thinking on would take about 48 hours at 1x caps and far longer at 8x:
Model
GSM8K (1,319) thinking off
GSM8K thinking on
AI2D (3,088) thinking off
AI2D thinking on
0.8B (bf16)
1.8 h
15.4 h
0.1 h
16.0 h
2B (bf16)
4.1 h
28.0 h
0.3 h
31.5 h
4B (bf16)
8.6 h
48.0 h
1.4 h
60.5 h
9B (4-bit)
4.7 h
32.1 h
0.6 h
40.7 h
The ceiling is physical: at thinking-on speeds the machine generates about 6.1 M tokens a day with the 0.8B, 2.6 M with the 2B, 1.2 M with the 4B and 2.2 M with the 9B (4-bit).
Datasets: cost follows answer length
Task
Prompt tokens
Answer tokens (thinking off)
Answer tokens (thinking on, 1x cap)
What dominates the cost
GSM8K (text math)
92
343
2345
generated reasoning tokens (decode)
AI2D (diagram MCQ, image ≤896 px)
312
7
1161
image prefill when off; reasoning tokens when on
Cost scales with generated tokens, not prompt size. Short-answer multiple choice with thinking off (a few tokens) is cheap for every model; chain-of-thought math is 100-1,000x more expensive per question. When choosing an evaluation set for this machine, estimate the answer length first.
9. Conclusions
Run the largest model whose weights fit in about 8 GB. On this Mac that is a 4B in bf16 or a 9B-class model at 4-bit; at the same memory, 4-bit buys a bigger and faster model that is at least as accurate without thinking.
Treat the token budget as a first-class setting. It decides whether thinking helps or hurts. Check how many answers hit the cap before comparing models or modes.
Speed is predictable. Tokens per second ≈ 125 ÷ weight size in GB. Use it to plan before running.
Memory is not the long-context problem; time is. A 25k-token trace costs under 1 GB but takes ~30 minutes on the 4B.
The machine is a reliable instrument. 27 hours of generation, no throttling, no crashes, including multi-hour single runs.
Practical recommendations
Interactive use, fast answers: 9B at 4-bit, thinking off (26 tok/s, 88% GSM8K).
Hard reasoning or diagrams, time to spare: 4B in bf16 with thinking on and a generous cap (84% at 1x, 95% at 8x on the sampled questions).
Prototyping and many quick evaluations: 0.8B or 2B; they leave several GB free and run at 30-70 tok/s.
Avoid: thinking mode with small caps on the 0.8B (94% truncation) and anything above ~8 GB of weights.
What we did not measure
Batched or concurrent generation (decode is bandwidth-bound, so batching should raise total throughput substantially, but we ran batch size 1), contexts beyond ~25k tokens, images larger than 896 px (they were downsized), power draw in watts and battery operation (it needs sudo powermetrics), and sustained load beyond single ~6-hour runs. All cells are n=100 or n=20 on one machine, so small differences are not meaningful.
What is next
Inference is half the story. The sibling project, Mac LLM/VLM Post-training Testing, asks the same capability question for training: SFT, GRPO, RLOO and GSPO, with LoRA and full fine-tuning, on the same machine. Early finding: the memory wall comes much sooner when you train. Read that post next.
Appendix
A. Full results table
Model
Task
Think
Accuracy (95% CI)
Acc. if finished
Capped
Gen tok
Decode tok/s
Prefill tok/s
TTFT s
Latency s
p95 s
s / correct
Peak GB
CPU %
GPU %
RAM GB
Swap GB
Therm. limit %
0.8B (bf16)
GSM8K (text)
off
54%(44–63)
66%
21%
357
73.5
1943
0.05
4.9
7.0
9.1
1.8
9
97
8.7
0.0
100
0.8B (bf16)
GSM8K (text)
on
22%(15–31)
100%
94%
2960
70.6
1952
0.05
42.0
44.3
190.8
1.8
10
99
8.9
0.0
100
0.8B (bf16)
AI2D (vision)
off
41%(32–51)
43%
5%
4
140.5
2658
0.12
0.2
0.3
0.4
2.4
8
88
8.2
0.0
100
0.8B (bf16)
AI2D (vision)
on
48%(38–58)
68%
43%
1308
71.0
2508
0.13
18.6
29.7
38.8
2.5
10
99
8.6
0.0
100
2B (bf16)
GSM8K (text)
off
77%(68–84)
91%
18%
349
31.5
929
0.10
11.2
16.3
14.6
4.3
9
98
10.0
0.0
100
2B (bf16)
GSM8K (text)
on
65%(55–74)
100%
49%
2320
30.4
934
0.10
76.4
102.3
117.5
4.3
9
99
10.1
0.0
100
2B (bf16)
AI2D (vision)
off
46%(37–56)
46%
0%
3
55.8
1166
0.26
0.3
0.5
0.7
5.1
6
95
9.0
0.0
100
2B (bf16)
AI2D (vision)
on
73%(64–81)
95%
34%
1132
31.1
1158
0.26
36.7
66.4
50.2
5.1
8
99
9.3
0.0
100
4B (bf16)
GSM8K (text)
off
85%(77–91)
93%
10%
328
14.2
390
0.24
23.4
36.2
27.5
8.7
10
98
12.6
2.4
100
4B (bf16)
GSM8K (text)
on
84%(76–90)
99%
28%
1842
14.1
384
0.23
131.0
219.0
156.0
8.7
8
99
12.3
1.7
100
4B (bf16)
AI2D (vision)
off
45%(36–55)
88%
49%
17
20.4
623
0.48
1.6
3.0
3.7
9.4
10
98
12.8
2.0
100
4B (bf16)
AI2D (vision)
on
85%(77–91)
99%
20%
992
14.2
615
0.48
70.5
145.3
83.0
9.4
7
99
12.8
2.1
100
9B (4-bit)
GSM8K (text)
off
88%(80–93)
100%
12%
336
26.5
402
0.23
12.9
19.6
14.7
6.0
7
99
11.3
2.3
100
9B (4-bit)
GSM8K (text)
on
70%(60–78)
100%
45%
2259
25.9
394
0.23
87.6
119.5
125.2
6.0
7
99
11.4
2.1
100
9B (4-bit)
AI2D (vision)
off
82%(73–88)
82%
0%
2
49.6
485
0.63
0.7
1.2
0.9
6.5
6
97
11.7
1.8
100
9B (4-bit)
AI2D (vision)
on
76%(67–83)
100%
36%
1214
26.0
487
0.62
47.5
80.1
62.5
6.5
7
99
11.7
1.8
100
B. Paired sign tests between neighbouring model sizes
Comparison
Task
Think
Only larger right
Only smaller right
Sign-test p
p < 0.05
0.8B (bf16) → 2B (bf16)
GSM8K (text)
off
28
5
0.000
yes
0.8B (bf16) → 2B (bf16)
GSM8K (text)
on
45
2
0.000
yes
0.8B (bf16) → 2B (bf16)
AI2D (vision)
off
23
18
0.533
no
0.8B (bf16) → 2B (bf16)
AI2D (vision)
on
31
6
0.000
yes
2B (bf16) → 4B (bf16)
GSM8K (text)
off
11
3
0.057
no
2B (bf16) → 4B (bf16)
GSM8K (text)
on
23
4
0.000
yes
2B (bf16) → 4B (bf16)
AI2D (vision)
off
20
21
1.000
no
2B (bf16) → 4B (bf16)
AI2D (vision)
on
14
2
0.004
yes
4B (bf16) → 9B (4-bit)
GSM8K (text)
off
7
4
0.549
no
4B (bf16) → 9B (4-bit)
GSM8K (text)
on
4
18
0.004
yes
4B (bf16) → 9B (4-bit)
AI2D (vision)
off
39
2
0.000
yes
4B (bf16) → 9B (4-bit)
AI2D (vision)
on
2
11
0.022
yes
C. Long-thinking runs (8x token caps)
Model
Task
Accuracy (95% CI, n=20)
Capped
Gen tok
Decode tok/s
Latency s
Max tok
Peak GB
Swap GB
0.8B (bf16)
GSM8K (text)
35%(18–57)
80%
20919
67.4
314
24576
2.1
1.8
0.8B (bf16)
AI2D (vision)
60%(39–78)
35%
6920
66.8
106
16384
2.4
1.7
2B (bf16)
GSM8K (text)
80%(58–92)
25%
9304
30.1
313
24576
4.5
1.7
2B (bf16)
AI2D (vision)
70%(48–85)
30%
6486
31.0
213
16384
4.9
1.7
4B (bf16)
GSM8K (text)
95%(76–99)
15%
5618
14.1
410
24576
9.4
2.8
4B (bf16)
AI2D (vision)
85%(64–95)
10%
3933
14.1
283
16384
9.2
2.0
9B (4-bit)
GSM8K (text)
95%(76–99)
0%
4386
25.7
172
10368
6.1
1.7
9B (4-bit)
AI2D (vision)
85%(64–95)
20%
4369
25.6
176
16384
6.4
1.7
D. Mitigation runs (top-k 20 + presence penalty 1.5)
Set
Model
Task
Re-run
Rescued
Still capped
Orig. acc
Acc after
1x
0.8B (bf16)
AI2D (vision)
10
2
80%
48%
50%
1x
0.8B (bf16)
GSM8K (text)
10
4
60%
22%
26%
1x
2B (bf16)
AI2D (vision)
10
3
70%
73%
76%
1x
2B (bf16)
GSM8K (text)
10
1
90%
65%
66%
1x
4B (bf16)
AI2D (vision)
10
3
70%
85%
88%
1x
4B (bf16)
GSM8K (text)
10
3
90%
84%
87%
1x
9B (4-bit)
AI2D (vision)
10
1
90%
76%
77%
1x
9B (4-bit)
GSM8K (text)
10
6
50%
70%
76%
8x
0.8B (bf16)
AI2D (vision)
6
4
33%
60%
80%
8x
0.8B (bf16)
GSM8K (text)
10
4
10%
35%
55%
8x
2B (bf16)
AI2D (vision)
5
3
20%
70%
85%
8x
2B (bf16)
GSM8K (text)
2
2
0%
80%
90%
8x
4B (bf16)
AI2D (vision)
2
1
50%
85%
90%
8x
4B (bf16)
GSM8K (text)
0
0
–
95%
95%
8x
9B (4-bit)
AI2D (vision)
3
2
33%
85%
95%
8x
9B (4-bit)
GSM8K (text)
0
0
–
95%
95%
Caveat. No control arm, tiny sets: indicative only.
E. Glossary
Term
Meaning
TTFT
Time To First Token: delay from sending the prompt until the first generated token appears (includes prefill).
Prefill tok/s
Prompt tokens processed per second before generation starts (compute-bound).
Decode tok/s
Generated tokens per second after the first token (memory-bandwidth-bound).
Latency / p95
End-to-end time per request (mean, and 95th percentile).
Capped / truncated
The model hit the max-token limit before finishing; counted as wrong unless a correct final answer already appeared.
Acc. if finished
Accuracy over only the samples that finished within the cap (not truncated).
Peak GB
Peak MLX (GPU) memory allocation during a request, from mx.get_peak_memory().
RAM / Swap
System RAM used and swap used (psutil), sampled every 0.5 s; maximum over the run.
CPU % / GPU %
Average system CPU utilisation and GPU device utilisation (ioreg 'Device Utilization %') during generation.
Therm. limit
macOS CPU speed limit from pmset; 100% = no thermal throttling.
95% CI
Wilson score interval for a proportion; with n=100 it is about ±10 points, with n=20 about ±20.
Sign test
Paired exact binomial test on questions where exactly one of two models is correct.
F. Method notes and reproduction
Latency, tokens and memory columns are averages over the samples of that cell; Peak GB, RAM and Swap are maxima; Therm. limit is the minimum.
The 9B was converted to 4-bit (group size 64) because bf16 (19 GB) does not fit in 16 GB, so it is not an apples-to-apples precision comparison with the other three.
Sampling at T=0.6 makes thinking-on numbers vary run to run.
Initial draft prepared using Claude, then reviewed, edited, and finalized by Azmine Toushik Wasi.