How far can a 16 GB MacBook go? Benchmarking Qwen3.5 from 0.8B to 9B



Mac LLM/VLM Inference Testing

November 18, 2026 · Apple M5 MacBook Pro, 16 GB · MLX · 1,760 measured inferences · about 15 min read

The question

Local models are good enough now that "can I run this on my laptop?" is a real question. But the answers people give are usually anecdotes: it "runs", it "feels fast". We wanted numbers, for every model size we could fit, and for the part that is easy to overlook: what the model does when you let it think.

So we took one 16 GB Apple M5 MacBook Pro and ran the Qwen3.5 family (0.8B, 2B, 4B, and a 4-bit 9B) through a text benchmark (GSM8K grade-school math) and a vision benchmark (AI2D science diagrams), each with thinking switched off and on. Every single request was instrumented: accuracy, time to first token, decode speed, tokens generated, GPU memory, system RAM, swap, CPU and GPU load and thermal state. Then we gave thinking mode 8x more tokens, and tried to rescue the runs where the model got stuck repeating itself.

This post walks through what we found, in the order we found it, and ends with what a MacBook like this can and cannot do.

The short answers

  • Bigger is better, and 4-bit is a bargain. GSM8K accuracy without thinking rises 54% → 77% → 85% → 88%. The 9B at 4-bit beats the 4B at full precision and is almost twice as fast.
  • Thinking is limited by its token budget, not by the model. With the standard caps it often hurts on math because answers never finish; with 8x the budget the 4B and 9B reach 95% on the same questions.
  • The MacBook is bandwidth-bound, not compute-bound. Speed tracks weight size almost perfectly (≈125 GB/s effective); a long context costs under 1 GB.
  • The practical ceiling is about 8 GB of weights. That is a 4B model in bf16, or a 9B-class model at 4-bit. Time, not memory, is what limits thinking-mode evaluation.
88%best GSM8K: 9B 4-bit, thinking off (26 tok/s)
85%best AI2D: 4B bf16, thinking on (14 tok/s)
≈125 GB/seffective memory bandwidth reached while decoding
0 throttlingCPU speed limit stayed at 100% in all 27 hours of generation

Jump to: setup · accuracy · thinking and tokens · loops · speed · cost · memory · what this Mac can do · conclusions · appendix

1. How we measured

The machine is an Apple M5 (10-core CPU/GPU) with 16 GB of unified memory, on AC power, with other apps closed, running MLX 0.32 and mlx-vlm 0.7.12. All four models are the same multimodal Qwen3.5 checkpoints, so one set of weights does both tasks. The 0.8B, 2B and 4B run in bf16. The 9B is 19 GB in bf16 and simply does not fit, so we converted it to 4-bit (5.5 GB).

GSM8K uses the first 100 test questions, scored by exact match on the final number. AI2D uses the first 100 test items with options numbered 1-4 (25% is chance). Thinking off is greedy decoding; thinking on samples at temperature 0.6, top-p 0.95. Token caps are 512 / 32 (off) and 3,072 / 2,048 (on) for GSM8K / AI2D, and a truncated answer counts as wrong. With n=100 per cell, results are accurate to about ±10 points, so we lean on large gaps and paired tests.

A caveat on AI2D with thinking off. That mode has a 32-token budget. The 4B likes to explain before answering and gets cut off 49% of the time, so its 45% reflects the format limit, not weak vision. The 9B answers directly (82%); the 0.8B and 2B sit near chance.

2. Does a bigger model help?

Yes, almost everywhere, with one surprise: at full thinking caps the 9B does worse than the 4B.

Accuracy by model and mode; error bars are 95% Wilson intervals (n=100). The dashed line is chance level on AI2D (25%).
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
0%20%40%60%80%100%GSM8Kthinking off0.8B (bf16) · GSM8K thinking off: 54%54%2B (bf16) · GSM8K thinking off: 77%77%4B (bf16) · GSM8K thinking off: 85%85%9B (4-bit) · GSM8K thinking off: 88%88%GSM8Kthinking on0.8B (bf16) · GSM8K thinking on: 22%22%2B (bf16) · GSM8K thinking on: 65%65%4B (bf16) · GSM8K thinking on: 84%84%9B (4-bit) · GSM8K thinking on: 70%70%AI2Dthinking off0.8B (bf16) · AI2D thinking off: 41%41%2B (bf16) · AI2D thinking off: 46%46%4B (bf16) · AI2D thinking off: 45%45%9B (4-bit) · AI2D thinking off: 82%82%AI2Dthinking on0.8B (bf16) · AI2D thinking on: 48%48%2B (bf16) · AI2D thinking on: 73%73%4B (bf16) · AI2D thinking on: 85%85%9B (4-bit) · AI2D thinking on: 76%76%accuracy (%)

Without thinking, GSM8K accuracy climbs steadily with size: 54%, 77%, 85%, 88%. Paired sign tests (appendix B) confirm the big steps: 0.8B to 2B is significant on GSM8K in both modes and on AI2D with thinking; 2B to 4B is significant with thinking on. From 4B to 9B-4bit, GSM8K without thinking is within noise.

The surprise is thinking mode. With thinking on, the 4B is significantly better than the larger 9B-4bit on both tasks (GSM8K 84% vs 70%, AI2D 85% vs 76%). Not because it reasons better, as the next chapter shows, but because the 9B runs out of tokens more often (45% vs 28% on GSM8K).

Takeaway. Model size buys accuracy, but only if the answer actually finishes. Whether the answer finishes depends on the token budget.

3. Thinking mode: powerful, until it runs out of tokens

Thinking helps vision, looks harmful on math, and the difference is almost entirely about truncation.

For the smaller models on AI2D, thinking is a clear win: 0.8B 41% to 48%, 2B 46% to 73%, 4B 45% to 85%. On GSM8K it goes the other way: the 0.8B falls from 54% to 22% and the 9B from 88% to 70%. Look at how often the models hit the cap:

Share of requests that hit the token cap. Up to 94% of the 0.8B's thinking runs on GSM8K never finish.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
0%20%40%60%80%100%GSM8Kthinking off0.8B (bf16) · GSM8K thinking off: 21%21%2B (bf16) · GSM8K thinking off: 18%18%4B (bf16) · GSM8K thinking off: 10%10%9B (4-bit) · GSM8K thinking off: 12%12%GSM8Kthinking on0.8B (bf16) · GSM8K thinking on: 94%94%2B (bf16) · GSM8K thinking on: 49%49%4B (bf16) · GSM8K thinking on: 28%28%9B (4-bit) · GSM8K thinking on: 45%45%AI2Dthinking off0.8B (bf16) · AI2D thinking off: 5%5%2B (bf16) · AI2D thinking off: 0%0%4B (bf16) · AI2D thinking off: 49%49%9B (4-bit) · AI2D thinking off: 0%0%AI2Dthinking on0.8B (bf16) · AI2D thinking on: 43%43%2B (bf16) · AI2D thinking on: 34%34%4B (bf16) · AI2D thinking on: 20%20%9B (4-bit) · AI2D thinking on: 36%36%hit token cap (%)

Thinking is expensive in tokens, from roughly 6x more on GSM8K to several hundred times more on AI2D, where the non-thinking answer is just a few tokens:

Mean generated tokens (log scale).
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
110100100010000GSM8Kthinking off0.8B (bf16) · GSM8K thinking off: 3573572B (bf16) · GSM8K thinking off: 3493494B (bf16) · GSM8K thinking off: 3283289B (4-bit) · GSM8K thinking off: 336336GSM8Kthinking on0.8B (bf16) · GSM8K thinking on: 296029602B (bf16) · GSM8K thinking on: 232023204B (bf16) · GSM8K thinking on: 184218429B (4-bit) · GSM8K thinking on: 22592259AI2Dthinking off0.8B (bf16) · AI2D thinking off: 442B (bf16) · AI2D thinking off: 334B (bf16) · AI2D thinking off: 17179B (4-bit) · AI2D thinking off: 22AI2Dthinking on0.8B (bf16) · AI2D thinking on: 130813082B (bf16) · AI2D thinking on: 113211324B (bf16) · AI2D thinking on: 9929929B (4-bit) · AI2D thinking on: 12141214mean generated tokens (log)

And the tokens are not evenly spent. Wrong answers are long: in thinking mode they use about 1.9x the tokens of correct ones. A model that is going to get it right usually does so in about 1362 tokens; one that is struggling keeps going until it hits the cap.

Mean tokens for correct vs wrong answers (thinking on, 1x caps).
correct answerswrong answers
0500100015002000250030000.8B (bf16)correct answers · 0.8B (bf16): 14821482wrong answers · 0.8B (bf16): 248524852B (bf16)correct answers · 2B (bf16): 13401340wrong answers · 2B (bf16): 258525854B (bf16)correct answers · 4B (bf16): 12161216wrong answers · 4B (bf16): 251125119B (4-bit)correct answers · 9B (4-bit): 14111411wrong answers · 9B (4-bit): 26172617mean tokens (thinking on, 1x)

What happens with 8x the budget?

To separate capability from budget we re-ran the first 20 questions of each task with caps 8x higher (24,576 tokens for GSM8K). If the cap were the whole story, accuracy would jump:

GSM8K, thinking on, same 20 questions at 1x and 8x caps.
1x cap (3072 tok)8x cap (24,576 tok)
0%20%40%60%80%100%0.8B (bf16)1x cap (3072 tok) · 0.8B (bf16): 15%15%8x cap (24,576 tok) · 0.8B (bf16): 35%35%2B (bf16)1x cap (3072 tok) · 2B (bf16): 50%50%8x cap (24,576 tok) · 2B (bf16): 80%80%4B (bf16)1x cap (3072 tok) · 4B (bf16): 70%70%8x cap (24,576 tok) · 4B (bf16): 95%95%9B (4-bit)1x cap (3072 tok) · 9B (4-bit): 65%65%8x cap (24,576 tok) · 9B (4-bit): 95%95%accuracy (%)
AI2D, thinking on, same 20 questions at 1x and 8x caps.
1x cap (2048 tok)8x cap (16,384 tok)
0%20%40%60%80%100%0.8B (bf16)1x cap (2048 tok) · 0.8B (bf16): 40%40%8x cap (16,384 tok) · 0.8B (bf16): 60%60%2B (bf16)1x cap (2048 tok) · 2B (bf16): 60%60%8x cap (16,384 tok) · 2B (bf16): 70%70%4B (bf16)1x cap (2048 tok) · 4B (bf16): 85%85%8x cap (16,384 tok) · 4B (bf16): 85%85%9B (4-bit)1x cap (2048 tok) · 9B (4-bit): 75%75%8x cap (16,384 tok) · 9B (4-bit): 85%85%accuracy (%)

It does, for every model: the 4B goes from 70% to 95% on GSM8K and the 9B from 65% to 95%, now with none of its samples truncated. With a big enough budget the 9B catches up with the 4B, which resolves the surprise in chapter 2. (n=20, so each number is good to about ±20 points; the direction is the same for all four models.) The price is time: the 4B's average answer on GSM8K takes ~131 s at 1x and ~410 s at 8x.

Share of requests hitting the cap, 1x vs 8x.
GSM8K, 1xGSM8K, 8xAI2D, 1xAI2D, 8x
0%20%40%60%80%100%0.8B (bf16)GSM8K, 1x · 0.8B (bf16): 95%95%GSM8K, 8x · 0.8B (bf16): 80%80%AI2D, 1x · 0.8B (bf16): 50%50%AI2D, 8x · 0.8B (bf16): 35%35%2B (bf16)GSM8K, 1x · 2B (bf16): 55%55%GSM8K, 8x · 2B (bf16): 25%25%AI2D, 1x · 2B (bf16): 40%40%AI2D, 8x · 2B (bf16): 30%30%4B (bf16)GSM8K, 1x · 4B (bf16): 40%40%GSM8K, 8x · 4B (bf16): 15%15%AI2D, 1x · 4B (bf16): 20%20%AI2D, 8x · 4B (bf16): 10%10%9B (4-bit)GSM8K, 1x · 9B (4-bit): 55%55%GSM8K, 8x · 9B (4-bit): 0%0%AI2D, 1x · 9B (4-bit): 40%40%AI2D, 8x · 9B (4-bit): 20%20%hit token cap (%)
Takeaway. "Thinking hurts math" was a budget artefact. Give it room and thinking becomes the best mode; but budget is wall-clock time on a laptop, so it is a real trade-off.

4. The ones that never finish: repetition loops, and a rescue attempt

More tokens fix most truncations, but not the loops. Those just loop for longer.

Not all truncated answers are alike. We classified each truncated thinking trace by whether its last 3,000 characters compress to under 15% of their size (a sure sign the model is repeating itself). At the standard caps, most truncations are genuinely long reasoning that would finish given more room:

Truncated thinking-on outputs at 1x caps (counts, GSM8K + AI2D).
repetition looplong but not looping
050100150200repetition loop: 2323long but not looping: 1141140.8B (bf16)repetition loop: 22long but not looping: 81812B (bf16)long but not looping: 48484B (bf16)long but not looping: 81819B (4-bit)

At 8x the picture flips: of the 39 truncated outputs from the 0.8B, 2B and 4B, 34 end in a loop. The extra budget rescued most of the answers that were merely slow, and what remains is mostly the model stuck in a cycle. The 9B almost never loops.

Truncated thinking-on outputs at 8x caps: the remainder is loops.
repetition looplong but not looping
051015202530repetition loop: 23230.8B (bf16)repetition loop: 77long but not looping: 442B (bf16)repetition loop: 44long but not looping: 114B (bf16)long but not looping: 449B (4-bit)

Can sampling settings pull a model out of a loop?

Qwen recommends top-k 20 and a presence penalty for thinking mode. Our first runs used neither, so we re-ran the truncated-and-wrong samples (up to 10 per task) with top-k 20 and presence penalty 1.5:

Share of re-run samples that became correct.
1x sets: rescued % of re-run8x sets: rescued % of re-run
0%20%40%60%80%100%0.8B (bf16)1x sets: rescued % of re-run · 0.8B (bf16): 30%30%8x sets: rescued % of re-run · 0.8B (bf16): 50%50%2B (bf16)1x sets: rescued % of re-run · 2B (bf16): 20%20%8x sets: rescued % of re-run · 2B (bf16): 71%71%4B (bf16)1x sets: rescued % of re-run · 4B (bf16): 30%30%8x sets: rescued % of re-run · 4B (bf16): 50%50%9B (4-bit)1x sets: rescued % of re-run · 9B (4-bit): 35%35%8x sets: rescued % of re-run · 9B (4-bit): 67%67%% of truncated+wrong samples now correct

It rescued roughly 20-35% of the 1x cases and 50-70% of the 8x cases. This is encouraging but weak evidence: there is no control arm (a re-run with the original settings), and the sets are tiny (2-20 samples per cell), so part of the rescue could be plain re-sampling luck. Details are in appendix D.

Takeaway. Budget fixes slowness; sampling settings may help with loops; bigger models loop less. Look at why answers are truncated before deciding the fix.

5. How fast is it, really?

Decode speed is set by one number: how many gigabytes of weights the chip must read per token.

Decode throughput (thinking on). It depends on the model, not the task.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
0.020.040.060.080.0GSM8K thinking on0.8B (bf16) · GSM8K thinking on: 70.670.62B (bf16) · GSM8K thinking on: 30.430.44B (bf16) · GSM8K thinking on: 14.114.19B (4-bit) · GSM8K thinking on: 25.925.9AI2D thinking on0.8B (bf16) · AI2D thinking on: 71.071.02B (bf16) · AI2D thinking on: 31.131.14B (bf16) · AI2D thinking on: 14.214.29B (4-bit) · AI2D thinking on: 26.026.0decode tokens / s

Generating a token means streaming essentially all the weights through the GPU once, so speed halves when weights double: 71 tok/s for the 0.8B, 30 for the 2B, 14 for the 4B. Multiply decode speed by weight size and you get the effective memory bandwidth, which lands between 112 and 143 GB/s for every model, most of what the chip can deliver. That is also why the 4-bit 9B (5.5 GB) generates at 26 tok/s, nearly twice the 4B in bf16 (8.5 GB).

Implied memory bandwidth = decode tok/s × weight size in GB, against Apple's quoted peak for the M5.
decode tok/s × weight GB
050100150200Apple-quoted M5 peak ≈ 153 GB/s0.8B (bf16)decode tok/s × weight GB · 0.8B (bf16): 1121122B (bf16)decode tok/s × weight GB · 2B (bf16): 1251254B (bf16)decode tok/s × weight GB · 4B (bf16): 1191199B (4-bit)decode tok/s × weight GB · 9B (4-bit): 143143implied GB/s

Prefill (reading the prompt) is compute-bound instead and runs at 400-2,600 tokens/s depending on the model; vision prompts carry about 200 extra image tokens, so they cost more to read.

Prefill throughput.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
050010001500200025003000GSM8K (~90 prompt tok)0.8B (bf16) · GSM8K (~90 prompt tok): 195219522B (bf16) · GSM8K (~90 prompt tok): 9349344B (bf16) · GSM8K (~90 prompt tok): 3843849B (4-bit) · GSM8K (~90 prompt tok): 394394AI2D (~310 prompt tok, incl. image)0.8B (bf16) · AI2D (~310 prompt tok, incl. image): 250825082B (bf16) · AI2D (~310 prompt tok, incl. image): 115811584B (bf16) · AI2D (~310 prompt tok, incl. image): 6156159B (4-bit) · AI2D (~310 prompt tok, incl. image): 487487prefill tokens / s
Mean end-to-end latency per request (log scale): sub-second for direct answers, minutes for thinking on the larger models.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
0.11101001000GSM8Kthinking off0.8B (bf16) · GSM8K thinking off: 4.94.92B (bf16) · GSM8K thinking off: 11.211.24B (bf16) · GSM8K thinking off: 23.423.49B (4-bit) · GSM8K thinking off: 12.912.9GSM8Kthinking on0.8B (bf16) · GSM8K thinking on: 42.042.02B (bf16) · GSM8K thinking on: 76.476.44B (bf16) · GSM8K thinking on: 131.0131.09B (4-bit) · GSM8K thinking on: 87.687.6AI2Dthinking off0.8B (bf16) · AI2D thinking off: 0.20.22B (bf16) · AI2D thinking off: 0.30.34B (bf16) · AI2D thinking off: 1.61.69B (4-bit) · AI2D thinking off: 0.70.7AI2Dthinking on0.8B (bf16) · AI2D thinking on: 18.618.62B (bf16) · AI2D thinking on: 36.736.74B (bf16) · AI2D thinking on: 70.570.59B (4-bit) · AI2D thinking on: 47.547.5mean latency, s (log)

6. What does a correct answer cost?

Direct answers are 6-20x cheaper than thinking on math, and the small models are not the cheap ones you would expect.

Wall-clock seconds per correct answer (total time / number correct), log scale.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
0.11101001000GSM8Kthinking off0.8B (bf16) · GSM8K thinking off: 9.19.12B (bf16) · GSM8K thinking off: 14.614.64B (bf16) · GSM8K thinking off: 27.527.59B (4-bit) · GSM8K thinking off: 14.714.7GSM8Kthinking on0.8B (bf16) · GSM8K thinking on: 190.8190.82B (bf16) · GSM8K thinking on: 117.5117.54B (bf16) · GSM8K thinking on: 156.0156.09B (4-bit) · GSM8K thinking on: 125.2125.2AI2Dthinking off0.8B (bf16) · AI2D thinking off: 0.40.42B (bf16) · AI2D thinking off: 0.70.74B (bf16) · AI2D thinking off: 3.73.79B (4-bit) · AI2D thinking off: 0.90.9AI2Dthinking on0.8B (bf16) · AI2D thinking on: 38.838.82B (bf16) · AI2D thinking on: 50.250.24B (bf16) · AI2D thinking on: 83.083.09B (4-bit) · AI2D thinking on: 62.562.5seconds per correct answer (log)

Dividing total time by the number of correct answers puts accuracy and speed on one axis. On GSM8K without thinking the 0.8B is cheapest per correct answer (about 9 s) but only because it is fast; the larger models waste fewer tokens (381 tokens per correct answer for the 9B against 662 for the 0.8B). With thinking on, every model pays about 120-190 s per correct GSM8K answer.

Generated tokens per correct answer, log scale.
0.8B (bf16)2B (bf16)4B (bf16)9B (4-bit)
110100100010000100000GSM8Kthinking off0.8B (bf16) · GSM8K thinking off: 6626622B (bf16) · GSM8K thinking off: 4544544B (bf16) · GSM8K thinking off: 3863869B (4-bit) · GSM8K thinking off: 381381GSM8Kthinking on0.8B (bf16) · GSM8K thinking on: 13456134562B (bf16) · GSM8K thinking on: 356935694B (bf16) · GSM8K thinking on: 219321939B (4-bit) · GSM8K thinking on: 32273227AI2Dthinking off0.8B (bf16) · AI2D thinking off: 992B (bf16) · AI2D thinking off: 664B (bf16) · AI2D thinking off: 39399B (4-bit) · AI2D thinking off: 33AI2Dthinking on0.8B (bf16) · AI2D thinking on: 272427242B (bf16) · AI2D thinking on: 155015504B (bf16) · AI2D thinking on: 116711679B (4-bit) · AI2D thinking on: 15981598generated tokens per correct answer (log)
Total generation time (26.7 h across 1,760 inferences): the 4B was the slowest model to benchmark at 10.1 h.
main benchmark (400 inferences)8x long-thinking (40 inferences)
0.02.04.06.08.00.8B (bf16)main benchmark (400 inferences) · 0.8B (bf16): 1.81.88x long-thinking (40 inferences) · 0.8B (bf16): 2.32.32B (bf16)main benchmark (400 inferences) · 2B (bf16): 3.53.58x long-thinking (40 inferences) · 2B (bf16): 2.92.94B (bf16)main benchmark (400 inferences) · 4B (bf16): 6.36.38x long-thinking (40 inferences) · 4B (bf16): 3.83.89B (4-bit)main benchmark (400 inferences) · 9B (4-bit): 4.14.18x long-thinking (40 inferences) · 9B (4-bit): 1.91.9hours of generation

7. How much of the 16 GB do we use?

The 0.8B and 2B are comfortable. The 4B in bf16 is right at the edge, and the 4-bit 9B has more headroom.

Peak MLX memory, peak system RAM and peak swap per model, against the 16 GB limit.
MLX peak GPU memorySystem RAM used (max)Swap used (max)
0.05.010.015.020.016 GB unified memory0.8B (bf16)MLX peak GPU memory · 0.8B (bf16): 2.52.5System RAM used (max) · 0.8B (bf16): 8.98.9Swap used (max) · 0.8B (bf16): 0.00.02B (bf16)MLX peak GPU memory · 2B (bf16): 5.15.1System RAM used (max) · 2B (bf16): 10.110.1Swap used (max) · 2B (bf16): 0.00.04B (bf16)MLX peak GPU memory · 4B (bf16): 9.49.4System RAM used (max) · 4B (bf16): 12.812.8Swap used (max) · 4B (bf16): 2.42.49B (4-bit)MLX peak GPU memory · 9B (4-bit): 6.56.5System RAM used (max) · 9B (4-bit): 11.711.7Swap used (max) · 9B (4-bit): 2.32.3GB

macOS and background services hold about 6.5 GB even with apps closed, leaving ≈9.5 GB for a model. The 4B in bf16 peaks at 9.4 GB of MLX memory and macOS used up to ~2.4 GB of swap, yet decode speed stayed flat at 14 tok/s, so the swap held idle pages, not the model. (The 9B runs showed ~2.3 GB of swap with only 6.5 GB of MLX memory: system-wide pressure.) The 0.8B and 2B never touched swap.

GPU utilisation was 97-99% for every thinking-on run and CPU utilisation only 6-12% (the CPU just feeds the GPU), so generation is fully GPU-bound. The thermal speed limit never dropped below 100%: the M5 did not throttle in 27 hours of generation, including single runs of about 6 hours.

8. So what can this MacBook do?

What fits, how long things take, and where the walls are.

≈9.5 GBusable for a model: macOS and background services keep ≈6.5 GB in use even with apps closed
≈125 GB/seffective memory bandwidth while decoding; tokens/s ≈ 125 ÷ weight GB
+0.9 GBextra memory for a ~25k-token generation: context is almost free
27 hof generation without throttling; peak RAM 12.8 GB

The memory budget

Where the 16 GB goes (peak working memory over all runs). The 4B in bf16 leaves almost nothing free; the 4-bit 9B has more headroom.
macOS + background (measured ≈6.5 GB)model weightsworking memory (activations + cache)left free
0510152016 GB unified memorymacOS + background (measured ≈6.5 GB): 6.56.5model weights: 1.61.6working memory (activations + cache): 0.90.9left free: 7.07.00.8B (bf16)macOS + background (measured ≈6.5 GB): 6.56.5model weights: 4.14.1working memory (activations + cache): 0.90.9left free: 4.44.42B (bf16)macOS + background (measured ≈6.5 GB): 6.56.5model weights: 8.58.5working memory (activations + cache): 0.90.9left free: 0.14B (bf16)macOS + background (measured ≈6.5 GB): 6.56.5model weights: 5.55.5working memory (activations + cache): 0.90.9left free: 3.03.09B (4-bit)

Context is nearly free

One of the pleasant surprises: a 25,000-token thinking trace adds under 1 GB. Qwen3.5 mostly uses gated-delta (linear) attention layers, whose state does not grow with length, so memory is decided by weights, not context.

Peak MLX memory above the weights for the longest request in each set.
longest 1x run (~3k tokens)longest 8x run (16-25k tokens)
0.000.200.400.600.801.001.200.8B (bf16)longest 1x run (~3k tokens) · 0.8B (bf16): 0.230.23longest 8x run (16-25k tokens) · 0.8B (bf16): 0.460.462B (bf16)longest 1x run (~3k tokens) · 2B (bf16): 0.210.21longest 8x run (16-25k tokens) · 2B (bf16): 0.420.424B (bf16)longest 1x run (~3k tokens) · 4B (bf16): 0.240.24longest 8x run (16-25k tokens) · 4B (bf16): 0.910.919B (4-bit)longest 1x run (~3k tokens) · 9B (4-bit): 0.420.42longest 8x run (16-25k tokens) · 9B (4-bit): 0.760.76GB above weights

What fits

Rows marked "(est.)" are extrapolations: weights = parameters × bytes per parameter (bf16 2, 8-bit ≈1.06, 4-bit ≈0.6), plus ~1 GB working memory against the usable ≈9.5 GB; decode speed = measured bandwidth ÷ weight size (this rule predicts the four measured models within about 15%).

Model sizePrecisionWeights GBOn this 16 GB MacDecode tok/s (thinking on)
0.8Bbf161.6fits comfortably71 (measured)
2Bbf164.1fits comfortably30 (measured)
4Bbf168.5tight (swap likely)14 (measured)
9B4-bit5.5fits comfortably26 (measured)
9Bbf1618.9 (est.)does not fit–
8B-class8-bit8.5 (est.)tight (swap likely)≈15
14B-class4-bit8.4 (est.)tight (swap likely)≈15
14B-class8-bit14.8 (est.)does not fit–
32B-class4-bit19.2 (est.)does not fit–

What an experiment costs in time

Time, not memory, is the practical limit for thinking-mode evaluation. The full GSM8K test split (1,319 questions) with the 4B thinking on would take about 48 hours at 1x caps and far longer at 8x:

ModelGSM8K (1,319) thinking offGSM8K thinking onAI2D (3,088) thinking offAI2D thinking on
0.8B (bf16)1.8 h15.4 h0.1 h16.0 h
2B (bf16)4.1 h28.0 h0.3 h31.5 h
4B (bf16)8.6 h48.0 h1.4 h60.5 h
9B (4-bit)4.7 h32.1 h0.6 h40.7 h

The ceiling is physical: at thinking-on speeds the machine generates about 6.1 M tokens a day with the 0.8B, 2.6 M with the 2B, 1.2 M with the 4B and 2.2 M with the 9B (4-bit).

Datasets: cost follows answer length

TaskPrompt tokensAnswer tokens (thinking off)Answer tokens (thinking on, 1x cap)What dominates the cost
GSM8K (text math)923432345generated reasoning tokens (decode)
AI2D (diagram MCQ, image ≤896 px)31271161image prefill when off; reasoning tokens when on

Cost scales with generated tokens, not prompt size. Short-answer multiple choice with thinking off (a few tokens) is cheap for every model; chain-of-thought math is 100-1,000x more expensive per question. When choosing an evaluation set for this machine, estimate the answer length first.

9. Conclusions

  1. Run the largest model whose weights fit in about 8 GB. On this Mac that is a 4B in bf16 or a 9B-class model at 4-bit; at the same memory, 4-bit buys a bigger and faster model that is at least as accurate without thinking.
  2. Treat the token budget as a first-class setting. It decides whether thinking helps or hurts. Check how many answers hit the cap before comparing models or modes.
  3. Speed is predictable. Tokens per second ≈ 125 ÷ weight size in GB. Use it to plan before running.
  4. Memory is not the long-context problem; time is. A 25k-token trace costs under 1 GB but takes ~30 minutes on the 4B.
  5. The machine is a reliable instrument. 27 hours of generation, no throttling, no crashes, including multi-hour single runs.

Practical recommendations

  • Interactive use, fast answers: 9B at 4-bit, thinking off (26 tok/s, 88% GSM8K).
  • Hard reasoning or diagrams, time to spare: 4B in bf16 with thinking on and a generous cap (84% at 1x, 95% at 8x on the sampled questions).
  • Prototyping and many quick evaluations: 0.8B or 2B; they leave several GB free and run at 30-70 tok/s.
  • Avoid: thinking mode with small caps on the 0.8B (94% truncation) and anything above ~8 GB of weights.

What we did not measure

Batched or concurrent generation (decode is bandwidth-bound, so batching should raise total throughput substantially, but we ran batch size 1), contexts beyond ~25k tokens, images larger than 896 px (they were downsized), power draw in watts and battery operation (it needs sudo powermetrics), and sustained load beyond single ~6-hour runs. All cells are n=100 or n=20 on one machine, so small differences are not meaningful.

What is next

Inference is half the story. The sibling project, Mac LLM/VLM Post-training Testing, asks the same capability question for training: SFT, GRPO, RLOO and GSPO, with LoRA and full fine-tuning, on the same machine. Early finding: the memory wall comes much sooner when you train. Read that post next.

Appendix

A. Full results table

ModelTaskThinkAccuracy (95% CI)Acc. if finishedCappedGen tokDecode tok/sPrefill tok/sTTFT sLatency sp95 ss / correctPeak GBCPU %GPU %RAM GBSwap GBTherm. limit %
0.8B (bf16)GSM8K (text)off54% (44–63)66%21%35773.519430.054.97.09.11.89978.70.0100
0.8B (bf16)GSM8K (text)on22% (15–31)100%94%296070.619520.0542.044.3190.81.810998.90.0100
0.8B (bf16)AI2D (vision)off41% (32–51)43%5%4140.526580.120.20.30.42.48888.20.0100
0.8B (bf16)AI2D (vision)on48% (38–58)68%43%130871.025080.1318.629.738.82.510998.60.0100
2B (bf16)GSM8K (text)off77% (68–84)91%18%34931.59290.1011.216.314.64.399810.00.0100
2B (bf16)GSM8K (text)on65% (55–74)100%49%232030.49340.1076.4102.3117.54.399910.10.0100
2B (bf16)AI2D (vision)off46% (37–56)46%0%355.811660.260.30.50.75.16959.00.0100
2B (bf16)AI2D (vision)on73% (64–81)95%34%113231.111580.2636.766.450.25.18999.30.0100
4B (bf16)GSM8K (text)off85% (77–91)93%10%32814.23900.2423.436.227.58.7109812.62.4100
4B (bf16)GSM8K (text)on84% (76–90)99%28%184214.13840.23131.0219.0156.08.789912.31.7100
4B (bf16)AI2D (vision)off45% (36–55)88%49%1720.46230.481.63.03.79.4109812.82.0100
4B (bf16)AI2D (vision)on85% (77–91)99%20%99214.26150.4870.5145.383.09.479912.82.1100
9B (4-bit)GSM8K (text)off88% (80–93)100%12%33626.54020.2312.919.614.76.079911.32.3100
9B (4-bit)GSM8K (text)on70% (60–78)100%45%225925.93940.2387.6119.5125.26.079911.42.1100
9B (4-bit)AI2D (vision)off82% (73–88)82%0%249.64850.630.71.20.96.569711.71.8100
9B (4-bit)AI2D (vision)on76% (67–83)100%36%121426.04870.6247.580.162.56.579911.71.8100

B. Paired sign tests between neighbouring model sizes

ComparisonTaskThinkOnly larger rightOnly smaller rightSign-test pp < 0.05
0.8B (bf16) → 2B (bf16)GSM8K (text)off2850.000yes
0.8B (bf16) → 2B (bf16)GSM8K (text)on4520.000yes
0.8B (bf16) → 2B (bf16)AI2D (vision)off23180.533no
0.8B (bf16) → 2B (bf16)AI2D (vision)on3160.000yes
2B (bf16) → 4B (bf16)GSM8K (text)off1130.057no
2B (bf16) → 4B (bf16)GSM8K (text)on2340.000yes
2B (bf16) → 4B (bf16)AI2D (vision)off20211.000no
2B (bf16) → 4B (bf16)AI2D (vision)on1420.004yes
4B (bf16) → 9B (4-bit)GSM8K (text)off740.549no
4B (bf16) → 9B (4-bit)GSM8K (text)on4180.004yes
4B (bf16) → 9B (4-bit)AI2D (vision)off3920.000yes
4B (bf16) → 9B (4-bit)AI2D (vision)on2110.022yes

C. Long-thinking runs (8x token caps)

ModelTaskAccuracy (95% CI, n=20)CappedGen tokDecode tok/sLatency sMax tokPeak GBSwap GB
0.8B (bf16)GSM8K (text)35% (18–57)80%2091967.4314245762.11.8
0.8B (bf16)AI2D (vision)60% (39–78)35%692066.8106163842.41.7
2B (bf16)GSM8K (text)80% (58–92)25%930430.1313245764.51.7
2B (bf16)AI2D (vision)70% (48–85)30%648631.0213163844.91.7
4B (bf16)GSM8K (text)95% (76–99)15%561814.1410245769.42.8
4B (bf16)AI2D (vision)85% (64–95)10%393314.1283163849.22.0
9B (4-bit)GSM8K (text)95% (76–99)0%438625.7172103686.11.7
9B (4-bit)AI2D (vision)85% (64–95)20%436925.6176163846.41.7

D. Mitigation runs (top-k 20 + presence penalty 1.5)

SetModelTaskRe-runRescuedStill cappedOrig. accAcc after
1x0.8B (bf16)AI2D (vision)10280%48%50%
1x0.8B (bf16)GSM8K (text)10460%22%26%
1x2B (bf16)AI2D (vision)10370%73%76%
1x2B (bf16)GSM8K (text)10190%65%66%
1x4B (bf16)AI2D (vision)10370%85%88%
1x4B (bf16)GSM8K (text)10390%84%87%
1x9B (4-bit)AI2D (vision)10190%76%77%
1x9B (4-bit)GSM8K (text)10650%70%76%
8x0.8B (bf16)AI2D (vision)6433%60%80%
8x0.8B (bf16)GSM8K (text)10410%35%55%
8x2B (bf16)AI2D (vision)5320%70%85%
8x2B (bf16)GSM8K (text)220%80%90%
8x4B (bf16)AI2D (vision)2150%85%90%
8x4B (bf16)GSM8K (text)00–95%95%
8x9B (4-bit)AI2D (vision)3233%85%95%
8x9B (4-bit)GSM8K (text)00–95%95%
Caveat. No control arm, tiny sets: indicative only.

E. Glossary

TermMeaning
TTFTTime To First Token: delay from sending the prompt until the first generated token appears (includes prefill).
Prefill tok/sPrompt tokens processed per second before generation starts (compute-bound).
Decode tok/sGenerated tokens per second after the first token (memory-bandwidth-bound).
Latency / p95End-to-end time per request (mean, and 95th percentile).
Capped / truncatedThe model hit the max-token limit before finishing; counted as wrong unless a correct final answer already appeared.
Acc. if finishedAccuracy over only the samples that finished within the cap (not truncated).
Peak GBPeak MLX (GPU) memory allocation during a request, from mx.get_peak_memory().
RAM / SwapSystem RAM used and swap used (psutil), sampled every 0.5 s; maximum over the run.
CPU % / GPU %Average system CPU utilisation and GPU device utilisation (ioreg 'Device Utilization %') during generation.
Therm. limitmacOS CPU speed limit from pmset; 100% = no thermal throttling.
95% CIWilson score interval for a proportion; with n=100 it is about ±10 points, with n=20 about ±20.
Sign testPaired exact binomial test on questions where exactly one of two models is correct.

F. Method notes and reproduction

  • Latency, tokens and memory columns are averages over the samples of that cell; Peak GB, RAM and Swap are maxima; Therm. limit is the minimum.
  • The 9B was converted to 4-bit (group size 64) because bf16 (19 GB) does not fit in 16 GB, so it is not an apples-to-apples precision comparison with the other three.
  • Sampling at T=0.6 makes thinking-on numbers vary run to run.




Share this article




Copyright © 2022-2026 Azmine Toushik Wasi. All rights reserved.