Teaching a 0.8B model to ignore distractors: SFT vs RL on math story problems



Mac LLM/VLM Post-training Testing · math-story-problems

November 18, 2026 · Qwen3.5-0.8B with LoRA · Apple M5 MacBook Pro, 16 GB · 6 runs

The question

On GSM8K, the model's own reasoning was already strong, and supervised fine-tuning made it worse. Would the picture change on a dataset where the answers are short and the failure mode is different?

We used azminetoushikwasi/math-story-problems. Every item is one arithmetic problem, three numbers joined by two plus or minus signs, written four ways: a bare equation, a short story, a story with distractor item types, and a long story with many irrelevant sentences. The answer is always a single integer. That makes it a clean test of one skill: pull the right numbers out of a story and calculate.

We trained on 500 items (each shown in one randomly chosen story form) and tested on 100 held-out items in all four forms, so each score rests on 400 prompts. Six recipes were compared on the same model: three kinds of supervised fine-tuning (SFT) and three reinforcement-learning (RL) methods.

The short answers

  • The untrained model fails on length, not arithmetic. It gets 90% of bare equations right but only 8% of long stories, and 18% of its answers overall run out of tokens.
  • SFT works here, when the reference solutions are executable. Equation-derived solutions took the model from 56% to 94%, and a gentler learning rate to 97.8%. Training on the model's own correct answers barely helped (64%).
  • RL needs no reference solutions and does as well or better. GRPO reached 96% after one epoch, RLOO reached 99.5% after one epoch, GSPO reached 95% after one epoch; final scores: GRPO 99%, RLOO 98%, GSPO 99.2%.
  • RL also makes the model concise. Mean answer length fell from 302 tokens to 149 (GRPO), 73 (RLOO), 223 (GSPO).

1. Where the untrained model fails

It can do the arithmetic. It cannot cope with long stories.

Untrained accuracy and truncation by question form (100 test questions each).
accuracyanswers cut off at the 512-token cap
0255075100Equationaccuracy · Equation: 9090answers cut off at the 512-token cap · Equation: 22Short storyaccuracy · Short story: 7474answers cut off at the 512-token cap · Short story: 33Distractor itemsaccuracy · Distractor items: 5454answers cut off at the 512-token cap · Distractor items: 11Long storyaccuracy · Long story: 88answers cut off at the 512-token cap · Long story: 6565% of 100 test questions

Accuracy falls steadily as the story gets longer: 90% for the bare equation, 74% for the short story, 54% with distractor items, and 8% for the long story. In the long-story form the model writes about 484 tokens on average, and 65% of its answers are cut off at the 512-token cap, so it is failing as much by rambling as by getting things wrong.

2. Supervised fine-tuning: it works, if the targets are executable

Imitating short, correct arithmetic lifts the model from 56% to as high as 97.8%.

The dataset has only final answers, so for SFT we generated the reference solutions from the equation: 2385 + 761 = 3146, then 3146 - 1063 = 2083, then The answer is: 2083. Three variants were tried: the original recipe (learning rate 2e-5, 5 epochs), a gentle one (5e-6, 2 epochs), and self-distillation, which trains on the model's own correct finished answers.

Test accuracy over the training epochs, all runs (epoch 0 is the untrained model).
SFT (equation solutions)SFT gentleSFT self-distilledGRPORLOOGSPO
0%25%50%75%100%012345SFT (equation solutions)SFT (equation solutions): 56% at 0SFT (equation solutions): 65% at 1SFT (equation solutions): 84% at 2SFT (equation solutions): 92% at 3SFT (equation solutions): 94% at 4SFT (equation solutions): 94% at 5SFT gentleSFT gentle: 56% at 0SFT gentle: 97% at 1SFT gentle: 98% at 2SFT self-distilledSFT self-distilled: 56% at 0SFT self-distilled: 68% at 1SFT self-distilled: 64% at 2GRPOGRPO: 56% at 0GRPO: 96% at 1GRPO: 97% at 2GRPO: 98% at 3GRPO: 99% at 4GRPO: 99% at 5RLOORLOO: 56% at 0RLOO: 100% at 1RLOO: 100% at 2RLOO: 100% at 3RLOO: 98% at 4RLOO: 98% at 5GSPOGSPO: 56% at 0GSPO: 95% at 1GSPO: 97% at 2GSPO: 99% at 3GSPO: 99% at 4GSPO: 99% at 5epochtest accuracy (all four forms)

The equation-derived solutions are a very different story from GSM8K's human-written ones. The model learns a ~42-token style it can carry out reliably, and the long-story form jumps from 8% to 96%. The gentle recipe gets there in a single epoch and never shows the early dip on bare equations that the original recipe does. Self-distillation, by contrast, only reinforces what the model already does: its data covers only the 65% of questions the model already solves, so it adds little where the model is weakest (the long story reaches 20%).

Takeaway. SFT quality depends on whether the reference solutions are something the model can execute. Short, correct, templated solutions work; on GSM8K the human solutions did not.

3. Reinforcement learning: the same reward, no reference solutions

The RL methods reach the top of the dataset, and they get there fast.

GRPO, RLOO and GSPO see only the questions and a correct-or-not reward: no reference solutions, no hints about the equation. Learning rate 2e-6, four sampled answers per question, up to 640 new tokens.

Training reward (share of sampled answers that finish and are correct), 25-step moving average.
GRPORLOOGSPO
0%25%50%75%100%0100200300400500600GRPORLOOGSPOoptimizer steptraining reward (25-step average)

GRPO scored 96% on the test set after a single epoch and RLOO scored 99.5% on the test set after a single epoch and GSPO scored 95% on the test set after a single epoch. By the end GRPO reached 99%, RLOO reached 98%, GSPO reached 99.2%. At this level (about 400 prompts, one seed) the differences between methods are within noise; what is clear is that RL matches the best SFT recipe without needing its reference solutions.

Final test accuracy by question form, all runs.
SFT (equation solutions)SFT gentleSFT self-distilledGRPORLOOGSPO
0255075100EquationSFT (equation solutions) · Equation: 8989SFT gentle · Equation: 9696SFT self-distilled · Equation: 8383GRPO · Equation: 9999RLOO · Equation: 9797GSPO · Equation: 9999Short storySFT (equation solutions) · Short story: 9595SFT gentle · Short story: 9999SFT self-distilled · Short story: 9090GRPO · Short story: 9999RLOO · Short story: 9898GSPO · Short story: 100100Distractor itemsSFT (equation solutions) · Distractor items: 9696SFT gentle · Distractor items: 9898SFT self-distilled · Distractor items: 6161GRPO · Distractor items: 9999RLOO · Distractor items: 9898GSPO · Distractor items: 9999Long storySFT (equation solutions) · Long story: 9696SFT gentle · Long story: 9898SFT self-distilled · Long story: 2020GRPO · Long story: 9999RLOO · Long story: 9999GSPO · Long story: 9999final test accuracy, %

RL also changes how the model writes

Because there is no length penalty, the answer length shrinking is a by-product: shorter answers finish more often and so earn more reward.

Mean answer length during RL training (25-step moving average).
GRPORLOOGSPO
01002003004005006000100200300400500600GRPORLOOGSPOoptimizer stepmean answer length (tokens, 25-step average)
Seconds per optimizer step during RL training: faster as answers shorten, which made these runs far cheaper than the GSM8K ones.
GRPORLOOGSPO
01020300100200300400500600GRPORLOOGSPOoptimizer stepseconds per step (25-step average)

Why the late steps change little

Once the model answers almost everything correctly, most question groups have four identical rewards and therefore no gradient: by the end, 98% (GRPO), 98.8% (RLOO), 97.8% (GSPO) of groups carry no signal. Training keeps running, but very little changes after the first epochs.

Share of question groups where all four sampled answers score the same (no learning signal).
GRPORLOOGSPO
0%25%50%75%100%0100200300400500600GRPORLOOGSPOoptimizer stepshare of question groups with no signal

4. SFT versus RL, side by side

Mean answer length at the end of training.
mean answer length at the end (tokens)
0100200300400SFT (equation solutions)mean answer length at the end (tokens) · SFT (equation solutions): 4242SFT gentlemean answer length at the end (tokens) · SFT gentle: 4242SFT self-distilledmean answer length at the end (tokens) · SFT self-distilled: 286286GRPOmean answer length at the end (tokens) · GRPO: 149149RLOOmean answer length at the end (tokens) · RLOO: 7373GSPOmean answer length at the end (tokens) · GSPO: 223223tokens
RunStatusEquationShort storyDistractor itemsLong storyAllTokensTruncatedHours
Untrained–90%74%54%8%56%30218%–
SFT (equation solutions)done89%95%96%96%94%420%0.3
SFT gentledone96%99%98%98%97.8%420%0.2
SFT self-distilleddone83%90%61%20%64%28614%0.4
GRPOdone99%99%99%99%99%1490%1.6
RLOOdone97%98%98%99%98%730%1.1
GSPOdone99%100%99%99%99.2%2230%2.3

5. Same methods, two datasets

The same recipes behave very differently on GSM8K and on math-story-problems.

Method (LoRA)GSM8K (untrained → final)GSM8K hoursmath-story (untrained → final)math-story hours
SFT, original recipe (human solutions on GSM8K; equation solutions here)54% → 22%0.356% → 94%0.3
SFT, gentle recipe54% → 37%0.156% → 97.8%0.2
SFT, self-distilled54% → 54%0.356% → 64%0.4
GRPO54% → 64%3.856% → 99%1.6
RLOO54% → 67%3.456% → 98%1.1
GSPO54% → 59%3.156% → 99.2%2.3

On GSM8K the model already reasons well in its own long, step-by-step style, supervised fine-tuning on human solutions pulled it away from that style, and RL gave gains of 5 to 13 points (up to 16 with full fine-tuning). Here the untrained model is held back by rambling, a short templated style is something it can execute, and so both SFT on clean solutions and RL give very large gains. The shared lesson: post-training helps when it teaches something the model can actually do, and the cheapest reliable signal is a verifiable reward.

6. How the MacBook performed

Accuracy is only half of a benchmark. This chapter reads the system counters logged at every step to see what the machine did while it trained: where the time went, how busy the GPU and CPU were, and whether the laptop ever slowed down.

These are the six 0.8B LoRA runs on math-story. Compared with the GSM8K runs of the same methods, answers are much shorter, so an RL step took 8 s on average instead of 19 s, and the whole RL run 1.7 h instead of 3.4 h.

74%GPU busy, average over all steps
14%CPU busy, average over all steps
100%lowest thermal speed limit seen in 6 h of training
14.1 GBhighest system RAM used (of 16 GB)
3.0-4.2 GBswap in use during runs
7.0 GBhighest MLX (GPU) memory

Across 6 runs (6 hours of training in total) the GPU was busy 74% of the time on average and the CPU only 14%: training is a GPU job, and the CPU mostly feeds it. The thermal speed limit never dropped below 100%, so the laptop never throttled, even through multi-hour runs on a plugged-in laptop.

An RL step is mostly generation

Each RL step first samples 16 answers and then trains on them. Sampling took 85-87% of the step time (86% on average). Sampling is memory-bandwidth bound exactly as in the inference benchmark, so the lever that matters is how many tokens you generate, not how clever the optimiser is. Batching helps a lot: 16 sequences decoded together reach 451-499 tokens/s in total, against about 72 tokens/s for a single stream on the same model.

Mean seconds per RL step, split into sampling (generation) and the policy update (training).
generationtraining
0.02.55.07.510.012.5GRPOgeneration · GRPO: 6.96.9training · GRPO: 1.01.0RLOOgeneration · RLOO: 4.44.4training · RLOO: 0.80.8GSPOgeneration · GSPO: 10.210.2training · GSPO: 1.71.7seconds per step

Throughput

Tokens per second: generation during RL sampling (batch of 16) and training (forward + backward) for every run.
generationtraining
0200400600SFTtraining · SFT: 170170SFT gentletraining · SFT gentle: 165165SFT self-distilledtraining · SFT self-distilled: 349349GRPOgeneration · GRPO: 451451training · GRPO: 125125RLOOgeneration · RLOO: 499499training · RLOO: 118118GSPOgeneration · GSPO: 480480training · GSPO: 160160tokens / s

GPU and CPU load

GPU utilisation sat between 51% and 92% for every run, while the CPU stayed at 9-20%.

Average GPU and CPU utilisation per run (system counters sampled during every step).
GPU busy %CPU busy %
0255075100SFTGPU busy % · SFT: 5353CPU busy % · SFT: 1616SFT gentleGPU busy % · SFT gentle: 5151CPU busy % · SFT gentle: 2020SFT self-distilledGPU busy % · SFT self-distilled: 9292CPU busy % · SFT self-distilled: 1414GRPOGPU busy % · GRPO: 8484CPU busy % · GRPO: 1414RLOOGPU busy % · RLOO: 8282CPU busy % · RLOO: 99GSPOGPU busy % · GSPO: 8282CPU busy % · GSPO: 99% busy

Memory: always at the limit

System RAM read 13.6-14.1 GB in every run, almost the whole 16 GB, and the swap file held 3.0-4.2 GB. This is mostly macOS using memory for caches and other apps, so it says more about the machine than about training, but it also means there is no slack: the runs that peaked at 7.0 GB of MLX memory had only a few GB of headroom. The steady swap use did not visibly slow the runs.

What each run cost

RunStepsHourss / stepSampling shareGen tok/sTrain tok/sGPUCPUPeak MLX GBRAM GBSwap GBGain (pts)Pts / hour
SFT6250.31.1––17053%16%4.013.63.1+38+127.8
SFT gentle2500.21.1––16551%20%3.813.84.2+41+271.6
SFT self-distilled2760.43.5––34992%14%7.013.73.0+7+16.6
GRPO6251.67.987%45112584%14%6.814.04.2+43+27.1
RLOO6251.15.285%49911882%9%6.913.93.3+42+39.5
GSPO6252.311.986%48016082%9%6.814.13.9+43+18.3

Gain is final minus untrained test accuracy in points; points per hour divides it by the wall-clock hours of the whole run, evaluations included (one evaluation of the test set takes about 118 s). Energy and battery use were not measured: that needs administrator access to the power counters.

Takeaway. On this MacBook training is GPU-bound and thermally comfortable; the cost of RL is almost all sampling time, so it scales with how many tokens the model writes, and the memory headroom, not the clock speed, is the constraint to plan around.

7. Conclusions

  1. On a clean, verifiable task, RL with a correctness-only reward matched the best supervised recipe without reference solutions.
  2. SFT works when reference solutions are short, correct and executable; a gentler learning rate was better than the aggressive one.
  3. Self-distillation preserves a model's style but cannot teach what the model never gets right.
  4. RL makes answers shorter as a side effect, which also made the RL runs several times cheaper than their GSM8K counterparts.
  5. After the first epoch, most RL groups carry no signal, which suggests shorter training would have been enough (not tested).

Caveats

One seed per recipe and 400 test prompts (100 per form), so differences of a few points are not meaningful near the 98-100% ceiling. The SFT targets are an easy deterministic template. Test equations never appear in the training data, but the problem templates do, so this measures skill on this family of problems, not general math. Late in some RL runs (e.g. RLOO), accuracy slipped a few points as answers kept shrinking; the final-epoch score is reported.

Appendix

A. Accuracy by epoch, all runs

RunEpochEquationShort storyDistractor itemsLong storyAllTokensTruncated
SFT (equation solutions)090%74%54%8%56%30218%
SFT (equation solutions)156%67%68%68%65%430%
SFT (equation solutions)269%89%87%89%84%420%
SFT (equation solutions)387%94%93%94%92%420%
SFT (equation solutions)489%95%95%95%94%420%
SFT (equation solutions)589%95%96%96%94%420%
SFT gentle090%74%54%8%56%30218%
SFT gentle197%97%97%97%97%420%
SFT gentle296%99%98%98%97.8%420%
SFT self-distilled090%74%54%8%56%30218%
SFT self-distilled189%92%83%10%68%28310%
SFT self-distilled283%90%61%20%64%28614%
GRPO090%74%54%8%56%30218%
GRPO197%98%96%91%96%1370%
GRPO297%97%97%97%97%1370%
GRPO397%98%98%99%98%1470%
GRPO499%100%99%99%99.2%1490%
GRPO599%99%99%99%99%1490%
RLOO090%74%54%8%56%30218%
RLOO199%100%100%99%99.5%1230%
RLOO2100%100%100%100%100%950%
RLOO399%100%100%99%99.5%720%
RLOO497%98%98%99%98%730%
RLOO597%98%98%99%98%730%
GSPO090%74%54%8%56%30218%
GSPO1100%99%97%84%95%2321%
GSPO296%99%98%96%97%2620%
GSPO3100%100%99%97%99%2280%
GSPO499%100%99%99%99.2%2210%
GSPO599%100%99%99%99.2%2230%

B. Method and data notes

  • Data: train 500 items from the 131,072-item training split (seeded; each uses one of the three story forms); evaluation on the first 100 items of the 3,072-item test split, in all four forms; greedy decoding, 512-token cap.
  • Hyperparameters: LoRA rank 16, scale 20; SFT lr 2e-5 (5 epochs); gentle SFT and self-distilled SFT lr 5e-6 (2 epochs); RL lr 2e-6, 4 prompts x 4 rollouts per step, temperature 1.0, up to 640 new tokens, correctness-only reward, 5 epochs; gradient checkpointing, clipping 1.0.
  • W&B project: macbook-posttrain-mathstory; all metrics, rollouts and evaluation answers are kept under runs_mathstory/; model weights were deleted locally after the runs (re-run train.py to regenerate). See also the research journal (JOURNAL.md) and the GSM8K report.




Share this article




Copyright © 2022-2026 Azmine Toushik Wasi. All rights reserved.