It learned a rate, not a rule

October 2026 · Post 6 in the series. Earlier: Keep the orchestrator dumb, What a dumb orchestrator actually looks like, What the smart parts are actually smart at, I built the parts. Here is what they said back, It read the story. Did it read the question?.

More data did not help on SVAMP, the test that changes a story to catch a guess. A new question on an old story did. Then I made that change stronger, and the result showed what the model had learned. It had learned how often to leave a number out. It had not learned which number to leave out.

The last post ended with three plans. The first, a role check in the floor, failed an offline test, so it did not receive a night on the graphics card. This post reports the other two plans and one more experiment. One passed its pass mark, and a second night on new seeds confirmed it. Two failed. The failure that taught me the most is the last one.

How the system works

The model reads a word problem from school maths and writes an expression over references to its numbers, such as ⟨#1−#2⟩. Exact, hand-written code, the floor, solves the expression. The model never calculates. The lower layers of its maths branch, the core, learn from a generated textbook, and then I freeze them. Real problems train only the layers above the core. The model has about eight million parameters and trains from zero on one graphics card.

Three test sets measure it:

• real_test: short problems from two older collections, MAWPS and ASDiv. Most need one or two steps.
• SVAMP: the same kind of story, changed to catch a solver that guesses from the story. Patel and colleagues changed the question, the order or the numbers, or they added a sentence with a number that the question does not need.
• GSM8K: longer problems with more steps.

Each experiment starts again from the same frozen core. Before each first run, I record the pass mark. Three training seeds give each result.

More data helps the test that looks like the data

Data teaches its own distribution. I added 4,176 problems from the GSM8K training set, with expressions that Claude wrote and the floor checked. GSM8K increased from 6.3% to 11.6%, on every seed. SVAMP did not move: 29.7% against 29.8%. real_test fell by 6.4 points. The pass mark was 35% on SVAMP, so the experiment failed.

With 514 added problems, GSM8K rose by 2 points and real_test fell by only 0.5 points on average. So the gain on the test that looks like the data came first. The cost on the short problems arrived with the larger dose, probably for two reasons. The older problems fell from 72% to 34% of the real data at the same number of steps. Also, 57% of the new expressions use one number twice, and the model copied that habit onto short problems. But that habit explains only about 34 of the 123 extra incorrect real_test answers.

The literature agrees with part of this. In MuggleMath, more GSM8K-style data helped GSM8K and gave little to no help on a harder test, MATH. A 70-billion-parameter model trained only on GSM8K-style data scored 22% on a Hungarian school exam, against 51% to 65% for its other data mixes. But the same paper reports that GSM8K-style data does lift SVAMP in large, pretrained models. My result is a small model from zero, at a fixed number of steps. It does not show that GSM8K data cannot help SVAMP.

Test the cause with a placebo

A gain needs a control that is the same in all ways except the cause. I gave Claude 1,200 real stories from MAWPS and ASDiv. For each story, it wrote a new question and a new expression, and the floor checked the expression. A second set, the placebo, used the same stories with the original question reworded. It kept the original expressions and used the same number of steps.

The new questions lifted SVAMP from 29.8% to 32.9%. The placebo fell to 28.5%, below the baseline on every seed. More time on the same stories did not help. The new questions did.

MetaMath points the same way. In one run on GSM8K, the authors added 20,000 examples to a base of 80,000. Reworded questions added 0.4 points, and questions that ask for a different number added 2.3 to 2.6 points. But a placebo with the same steps is rare in this work. When other researchers added a control of the same size, a gain sometimes disappeared. On two language tests, SNLI and BoolQ, Huang, Liu and Bowman found that edited pairs did no better than the same amount of plain data.

Check a narrow pass on new seeds

A pass by one problem needs a second night. The result passed its mark, the baseline plus 3 points, by one problem in 2,994. That is too small to believe, so I ran a second night on three new seeds. I also removed 79 SVAMP problems near a training problem. On the rest, the new questions led on all three new seeds. Over six seeds the gain is 3.6 points, and the new questions lead on every seed.

The new questions also changed how the model uses the question. When I delete the question from a SVAMP problem, the baseline keeps 61% to 71% of its score over six seeds. With the new questions, the model keeps 54% to 60%, less on every seed. This test scores the first answer without the answer check, because the check reads the question. The model depends on the question more.

The gain and the cost were one effect

When a change helps, find the rows it helped. 39% of the new questions use fewer numbers than the original question did. The model learned that a question can ignore a number, and it began to leave out numbers that it needed. One SVAMP problem says that a machine worked for 5 minutes and can make 4 shirts each minute. The model wrote ⟨#2⟩, the rate alone, and stopped. Over three seeds on SVAMP and real_test together, incorrect answers that leave out a needed number increased from 410 to 891.

I divided the far SVAMP problems into two kinds. In subset problems, the correct expression uses fewer numbers than the problem has. In full problems, it uses all of them. On the first three seeds and a slightly different far list, all of the gain was on the subset problems: 7.6% to 17.0%. On the full problems, accuracy fell a little, from 47.1% to 45.1%.

So the gain and the cost were not two effects. They were one effect. The model learned to leave numbers out. That helps where a number must be ignored, and it hurts where every number counts. I did not find a paper that reports this cost. The published methods that write new questions, such as MetaMath's backward questions, keep every number in use.

The model learned a rate, not a rule

Make a lever stronger, and it shows what it is. SVAMP makes many subset problems with one trick: it adds a sentence with a number that the question does not need. I did the same to the training stories. Claude wrote one added sentence for each story, with one new number. The question and the expression did not change, apart from the reference numbers. Another mix, the mechanism test, used the new questions without the subset questions.

On the subset problems, the added sentences lifted accuracy to 24.3%, more than three times the baseline. On the full problems, accuracy fell to 41.6%, and real_test fell from 74.0% to 70.5%. The experiment failed on two parts of its pass mark. SVAMP rose by only 1.3 points over the new questions, under the 2 points I asked for. On every seed, real_test fell by more than 2 points.

On SVAMP, the four training mixes form one line: subset accuracy increases as full accuracy decreases. I also measured the rate directly, as the share of all SVAMP answers that use fewer numbers than the problem has. It increases from 19% to 31%, 37% and 51%. The more training questions ignore a number, the more often the model leaves one out. Take the SVAMP problems that the new questions solved and the added sentences did not. In 41% of them, the answer leaves out a needed number.

One effect, two test sets

The same effect explains both test sets. On subset problems, the added sentences gained about 7 points over the new questions on SVAMP, and about 9 points on real_test. On full problems, they lost on both. SVAMP increased overall because 45% of its far problems are subset problems. On real_test, only 16% are, so its total fell.

The totals on real_test do not form a line: 74.0%, 73.1%, 74.0% and 70.5%. A small change in the rate does not show in a test that has few subset problems. The model did not learn two things. It learned one rate, and the two test sets weigh it differently.

The rest of the evidence

Report the parts that did not fit. The mechanism test kept 57% of the subset gain, against a pass mark of less than half, so it failed near the mark. The added sentences also trained for more steps (21,688 against 18,059), and I did not test whether the extra steps cost real_test. A blind audit found 3 incorrect added sentences in 60, at the limit: each was between a noun and its pronoun. But the mechanism mix has no added sentences and is on the same line. So these errors do not explain the loss on full problems.

Other researchers measured distractors at test time. In GSM-Symbolic, one clause that needs no operation took a small model from 83.7% on GSM8K to 18.0%. In GSM-DC, a 1-billion-parameter model trained on synthetic problems with 0 to 2 distractors scored 67.3% step accuracy on clean problems. Trained on clean problems, it scored 82.0%. Training with many distractors did not cause that loss. I did not find a paper that relates two rates: how often training questions ignore a number, and how often a model drops one.

This is a known kind of failure, under another name. McCoy, Pavlick and Linzen showed that inference models adopt a shortcut, such as word overlap, when the training data supports it. My model found a shortcut in the statistics of the questions.

Where this sits

Say where the result is, and what is common. It is standard to translate a word problem into an expression and let code solve it. Older solvers such as GTS and Graph2Tree do it, and so do PAL and TinyGSM. On SVAMP, my 33% is near the level of the solvers that Patel and colleagues trained from zero in 2021: 30.8% and 36.5%. It is below the solvers that start from a pretrained model. On GSM8K my model is weak.

An older solver had the part that my model does not have. Roy and Roth trained a classifier that scores, for each number, whether the answer uses it. The solver uses that score when it builds the expression. I did not find that step in the end-to-end solvers that came later. My model has no label for "this number is not used". The question is its only signal.

What I have not seen often is the combination. The maths core is frozen, and the router is a fixed rule. The model has eight million parameters and trains from zero. Each experiment has a pass mark that I record first. I did not find another study of word problems in this form. That does not mean that there is none.

What it costs

Each experiment costs one or two nights on one graphics card. The four nights in this post took 6 h 32 m, 5 h 30 m, 4 h 58 m and 5 h 30 m. Claude wrote the training data. The GSM8K expressions took about 5.5 million tokens. Each later set (the new questions, the placebo and the added sentences) took about 1 to 1.2 million.

The small size is a cost too. Three seeds give each result, and the SVAMP effects are one or two seed spreads. A pass by one problem needed a second night before I could believe it. Three results are clear: the GSM8K rise, the six-seed gain of 3.6 points, and the 7-point subset gain of the added sentences. Their 1.3-point gain on the far SVAMP problems is not clear.

What is next

Remove the rate from the data, so that only the question can teach the rule. The next experiment uses pairs. Each story with an added sentence also has a twin with the same sentence and a question that needs the new number. Then "an extra number is never used" is false in the training data. Only the question tells the model which number to leave out.

The literature gives a caution. Joshi and He found that pairs may not teach the features that the edits do not change. In the McCoy study, added training examples against the shortcut cost the two weaker models 3 to 6 points on their clean test. So the pass mark for the next experiment must include the full problems and real_test, not only the problems that it is designed to help.

Post one said that tool use works better than simulation in the weights. The model now does only the part that no library does: it reads the language. These three experiments show how it reads. It learns what the training data makes common. If your model improves, ask which rate it learned.