October 2026 · Post 5 in the series. Earlier: Keep the orchestrator dumb, What a dumb orchestrator actually looks like, What the smart parts are actually smart at, I built the parts. Here is what they said back.
In the last post, the model had one job. It translates a school maths word problem into an expression, and exact code solves it. Since then I did five experiments with real word problems. Before each first run, I recorded its pass mark. One experiment passed.
Essentially, this is the result. Two things made the model better: more exact code around it, and more real problems that people wrote. Three things did not: a judge trained on the model, the agreement of the model with itself, and a larger model. On SVAMP, a test set with questions written to catch this error, the error that remains is the question. The model reads the question, but without the question its first answer still keeps about 70% of its score.
The limit of scale is the same as before: about eight million parameters, a few hours of training, a few thousand test problems. This can show that a mechanism fails. It cannot show that an architecture is correct.
How the system works
- The floor is all the exact, hand-written code in the system. It never learns, and tests check each part of it against independent code.
- The indexer is part of the floor. It gives each number in the problem a reference: the first number is
#1, the second is#2. - The model reads the problem and writes an expression over those references, such as
⟨#1−#2⟩. It never writes a digit. The floor solves the expression and writes the steps and the answer. - The maths branch is the part of the model that writes expressions. Its lower layers, the core, learn from a generated textbook, and then I freeze them.
Give the exact code every job that it can do
Real problems train the layers above a frozen core. A hash of the core weights shows that they do not change. With training above the core only, accuracy on new real problems increased from 2% to 41%, against a pass mark of 60%. Across all three test sets, half of the incorrect answers did not use a number that they needed. The router, a small learned part, decides how much of each piece of text goes to the maths branch. It also started to send ordinary English there: 5.2% to 5.7% of its weight, against a limit of 5%.
Two more jobs go to the floor. Exact code knew how to do both jobs, but the model had to do them. The indexer knows which number is #2, so now it writes the reference into the text: cut down 18 (#2) trees. The floor knows when an expression is open, so now a mask with no parameters sends the expression to the maths branch, and nothing else.
That mask is the dumb orchestrator from post two in its most simple form. With both changes, the experiment passed: 63%, 66% and 63% in three training runs, against 60%. The mask gave about 12 points and the references gave about 4. With the mask alone, two of the three runs stayed below 60%.
Let the floor check the answer
An answer check costs nothing. The model now gives eight candidate expressions, and the floor solves each one. If the first answer cannot fit the question, the first candidate that fits replaces it. A negative number of cages cannot fit, and half a child cannot fit. This added 5 points on real problems and 8 points on SVAMP. It did not change one correct answer into an incorrect answer, because it changes only the answers that its rule rejects.
Templates do not transfer. In the same experiment, I trained the model on generated stories of six types, such as change, compare and rate. Each story had a pair that changed only the phrase that decides the operation. On new stories of the same types, accuracy increased from 56% to 99%. On SVAMP, without the answer check, it increased by 0.3 of a point. The model learned my templates, not the language.
The model cannot judge its own answers
With the answer check, one of the eight candidates has the correct answer for 57% of SVAMP problems. The model selects it for only 27%. Its own probabilities could not select better: in an offline test, no rule that used them gained more than half a point. I tried two other signals:
- A verifier. This is a small extra network that reads the internal state of the model. I trained it to find the correct expression among expressions that are almost correct.
- Agreement. I wrote each problem four more times, with new numbers and the same words. A candidate that the model writes for all four copies is probably what the words ask for.
The verifier made the result much worse, and agreement did nothing. On real problems, the verifier changed 662 correct answers to incorrect answers, and it corrected only 75. With both signals together, most of the damage (69%) kept the correct numbers and changed the operator. The verifier was trained to make exactly that decision, and it failed.
Agreement measures something else: confidence. On real problems, when the first answer is the same for all four copies, it is correct 81% of the time. When it is never the same, it is correct 30% of the time. Confidence tells you when not to trust the model, but not which candidate to select. Even a perfect selector in the problems where the copies disagree gains only 6 points on SVAMP. I keep two rules from this experiment:
- A selector must be better than the order of the model on saved candidates before it can use GPU time.
- A selector must not replace an answer that already fits, unless it has a clear margin.
Check the data before you blame the model
The last experiment changed the training data and the size of the model. I added problems from GSM8K, a set of longer school maths problems with several steps. I used only short problems whose answer the floor can calculate from references alone. Most of the others needed a constant, were too long, or used a reference in a way that the rules do not permit. Of 7,473 problems, 1,101 passed, which gave 40% more real training problems. For size, I put six layers above the frozen core, not two.
The data helped in each of the three training runs: SVAMP increased by 5 points on average, from 25% to 30%. The size did nothing. Each difference was smaller than the differences between training runs, and each training step took about 50% more time. Three times more training on the same data helped only the familiar test set. The best model was the small model with more data.
This is not a verdict on the bitter lesson, the idea that general methods with more computation win in the end. It is one training length, at a small size, with a narrow slice of new data. But at this size, the model did not have too few parameters. It had too few examples. The experiment failed its gate: the pass mark was 35%, and SVAMP stopped at 30%.
The question on the page
If we look deeper at the incorrect answers, we see a pattern:
- Paco has salty cookies and sweet cookies, and the question asks how many salty cookies are left. The model used all four numbers.
- A problem says that there are 228 more girls than boys, and asks how many boys there are. The model added the two numbers.
- A story gives a number of birds and a number of storks, and asks how many birds sit on the fence. The model added the storks.
A model that adds the storks to the birds has read many stories about birds. I tried to solve a different problem, which was how to select from the candidates. The real issue is the question.
SVAMP is made to find this error. One of its types of change asks a different question about the same story. Their paper trained solvers with the questions and tested them without the questions. On SVAMP, their best solver was correct on 44% with the question and 31% without it. Twenty incorrect answers are not evidence, so I repeated their test on my model.
Delete the question and measure
I deleted the question from 957 SVAMP problems and kept all other text. The other 41 problems have a number in the question, so I did not use them. I compared the first answer of the model, without the answer check, because the check reads the question. With the question, the models were correct on about 20% of the problems. Without it, they were correct on about 14%. For each of the four models, that is 66% to 72% of the score, which is the ratio of the best solver in the paper.
But the model does not ignore the question. When I delete it, the expression changes in about 72% of the problems. On average, less than half of the correct answers keep the same expression. Each run also has 33 to 52 problems that are correct only without the question.
The question also changes most of the incorrect answers: only about a quarter of them stay the same without it. The model reads the question, but the question does not move it to the correct expression. More data did not change this, because the models with GSM8K problems had the same ratio. A problem with no question is unlike all the training problems, so some of these changes can come from that alone. This test was after the gate and had no pass mark.
What it costs
The floor is not free. With the mask, the general part of the model writes all the text. At that time, the mask and the references together made the model a little worse at ordinary text. Its loss increased from 1.26 to 1.31 nats. Each experiment also costs one or two nights of GPU time.
Post one said that tool use works better than simulation in the weights, and each job given to exact code removed a failure. Post two said that the orchestrator must be dumb and easy to check. The one experiment that passed made the router a mask with no parameters, and it put the references in the text. The model now does only the part that no library does: it reads the language. Deterministic code is on all sides of it, and it shows me the references, the operator and whether the answer can fit. It cannot show me which phrase decided the operator.
What is next
The next experiments aim at the question:
- A role check in the floor. The floor can know which number is a total, a part, a rate or a change. It can reject an expression that subtracts a rate from a count.
- More real problems, which include GSM8K problems that need a constant such as "twice" or "half". The floor checks each expression.
- Different questions about the same story, which is one of the types of change in SVAMP.
I still owe you one argument from the last post: that a physics region will read the frozen maths core. I did not build the physics region, so I cannot make that argument yet. In each run, the frozen core matched its own hash at the end, bit for bit. That is the condition that the argument needs, and only that.
Comments
Post a Comment