Skip to main content

Posts

Showing posts from October, 2026

It learned a rate, not a rule

October 2026 · Post 6 in the series. Earlier: Keep the orchestrator dumb , What a dumb orchestrator actually looks like , What the smart parts are actually smart at , I built the parts. Here is what they said back , It read the story. Did it read the question? . More data did not help on SVAMP, the test that changes a story to catch a guess. A new question on an old story did. Then I made that change stronger, and the result showed what the model had learned. It had learned how often to leave a number out. It had not learned which number to leave out. The last post ended with three plans. The first, a role check in the floor, failed an offline test, so it did not receive a night on the graphics card. This post reports the other two plans and one more experiment. One passed its pass mark, and a second night on new seeds confirmed it. Two failed. The failure that taught me the most is the last one. How the system works The model reads a word problem from school maths and writes a...

It read the story. Did it read the question?

October 2026 · Post 5 in the series. Earlier: Keep the orchestrator dumb , What a dumb orchestrator actually looks like , What the smart parts are actually smart at , I built the parts. Here is what they said back . In the last post , the model had one job. It translates a school maths word problem into an expression, and exact code solves it. Since then I did five experiments with real word problems. Before each first run, I recorded its pass mark. One experiment passed. Essentially, this is the result. Two things made the model better: more exact code around it, and more real problems that people wrote. Three things did not: a judge trained on the model, the agreement of the model with itself, and a larger model. On SVAMP, a test set with questions written to catch this error, the error that remains is the question. The model reads the question, but without the question its first answer still keeps about 70% of its score. The limit of scale is the same as before: about eight mill...

I built the parts. Here is what they said back.

At the end of the last post I said I owed you something: what runtime authority over coordinates actually costs, and how you would train a model to use it. I've spent the time since building instead of writing - small language models, from under a million parameters to about eight million, trained on a desktop GPU or less, with each claim I could build at that size turned into an experiment whose pass mark I wrote down before the first run. This is the report. It is mostly a list of things that didn't work, and I've come to think the list is worth more than a win would have been, because each failure moved somewhere specific. I'll keep the scale caveat in front, where it belongs: everything below is small models, short training, synthetic data. It can falsify a mechanism. It can't vindicate an architecture. A note on units, because I'll use one: loss here is measured in nats , the model's average surprise per character. On text these models sit around 1.3 -...