The lazy answer is: bigger. More parameters pointed at a narrower problem. That answer quietly gives the whole argument back to the monolith — if depth is just scale, the monolith wins by having more of it, and my two posts collapse into an efficiency argument about compute budgets.
Depth isn't scale. Depth is owning the representation.
The habit nobody questions
Here is how audio has been fed to models for most of the last decade, and how a great deal of it still is. Take the waveform. Cut it into windows. Run a Short-Time Fourier Transform over each. Stack the results into a spectrogram. Hand the spectrogram — a picture — to a CNN or a transformer, which does what those architectures do to pictures.It works. That is the problem. It works well enough that nobody looks at what happened in step two.
Somebody, at design time, picked a window length. 25 milliseconds, probably, because that's what everyone picks. That single number decided what the model is allowed to know for the rest of its life, and it decided it before the model existed, for every input it will ever see, knowing nothing about what any of them would contain. The model doesn't get a vote. It's handed a picture and told the picture is the world.
The obvious objection is that this is dated — nobody feeds spectrograms anymore, the field moved to raw waveforms years ago. That's fair, and it's worth following. wav2vec and its descendants do take the waveform directly and learn a frontend instead of being handed one. Now look at what they learn it into: a convolutional stack that strides at roughly twenty milliseconds and emits something like fifty frames a second. The filters are learned. The stride is not. Someone still picked the receptive field and the hop, at design time, before the model existed, for every input it would ever see. Learning the filters changed who chose the number. It didn't give the model a vote.
So: we didn't give the model the Fourier transform. We used the Fourier transform on the model's behalf and gave it the receipt.
Give it the machinery, not the output
The alternative is the thing I keep coming back to: don't limit the submodel to consuming FFT output. Give it the frequency-domain mathematics as a capability it can invoke — the transform, the inverse, and the freedom to decide which side of the boundary a problem should be solved on. There's a real prize behind that door, the convolution theorem: what costs a brutal drag-and-integrate in the time domain is a pointwise multiply in the frequency domain, and even paying for the transform and its inverse that's n log n against n². But the efficiency was never the point. The point is who gets to decide.
Now look at what the window becomes. Not a constant baked in by whoever wrote the data loader, but a decision the model makes per problem, with the input in front of it. That matters because of the one thing no architecture escapes: the Gabor limit, the uncertainty principle wearing a signal-processing hat. Time resolution and frequency resolution trade against each other, always. Domain-shifting doesn't buy you both — nothing does. It buys the choice of which blindness to accept, made per problem rather than once, forever, by a default in a config file. A sharp transient and a sustained tone want opposite ends of that trade, and a system that can only offer them the same analysis is not deep. It's just large.
We already grant this privilege one level up. Code-interpreter tool use exists because a model needing an exact computation is better off writing the code and running it than approximating it in weights — the orchestrator chooses the computation instead of being handed the answer. Then we stopped, as if choosing were something only coordinators do. The submodel should have it too: not "here is your spectrogram," but "here is the signal and here is the mathematics; you decide."
I won't pretend I can hand you the implementation in a paragraph, but I should be precise about what I'm not saying, because the tool-use parallel invites the wrong reading. This is not a tool call in the LLM sense. No submodel is writing Python and shelling out to a DSP library mid-inference. The comparison to code-interpreter is about authority, not mechanism — about who decides, not about how the decision gets executed.
What I mean is narrower and more awkward: the transform, the inverse, and the window are differentiable operations inside the forward pass, and the parameters governing them — window length, hop, which side of the boundary to work on — are computed from the input rather than frozen in a config file. The submodel doesn't call out to anything. It reshapes what it's doing to the signal in front of it, while it's looking at it. Nothing shipping today does this — the operator networks that carry a transform inside the architecture still fix their modes in advance, and the learned frontends still freeze the hop — so take this as a direction, not a repo you can clone. Everything hard lives in what that authority actually costs on real hardware, and in how you would train something to exercise it — which is its own post, and the one I owe you next.
Biology already built this
In post one I asserted that transformers are bad at continuous signal, and asserted is the right word — I never showed the alternative done properly. The example is sitting inside your head.The cochlea is a frequency analyser made of geometry: a wave travels through the fluid, displaces the basilar membrane, and peaks at a place determined by frequency — high at the base, low further down at the apex. Elegant, and it has a cost. When a broadband sound enters your ear — a click, a hard consonant — the high frequencies stimulate their hair cells before the low frequencies have finished travelling. Everything arrived at once, and the ear smeared it across time. The hardware that gives you frequency resolution just destroyed your timing.
One structure in the brainstem exists to undo exactly that. Octopus cells, in the cochlear nucleus, reach thick dendrites clear across the array of auditory nerve fibres. On the accepted account the geometry is the fix: the fibres carrying the early-arriving high frequencies connect where the path to the cell body is longer, the late-arriving low frequencies where it's shorter. The delays cancel. Every component of that click crashes into the soma at the same instant, and the cell — built as a coincidence detector — fires once, with sub-millisecond precision, reconstructing the timing of the original wavefront.
I want to be careful not to oversell one cell, though, because the scoping is the interesting part. Octopus cells are a tiny piece of hearing, and they are not where pitch or timbre come from. The auditory nerve doesn't feed a clever cell — it fans out.
The same signal splits into parallel populations, each built differently. Bushy cells preserve timing so ferociously that the binaural nuclei downstream can compare your two ears. Stellate and chopper cells fire on their own regular clock, encoding the periodicity and spectral envelope that become pitch and timbre. Fusiform cells in the dorsal nucleus read the spectral notches that tell you a sound came from above. Pitch, timbre, onset, location — none of it lives in a cell. All of it lives in a collection of specialists, handed the same input, each deep in a different regime. "A click, to your left" is not one submodel's report. It's the assembled summary of several.
And notice what the octopus cell refuses to do: consume the representation it was handed. The cochlea gives it a smeared, frequency-dependent mess, and the cell inverts the smear rather than reasoning over it — it treats the encoding of its input as something to be corrected, not as the world. The geometry is fixed, so the cell doesn't choose that correction the way I want a model to; it was born with the answer. But it owns it. And that is the thing a frame-fed network never does. It takes its input's framing as ground truth, every time, because nobody ever gave it the means to do otherwise.
That the cell can't choose is the argument, incidentally, not a hole in it. Evolution could hardwire the correction because the octopus cell faces one cochlea, with one delay profile, for the whole of its life — a constant distortion deserves a constant fix, and paying for flexibility you'll never use is how you lose. Nothing in a data loader has earned that. A model faces arbitrary inputs from arbitrary sources, and the very reasoning that makes fixed geometry right for the cell is what makes fixed framing wrong for the model. Depth for the cell is ownership. Depth for us has to be ownership plus the right to re-choose.
The bill
One objection I can't dissolve, so here it is with my best answer, which isn't a refutation. The end-to-end objection from post one is stronger here. If domain choice matters, a big enough model learns it, and the bitter lesson has a long record of eating exactly this kind of hand-drawn structure. My answer is the same as before and no more comfortable: the transform isn't a heuristic to be learned around, it's a change of coordinates. Handing a system the coordinates it works in is not the same category of thing as handing it a prior — a prior is a guess about the answer, coordinates are where you do the arithmetic. Scale erodes guesses.But I have to concede the part that hurts, because it's sitting a few paragraphs up in my own post. Scale plainly can approximate a better basis from data — that is arguably what a learned frontend is doing, and I'd be a hypocrite to use wav2vec as evidence and then pretend it isn't. What it approximates it approximates inside a fixed graph, though: whatever basis it settles on, it settles on once, and applies to every input it will ever see. The thing scale hasn't shown is the other one — runtime authority to change coordinates mid-problem because the signal in front of it asked for different ones. That's the narrow claim, and it's narrower than the one I'd rather be making.
What the seam was for
None of this breaks the contract from post two — it's what makes the contract worth signing. Below the seam, the submodel can be as exotic as it likes: shift domains, run at its own clock, hold whatever representation the problem demands, change it mid-stream. Above the seam, none of that leaks. What crosses is still onset_detected, speaker_changed, confidence: 0.4. The orchestrator neither knows nor cares which domain the answer was computed in — and that only pays if the smart parts actually use the freedom the seam gives them.So: keep the coordinator dumb, and then make sure the parts underneath are deep in the only sense that matters. Not larger. Not more attentive. Sovereign over how they see the problem — including the right to see it a different way tomorrow.
A model handed a fixed frame rate is looking at a photograph of the signal that someone else framed. Give it the lens.
Comments
Post a Comment