# Surfing the Frontier Wave that rests between The Bitter Lesson and No Free Lunch

> No free lunch says specialization wins; the bitter lesson says scale does. Both are right, and the frontier where they intersect is where science accelerates.

Author: Scott Clark  
Published: 2026-07-24  
Series: Positions  
Canonical: https://talariasci.com/blog/no-free-lunch-bitter-lesson

---

Two famous ideas often get invoked to justify opposite strategies. The no-free-lunch theorems say that no method outperforms the rest unless its assumptions match the problem in front of it, and they get cited to argue that specialists always win (by going broad, you can't beat a method going deep). The bitter lesson says that scale (generalization) beats cleverness (specialization), and it gets cited to argue that you should stop designing bespoke models and features and just ride the scaling curve. Most people quote whichever one helps them win the argument they are trying to make. Both are right about the claims they actually make, and the frontier sits where they intersect. That intersection is good news if you do science for a living: the interesting research lives past the edge of what any engine has validated (out of distribution), and the engines keep making working at that edge easier (accelerate through distribution). Below I'll define both ideas precisely, walk through where each one applies, and lay out what the intersection means for where you should spend your own time and effort.

![Scale is a rising wave: each generation submerges more of what specialists built, but the wave advances less and performance falls as validated evidence thins.](https://talariasci.com/images/blog/no-free-lunch-bitter-lesson/fig1-wave.png)

**No free lunch** is a family of theorems [Wolpert and Macready proved in 1997](https://ieeexplore.ieee.org/document/585893). What the theorems prove: averaged over all possible problems, every optimizer performs identically, so any real advantage must come from assumptions matched to a restricted problem class (knowing and leveraging a prior). That motivates specialization, but it does not hand the specialist a guaranteed win. The intuition is: pound for pound, the specialist beats the generalist on its home turf. This is an empirical specialization claim, not a No Free Lunch corollary. An explicit example: at current model sizes, a generic LLM does not beat a classical numerical PDE solver on the solver's validated problem class out of the box, even if it can explain how the PDE works and the historical literature around it in natural language.

**The bitter lesson** is [Richard Sutton's 2019 essay](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) observing that, across 70 years of AI research, general methods that can soak up computation (search and learning) keep beating methods built from human knowledge of the domain. The intuition is: scale beats cleverness. It is not a theorem, and it is not a guarantee, but it is a historical pattern across the cases Sutton surveys, and it has held up time and time again in AI over the last decades.

**An ensemble** is a family of methods brought to solve a problem together. The simplest form is a router: something decides which method to trust on each piece of the problem, finding the right tool for the job, chosen per job. The stronger form is compositional: the methods exchange information and constrain and learn from each other as they work. The distinction is important in the harness argument below.

## What no free lunch says

The statement gets overclaimed routinely, so it's worth getting the precise version right. Take a finite search space and a finite set of possible objective values, weight all possible objective functions uniformly, and consider any optimizer that never revisits a point. Averaged over that space of problems, every such optimizer sees exactly the same distribution of results for a fixed evaluation budget. Your clever algorithm, my clever algorithm, and random search without replacement come out identical. Under that uniform average of all possible objective functions, any advantage on one class of problems is paid for on the complement. Loosely put: if you are good at one thing you are probably bad at another, and if you are ok at everything you pay for that by not being great at any specific thing.

The theorem does not say a specialized tool always wins in practice, because real problems aren't drawn uniformly from the space of all functions. What it says is narrower and more useful: outperformance has to come from somewhere, and the only place available is the match between your method's assumptions and the actual distribution of problems you face. If your method beats random search, it's because its priors line up with your problem class. **The gain lives in the match, not in the method.**

I built a company on robust black box optimization methods, so this theorem sat underneath everything we did. The work was never about trying to beat the No Free Lunch theorem; it was about closing the efficiency gaps it allows. On the problems SigOpt targeted (expensive evaluations, moderate effective dimensionality, response surfaces well-enough-behaved for certain kernels) there is a large gap between random search and naive Bayesian Optimization (BO), and another between naive BO and clever BO, and closing those gaps was the business. Note which part is which: the expense of the evaluations is the reason to reach for BO at all; the smoothness is the assumption a GP model makes, etc. When the assumptions held, the method was sample-efficient. Where it weakened (very high-dimensional, discontinuous, or heavily categorical problems) the advantage narrowed, and occasionally a customer brought a problem where other methods, including genetic algorithms, were competitive under the same evaluation budget. The gain lived in finding the match for each problem and a router that could find these matches. The limits were when that still couldn't beat the custom method for that specific problem.

## What the bitter lesson says

In 2019 Richard Sutton wrote down the pattern he'd watched repeat for decades: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." The general methods he means are search and learning, the two that keep scaling as compute gets cheaper with Moore's and Huang's Laws.

History mostly backs him, though the details are sometimes messier than the retellings. Deep Blue is remembered as brute-force search beating grandmaster knowledge, but [the team's own account](https://doi.org/10.1016/S0004-3702(01)00129-1) describes chess-specific hardware, hand-tuned evaluation features, and grandmaster consulting: a specialized structure wrapped around massive search. Hand-engineered vision features like [SIFT](https://doi.org/10.1023/B:VISI.0000029664.99615.94) were displaced from the leading image-classification benchmarks by convnets once data, GPUs, and training technique caught up, a trajectory that took years. And in 2017, AlphaGo Zero trained without human game data (self-play under the supplied rules of Go) and defeated AlphaGo Lee, the version that had beaten Lee Sedol, a hundred games to nothing ([Silver et al., 2017](https://www.nature.com/articles/nature24270)). The direction is real, but the bitter part is personal: careers of carefully built-in human knowledge, outrun eventually by whoever bet on the scaling curve.

Each idea also has a lazy reading. The lazy reading of Sutton: design is obsolete, wait for the next model, there is no hope in trying. The lazy reading of No Free Lunch: a specialized tool, once built, is safe indefinitely, there is no hope in competing. Both readings are comfortable, and both dissolve once the two claims are stated carefully enough to compare.

## Where each one applies

Stated carefully, the two ideas are not even in tension, because they do not run on the same axis. No free lunch is a distributional result: sustained outperformance requires assumptions matched to a non-uniform problem class. The bitter lesson is a historical observation about which kinds of methods kept improving as computation got cheap: in the cases Sutton surveys, the general methods walked past the specialists, again and again, once there was *enough data and compute* to spend. Abundant compute that unlocked massive scale did not repeal the theorem, and the theorem does not cap the scaling curve. What compute changed is where the required match comes from, and in areas where data is scarce there are still lessons to be learned.

When compute is scarce, you get one bet, so you encode everything you know about the problem into it: the features, the heuristics, the grandmasters in the room. In board games the expensive part was never checking the outcome (legal moves and terminal results cost almost nothing to compute) so the cost lived in encoding the knowledge and tactics, and under those budgets encoding was the right play.

Science is not a board game, though: the evaluation of a hypothesis is often the dominant cost. When compute is abundant, you no longer have to manually encode the domain. You can train over broad representations and let data select from a hypothesis family no human could enumerate: searched architectures, learned features, training runs and the long tail of post-training after them. Representation, objectives, curation, and validation all remain part of the method; the hand-encoding is the part that shrinks. A large trained model is the outcome of that paid-up search: one hypothesis selected from an enormous family, under the fixed biases of an architecture, an objective (or set of evals to overfit to), and an optimizer, with the data and the training objective doing the picking. It can express many task-conditional behaviors, which is what makes it feel general. Its assumptions did not vanish; they moved into the architecture and the training recipe.

The lunch still isn't free, scale just buys more of the menu. The match between assumptions and problems still has to come from somewhere; what changed is how. We used to pay in human priors, one method at a time. Now we mostly pay in compute and data, wholesale. And the wave keeps rising wherever compute and in-distribution data keep scaling.

![The tide coming in: each model generation raises the water, t1 to t2 to t3.](https://talariasci.com/images/blog/no-free-lunch-bitter-lesson/fig1-wave-animated.gif)

*The tide coming in: each model generation raises the water, t1 to t2 to t3.*

The picture for this is a wave. Problems run left to right by how much evidence exists for ones like them (the axis is anchored to a fixed snapshot of the evidence, so a problem keeps its place while the curves above it move), and every gray peak is a specialized method, each buying height with narrowness (specificity). The blue curves are successive generations of the general engine, and each generation raises the water: the small peaks on the left have been under water since t2, the taller one held out until t3, and Sutton's seventy years are this tide coming in. The right side of the picture is where it gets interesting for a researcher: where the validated evidence thins, the wave advances less and the waterline falls, and the specialized solver out in the band still stands above it. Nothing in the picture says the wave stops rising. It says the wave rises where the data is.

The precise reconciliation, then: sustained outperformance requires assumptions matched to your actual problem distribution, and scale is the most successful way the field has yet found to manufacture that match wherever the data exists to learn it from. Scale buys generality over a distribution. It can even help generate evidence where a valid signal exists: synthetic data, simulation, self-play, automated experimentation. What it cannot do is determine behavior in a regime with little evidence and no assumptions connecting that regime to the ones observed. To be precise, **out of distribution** here means outside the regime where the engine's performance has been trained and validated, because the mechanism shifted, the inputs shifted, or the evidence was never collected. Out of distribution is precisely where research lives.

## Where the engine runs out

That paid-up search is only as good as whatever data is in the evidence. Where evidence is dense, training has usually selected the right behavior for you already, which is why a benchmark-shaped question feels free: the match was purchased years ago and amortized over everyone. Out of distribution, the engine doesn't refuse to answer, and a refusal would almost be safer. This leads to spiky intelligence: it can disprove a decades old conjecture by putting together known math in other domains, but it still tells you to walk home from the carwash.

Take a question from the working end of computational science: how a particular numerical method degrades in a physical regime well past anything in the published record or data. Even a model with broad literature coverage has only a thin file of evidence there, and thin evidence doesn't stop a model from answering. It produces a plausible extrapolation from adjacent regimes, fluent and formatted, and the fluency is not a calibration signal. The answer came from the model's learned distribution over what answers look like, not from evidence about your regime, and nothing external has validated it. That sets up the failure mode: accept it as a baseline, and every downstream comparison inherits it. Weeks later, what reads as an improvement can be a measurement of the baseline being wrong, with nothing in the pipeline positioned to notice. In a chat window, this failure mode looks like a hallucination. In a research campaign, it compounds into bad science.

So the bet is not a specialized model; it's a specialized structure around a general model, where the structure knows when and how to call the engine and when and how to call a specialized method. Call that structure a harness: the fixed apparatus around the engine that decides what gets asked, what gets checked, what tools get called, and what gets believed. The point of the whole apparatus fits in one sentence: give the general model the tools to learn what it does not know, and the physics guardrails (or your domain's equivalent) so that nothing unsupported enters the record unlabeled; every output gets verified, rejected, or explicitly marked as an abstention before anything downstream depends on it. A model can always generate a guess; the guarantee lives in what the system accepts. For a proposed numerical result, acceptance looks like this: run a numerical solver at multiple resolutions, check units and invariants, require convergence, compare against a declared baseline, and keep the provenance of every step.

## More than the best of its parts

An engine plus a specialist method, held in a harness, is an ensemble, and the simplest way to run one is as a router: send each problem to whichever method is better there. Then collect, problem by problem, the best of the parts, the upper envelope of the two methods, assuming the router always knows which one to pick. A real router approaches that envelope only as far as its selection signal can predict, on problems it has not seen, which method to use, and earning that signal is a large part of what a harness is for.

But the max of the methods is not the ceiling, because a router is not the only ensemble. Run the members compositionally (the engine proposes where the specialist is rigid, the specialist constrains where the engine would guess, each catches errors the other cannot see) and the combination can do better than either member does alone. That is the operational meaning of "approaching the problem from different directions": each member contributes information the other does not have, and their failure modes do not fully overlap if they are even slightly orthogonal. An LLM that knows how to use a solver well can be stronger than the LLM or the solver alone. The existence proofs are already published: [FunSearch](https://www.nature.com/articles/s41586-023-06924-6) paired a language-model proposer with an automatic evaluator and produced new mathematical constructions, [AlphaProof and AlphaGeometry](https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/) pair one with a symbolic verifier and reach olympiad-level validated results, and [AlphaGo](https://www.nature.com/articles/nature16961) itself was networks plus tree search, stronger than either component alone. The fence is worth drawing honestly: the gain is conditional, not automatic. It has to be validated on held-out problems, it shrinks as the methods' errors grow redundant, and it closes to zero where one method adds nothing after conditioning on the other (ie running the same deterministic method twice). It is also a capability claim rather than an efficiency claim: the ensemble spends the engine plus the specialist plus the orchestration between them, so whether it also wins pound for pound is an empirical question under a matched budget.

![A harness does more than pick: where the engine and the specialist approach from different directions, the ensemble can exceed the best of its parts.](https://talariasci.com/images/blog/no-free-lunch-bitter-lesson/fig2-ensemble.png)

In the figure, the dashed line is the oracle upper envelope of these two components: the best a perfect router could collect from these two methods, problem by problem. It rides the engine where the engine wins, kinks at the crossover, and rides the specialist where the specialist wins. The solid curve above it is the harnessed ensemble. The sketch assumes the members have nearly nothing to teach each other at the edges, so the ensemble sits near the envelope there; near the crossover, where both members carry non-redundant information, it rises above. Error redundancy is a condition of the claim, not something this axis displays, and the curve illustrates the claim rather than evidencing it.

The gap near the crossover is what you gain from a well designed harness over both methods. It is the application that lives on the compute substrate of the methods it harnesses.

## Where the specialization lives

The harness carries at least three kinds of structure, and none of it has to live in weights; parts can be learned, a router or a calibration model, without changing the argument. The first is what a valid experiment is in this domain: the units, the invariants, the convergence criteria, the checks a good referee would run. The second is routing: which calls are in-distribution for the engine (write the plotting code, summarize the literature) and which need verification first (propose the baseline, interpret the anomaly). The third is what a claim has to survive before it counts as a result: comparisons declared before the evaluation when the claim is strong enough to warrant it, honest statistics, replication. Exploratory work runs under lighter confirmatory gates (it's confirmation that has to be earned) but provenance, physical constraints, and the speculative label never come off. The scientist pushes out of distribution using specialized methods as guardrails and the foundational model helps fill in behind.

None of this re-derives what the engine already knows. All of it is selection, constraint, and validation: the part of the match the data hasn't paid for yet, supplied from outside. That's where no-free-lunch specialization went: out of the weights and into the protocol.

## Maybe scale eats the harness too

The strong version of the pure-scale position: everything I just described is scaffolding, and Sutton's essay is a list of scaffolding that got eaten. Every year the frontier of "needs a specialist" recedes; questions that needed careful tooling in 2023 are chat-window questions now. A consistent Suttonian would say the harness is another layer of manually engineered domain structure, and its turn is coming. If you are in grad school right now, watching models prove theorems left and right on twitter between problem sets, this is the version of the argument that keeps you up at night: why spend five years becoming a specialist if the wave is coming for specialists?

I'm taking the other side for science in particular. Some of this scaffolding will get absorbed; probably a lot of it over a long enough time horizon. But I believe there will always be a frontier, by definition. Whatever is on the edge of being absorbed is still out of distribution, unknown, and worth exploring. As the models absorb more the frontier moves too. Which is why the time becoming an expert is not wasted: the specialist who explores and works at the moving edge gets to the interesting questions faster than anyone and stays there.

And even if the wave rises faster than I expect, there is a reason to play anyway, and it's the part that excites me most. Accelerating scientific discovery is worth doing whether the acceleration ends up living in the model or in the harness. If a harness makes science faster and more effective for a decade and then gets absorbed, the discoveries stay discovered, and the abundance they bought does not get returned. The integral of abundance is maximized. The harness can lose in the very long run and the bet still pays, because at the end of the day more science, sooner, is the prize either way.

## The bet

This thesis is what Talaria Scientific is built on: a harness for computational science that carries the domain's structure (convergence checks, physically grounded baselines, statistical gates, provenance) and treats the general models as engines that keep improving underneath it, so the scientist stays on ideas and taste. How the harness estimates its routing signals, draws its out-of-distribution boundaries, and measures error redundancy is its own series of design notes, and that series is on its way.

If you're deciding where to spend your own time and effort, the posture falls out directly. Where the engine is validated for your accuracy, cost, and reliability bar, don't fight it; any part of your tool that only compensates for weak general capability is on the engine's menu next (the classic ChatGPT wrappers of the world). Out of distribution, your cleverness is the match, and it is more than verification: spend it first on the ideas and insights nobody has published, the experiments nobody has run, and then on the objective, the baselines, and the selection signal that lets you trust what comes back. The checks exist to serve and accelerate the ideas, not the other way around.

Pushing into territory no one has validated, with better instruments than the generation before, is what science has always been: the mirrors, then the computers, and now the engines each took over the part they could hold so the questions could move further out, the flywheel could spin faster. The promise of abundance sitting behind all the AI noise (the new materials, the cheaper energy, the medicine nobody has designed yet) does not arrive on its own; it arrives one validated result at a time, pushed by people working past the edge of the record, and that mantle is on us. The difference is the pace we can push at: with the schlep in the harness and the record keeping itself, the flywheel from idea to trusted result to next idea spins faster than it ever has. If you are the one doing the pushing, the wave is not your competition, it is a surfing opportunity.

---

Source: https://talariasci.com/blog/no-free-lunch-bitter-lesson (Talaria Scientific)