Talaria Scientific Blog

Surfing the Frontier Wave that rests between The Bitter Lesson and No Free Lunch

Fri, 24 Jul 2026 00:00:00 GMT

Two famous ideas often get invoked to justify opposite strategies. The no-free-lunch theorems say that no method outperforms the rest unless its assumptions match the problem in front of it, and they get cited to argue that specialists always win (by going broad, you can't beat a method going deep). The bitter lesson says that scale (generalization) beats cleverness (specialization), and it gets cited to argue that you should stop designing bespoke models and features and just ride the scaling curve. Most people quote whichever one helps them win the argument they are trying to make. Both are right about the claims they actually make, and the frontier sits where they intersect. That intersection is good news if you do science for a living: the interesting research lives past the edge of what any engine has validated (out of distribution), and the engines keep making working at that edge easier (accelerate through distribution). Below I'll define both ideas precisely, walk through where each one applies, and lay out what the intersection means for where you should spend your own time and effort.

Scale is a rising wave: each generation submerges more of what specialists built, but the wave advances less and performance falls as validated evidence thins.

No free lunch is a family of theorems Wolpert and Macready proved in 1997. What the theorems prove: averaged over all possible problems, every optimizer performs identically, so any real advantage must come from assumptions matched to a restricted problem class (knowing and leveraging a prior). That motivates specialization, but it does not hand the specialist a guaranteed win. The intuition is: pound for pound, the specialist beats the generalist on its home turf. This is an empirical specialization claim, not a No Free Lunch corollary. An explicit example: at current model sizes, a generic LLM does not beat a classical numerical PDE solver on the solver's validated problem class out of the box, even if it can explain how the PDE works and the historical literature around it in natural language.

The bitter lesson is Richard Sutton's 2019 essay observing that, across 70 years of AI research, general methods that can soak up computation (search and learning) keep beating methods built from human knowledge of the domain. The intuition is: scale beats cleverness. It is not a theorem, and it is not a guarantee, but it is a historical pattern across the cases Sutton surveys, and it has held up time and time again in AI over the last decades.

An ensemble is a family of methods brought to solve a problem together. The simplest form is a router: something decides which method to trust on each piece of the problem, finding the right tool for the job, chosen per job. The stronger form is compositional: the methods exchange information and constrain and learn from each other as they work. The distinction is important in the harness argument below.

What no free lunch says

The statement gets overclaimed routinely, so it's worth getting the precise version right. Take a finite search space and a finite set of possible objective values, weight all possible objective functions uniformly, and consider any optimizer that never revisits a point. Averaged over that space of problems, every such optimizer sees exactly the same distribution of results for a fixed evaluation budget. Your clever algorithm, my clever algorithm, and random search without replacement come out identical. Under that uniform average of all possible objective functions, any advantage on one class of problems is paid for on the complement. Loosely put: if you are good at one thing you are probably bad at another, and if you are ok at everything you pay for that by not being great at any specific thing.

The theorem does not say a specialized tool always wins in practice, because real problems aren't drawn uniformly from the space of all functions. What it says is narrower and more useful: outperformance has to come from somewhere, and the only place available is the match between your method's assumptions and the actual distribution of problems you face. If your method beats random search, it's because its priors line up with your problem class. The gain lives in the match, not in the method.

I built a company on robust black box optimization methods, so this theorem sat underneath everything we did. The work was never about trying to beat the No Free Lunch theorem; it was about closing the efficiency gaps it allows. On the problems SigOpt targeted (expensive evaluations, moderate effective dimensionality, response surfaces well-enough-behaved for certain kernels) there is a large gap between random search and naive Bayesian Optimization (BO), and another between naive BO and clever BO, and closing those gaps was the business. Note which part is which: the expense of the evaluations is the reason to reach for BO at all; the smoothness is the assumption a GP model makes, etc. When the assumptions held, the method was sample-efficient. Where it weakened (very high-dimensional, discontinuous, or heavily categorical problems) the advantage narrowed, and occasionally a customer brought a problem where other methods, including genetic algorithms, were competitive under the same evaluation budget. The gain lived in finding the match for each problem and a router that could find these matches. The limits were when that still couldn't beat the custom method for that specific problem.

What the bitter lesson says

In 2019 Richard Sutton wrote down the pattern he'd watched repeat for decades: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." The general methods he means are search and learning, the two that keep scaling as compute gets cheaper with Moore's and Huang's Laws.

History mostly backs him, though the details are sometimes messier than the retellings. Deep Blue is remembered as brute-force search beating grandmaster knowledge, but the team's own account describes chess-specific hardware, hand-tuned evaluation features, and grandmaster consulting: a specialized structure wrapped around massive search. Hand-engineered vision features like SIFT were displaced from the leading image-classification benchmarks by convnets once data, GPUs, and training technique caught up, a trajectory that took years. And in 2017, AlphaGo Zero trained without human game data (self-play under the supplied rules of Go) and defeated AlphaGo Lee, the version that had beaten Lee Sedol, a hundred games to nothing (Silver et al., 2017). The direction is real, but the bitter part is personal: careers of carefully built-in human knowledge, outrun eventually by whoever bet on the scaling curve.

Each idea also has a lazy reading. The lazy reading of Sutton: design is obsolete, wait for the next model, there is no hope in trying. The lazy reading of No Free Lunch: a specialized tool, once built, is safe indefinitely, there is no hope in competing. Both readings are comfortable, and both dissolve once the two claims are stated carefully enough to compare.

Where each one applies

Stated carefully, the two ideas are not even in tension, because they do not run on the same axis. No free lunch is a distributional result: sustained outperformance requires assumptions matched to a non-uniform problem class. The bitter lesson is a historical observation about which kinds of methods kept improving as computation got cheap: in the cases Sutton surveys, the general methods walked past the specialists, again and again, once there was enough data and compute to spend. Abundant compute that unlocked massive scale did not repeal the theorem, and the theorem does not cap the scaling curve. What compute changed is where the required match comes from, and in areas where data is scarce there are still lessons to be learned.

When compute is scarce, you get one bet, so you encode everything you know about the problem into it: the features, the heuristics, the grandmasters in the room. In board games the expensive part was never checking the outcome (legal moves and terminal results cost almost nothing to compute) so the cost lived in encoding the knowledge and tactics, and under those budgets encoding was the right play.

Science is not a board game, though: the evaluation of a hypothesis is often the dominant cost. When compute is abundant, you no longer have to manually encode the domain. You can train over broad representations and let data select from a hypothesis family no human could enumerate: searched architectures, learned features, training runs and the long tail of post-training after them. Representation, objectives, curation, and validation all remain part of the method; the hand-encoding is the part that shrinks. A large trained model is the outcome of that paid-up search: one hypothesis selected from an enormous family, under the fixed biases of an architecture, an objective (or set of evals to overfit to), and an optimizer, with the data and the training objective doing the picking. It can express many task-conditional behaviors, which is what makes it feel general. Its assumptions did not vanish; they moved into the architecture and the training recipe.

The lunch still isn't free, scale just buys more of the menu. The match between assumptions and problems still has to come from somewhere; what changed is how. We used to pay in human priors, one method at a time. Now we mostly pay in compute and data, wholesale. And the wave keeps rising wherever compute and in-distribution data keep scaling.

The tide coming in: each model generation raises the water, t1 to t2 to t3.

The picture for this is a wave. Problems run left to right by how much evidence exists for ones like them (the axis is anchored to a fixed snapshot of the evidence, so a problem keeps its place while the curves above it move), and every gray peak is a specialized method, each buying height with narrowness (specificity). The blue curves are successive generations of the general engine, and each generation raises the water: the small peaks on the left have been under water since t2, the taller one held out until t3, and Sutton's seventy years are this tide coming in. The right side of the picture is where it gets interesting for a researcher: where the validated evidence thins, the wave advances less and the waterline falls, and the specialized solver out in the band still stands above it. Nothing in the picture says the wave stops rising. It says the wave rises where the data is.

The precise reconciliation, then: sustained outperformance requires assumptions matched to your actual problem distribution, and scale is the most successful way the field has yet found to manufacture that match wherever the data exists to learn it from. Scale buys generality over a distribution. It can even help generate evidence where a valid signal exists: synthetic data, simulation, self-play, automated experimentation. What it cannot do is determine behavior in a regime with little evidence and no assumptions connecting that regime to the ones observed. To be precise, out of distribution here means outside the regime where the engine's performance has been trained and validated, because the mechanism shifted, the inputs shifted, or the evidence was never collected. Out of distribution is precisely where research lives.

Where the engine runs out

That paid-up search is only as good as whatever data is in the evidence. Where evidence is dense, training has usually selected the right behavior for you already, which is why a benchmark-shaped question feels free: the match was purchased years ago and amortized over everyone. Out of distribution, the engine doesn't refuse to answer, and a refusal would almost be safer. This leads to spiky intelligence: it can disprove a decades old conjecture by putting together known math in other domains, but it still tells you to walk home from the carwash.

Take a question from the working end of computational science: how a particular numerical method degrades in a physical regime well past anything in the published record or data. Even a model with broad literature coverage has only a thin file of evidence there, and thin evidence doesn't stop a model from answering. It produces a plausible extrapolation from adjacent regimes, fluent and formatted, and the fluency is not a calibration signal. The answer came from the model's learned distribution over what answers look like, not from evidence about your regime, and nothing external has validated it. That sets up the failure mode: accept it as a baseline, and every downstream comparison inherits it. Weeks later, what reads as an improvement can be a measurement of the baseline being wrong, with nothing in the pipeline positioned to notice. In a chat window, this failure mode looks like a hallucination. In a research campaign, it compounds into bad science.

So the bet is not a specialized model; it's a specialized structure around a general model, where the structure knows when and how to call the engine and when and how to call a specialized method. Call that structure a harness: the fixed apparatus around the engine that decides what gets asked, what gets checked, what tools get called, and what gets believed. The point of the whole apparatus fits in one sentence: give the general model the tools to learn what it does not know, and the physics guardrails (or your domain's equivalent) so that nothing unsupported enters the record unlabeled; every output gets verified, rejected, or explicitly marked as an abstention before anything downstream depends on it. A model can always generate a guess; the guarantee lives in what the system accepts. For a proposed numerical result, acceptance looks like this: run a numerical solver at multiple resolutions, check units and invariants, require convergence, compare against a declared baseline, and keep the provenance of every step.

More than the best of its parts

An engine plus a specialist method, held in a harness, is an ensemble, and the simplest way to run one is as a router: send each problem to whichever method is better there. Then collect, problem by problem, the best of the parts, the upper envelope of the two methods, assuming the router always knows which one to pick. A real router approaches that envelope only as far as its selection signal can predict, on problems it has not seen, which method to use, and earning that signal is a large part of what a harness is for.

But the max of the methods is not the ceiling, because a router is not the only ensemble. Run the members compositionally (the engine proposes where the specialist is rigid, the specialist constrains where the engine would guess, each catches errors the other cannot see) and the combination can do better than either member does alone. That is the operational meaning of "approaching the problem from different directions": each member contributes information the other does not have, and their failure modes do not fully overlap if they are even slightly orthogonal. An LLM that knows how to use a solver well can be stronger than the LLM or the solver alone. The existence proofs are already published: FunSearch paired a language-model proposer with an automatic evaluator and produced new mathematical constructions, AlphaProof and AlphaGeometry pair one with a symbolic verifier and reach olympiad-level validated results, and AlphaGo itself was networks plus tree search, stronger than either component alone. The fence is worth drawing honestly: the gain is conditional, not automatic. It has to be validated on held-out problems, it shrinks as the methods' errors grow redundant, and it closes to zero where one method adds nothing after conditioning on the other (ie running the same deterministic method twice). It is also a capability claim rather than an efficiency claim: the ensemble spends the engine plus the specialist plus the orchestration between them, so whether it also wins pound for pound is an empirical question under a matched budget.

A harness does more than pick: where the engine and the specialist approach from different directions, the ensemble can exceed the best of its parts.

In the figure, the dashed line is the oracle upper envelope of these two components: the best a perfect router could collect from these two methods, problem by problem. It rides the engine where the engine wins, kinks at the crossover, and rides the specialist where the specialist wins. The solid curve above it is the harnessed ensemble. The sketch assumes the members have nearly nothing to teach each other at the edges, so the ensemble sits near the envelope there; near the crossover, where both members carry non-redundant information, it rises above. Error redundancy is a condition of the claim, not something this axis displays, and the curve illustrates the claim rather than evidencing it.

The gap near the crossover is what you gain from a well designed harness over both methods. It is the application that lives on the compute substrate of the methods it harnesses.

Where the specialization lives

The harness carries at least three kinds of structure, and none of it has to live in weights; parts can be learned, a router or a calibration model, without changing the argument. The first is what a valid experiment is in this domain: the units, the invariants, the convergence criteria, the checks a good referee would run. The second is routing: which calls are in-distribution for the engine (write the plotting code, summarize the literature) and which need verification first (propose the baseline, interpret the anomaly). The third is what a claim has to survive before it counts as a result: comparisons declared before the evaluation when the claim is strong enough to warrant it, honest statistics, replication. Exploratory work runs under lighter confirmatory gates (it's confirmation that has to be earned) but provenance, physical constraints, and the speculative label never come off. The scientist pushes out of distribution using specialized methods as guardrails and the foundational model helps fill in behind.

None of this re-derives what the engine already knows. All of it is selection, constraint, and validation: the part of the match the data hasn't paid for yet, supplied from outside. That's where no-free-lunch specialization went: out of the weights and into the protocol.

Maybe scale eats the harness too

The strong version of the pure-scale position: everything I just described is scaffolding, and Sutton's essay is a list of scaffolding that got eaten. Every year the frontier of "needs a specialist" recedes; questions that needed careful tooling in 2023 are chat-window questions now. A consistent Suttonian would say the harness is another layer of manually engineered domain structure, and its turn is coming. If you are in grad school right now, watching models prove theorems left and right on twitter between problem sets, this is the version of the argument that keeps you up at night: why spend five years becoming a specialist if the wave is coming for specialists?

I'm taking the other side for science in particular. Some of this scaffolding will get absorbed; probably a lot of it over a long enough time horizon. But I believe there will always be a frontier, by definition. Whatever is on the edge of being absorbed is still out of distribution, unknown, and worth exploring. As the models absorb more the frontier moves too. Which is why the time becoming an expert is not wasted: the specialist who explores and works at the moving edge gets to the interesting questions faster than anyone and stays there.

And even if the wave rises faster than I expect, there is a reason to play anyway, and it's the part that excites me most. Accelerating scientific discovery is worth doing whether the acceleration ends up living in the model or in the harness. If a harness makes science faster and more effective for a decade and then gets absorbed, the discoveries stay discovered, and the abundance they bought does not get returned. The integral of abundance is maximized. The harness can lose in the very long run and the bet still pays, because at the end of the day more science, sooner, is the prize either way.

The bet

This thesis is what Talaria Scientific is built on: a harness for computational science that carries the domain's structure (convergence checks, physically grounded baselines, statistical gates, provenance) and treats the general models as engines that keep improving underneath it, so the scientist stays on ideas and taste. How the harness estimates its routing signals, draws its out-of-distribution boundaries, and measures error redundancy is its own series of design notes, and that series is on its way.

If you're deciding where to spend your own time and effort, the posture falls out directly. Where the engine is validated for your accuracy, cost, and reliability bar, don't fight it; any part of your tool that only compensates for weak general capability is on the engine's menu next (the classic ChatGPT wrappers of the world). Out of distribution, your cleverness is the match, and it is more than verification: spend it first on the ideas and insights nobody has published, the experiments nobody has run, and then on the objective, the baselines, and the selection signal that lets you trust what comes back. The checks exist to serve and accelerate the ideas, not the other way around.

Pushing into territory no one has validated, with better instruments than the generation before, is what science has always been: the mirrors, then the computers, and now the engines each took over the part they could hold so the questions could move further out, the flywheel could spin faster. The promise of abundance sitting behind all the AI noise (the new materials, the cheaper energy, the medicine nobody has designed yet) does not arrive on its own; it arrives one validated result at a time, pushed by people working past the edge of the record, and that mantle is on us. The difference is the pace we can push at: with the schlep in the harness and the record keeping itself, the flywheel from idea to trusted result to next idea spins faster than it ever has. If you are the one doing the pushing, the wave is not your competition, it is a surfing opportunity.

A system of record for computational science

Fri, 24 Jul 2026 00:00:00 GMT

Talaria keeps a system of record underneath your research: every idea, run, result, and decision, kept with its provenance, whether or not it worked out. We have argued (Why I'm building Talaria Scientific) that validation, not ideas, bottlenecks computational science, and that a harness can take the schlep without taking the science. This piece explains the memory systems that make that safe: how the record is structured, and why each structural choice was made, failure mode by failure mode.

The cold restart problem

If you put a project down in May and pick it up in September the first week or more back goes to archaeology. What did I try last? Which variants actually ran? Which configuration produced the plot in the draft? Why was the second baseline abandoned? Did the "good" result predate or postdate the preprocessing fix? This is compounded if you hand the project to a collaborator, or a new student, or a skeptical reviewer.

The research often moves faster than the record keeping, every interruption resets the context, and the incentives all point at the next result, not the receipts of the last one. A lab notebook only works if the scientist keeps it up to date, and there are better uses for attention. So the record slowly diverges from the work, and everything downstream of the record (reproduction, collaboration, review) inherits the gaps.

The system of record we have built inverts the lab notebook ritual: a record the work keeps of itself, instead of one the scientist has to remember to keep. When the harness runs your validation, the bookkeeping is not a separate chore that competes with the science; it is a side effect of the machinery doing the work. We have built the shape that record should take in a very opinionated way, which we explain below.

The shape

At the top of the system of record sits a citation directed acyclic graph (DAG): the literature you build on, linked so every claim traces to its source chronologically and by reference. The graph is directed because each reference cites others and acyclic because, in the record, you can't cite the future. Below it grows a triad of hypothesis, results, and findings DAGs. A hypothesis graph holds what you're explicitly asking for each experiment/run. A results graph holds what actually ran, each result carrying the exact code, configuration, and environment it ran with; exact, because that is what makes a rerun a reproduction instead of a reenactment. A findings graph holds what you now believe, an updated posterior from the results, each finding pinned to what led to it. These graphs are directed because each new entry comes from what was learned before and acyclic because that is how the scientific process behaves by construction; you hypothesize, test, and then learn. Overshadowing all of these DAGs is a concept graph: the living map of definitions and connections the other graphs share, information about the field, baselines, definitions, statistics, and more. Time flows down the page, the references above the ideas, the work growing beneath it.

The system of record: references above, the hypothesis-results-findings triad below, concepts overshadowing all of it. Every result answers a hypothesis and findings carry their evidence with them.

The layers are separated intentionally; what you read, what you ask, what you run, and what you believe are different kinds of claims with different lifetimes and provenance. A system of record should not blur them (ie a notebook page where a citation, a hunch, a number, and a conclusion share a line) because it can't answer the questions that matter later: what evidence does this belief rest on, and what exactly produced that evidence? In the record, those questions are edges you follow on a graph, not memories you reconstruct.

Five design decisions, and the failure mode each one kills

This structure was built in an opinionated way, and each is a design choice aimed at a specific way research bookkeeping can go wrong.

An untested hypothesis is first-class. Parked ideas aren't clutter, they're your research agenda: the queue you, a student, or the harness can pick up next. The failure mode this kills is the one the manifesto (Why I'm building Talaria Scientific) opened with: the reviewer suggestion, the seminar question, and the shower thought all evaporating because nothing held them. In the record they wait, visibly unpaired with results, until someone (or the harness, within its budget) picks them up.

A result that answers no hypothesis is not allowed to exist. Orphan numbers are how spurious findings start: fifty configurations ran, one "worked," and nobody remembers what question it was answering. Forcing every result to name its hypothesis is the structural version of pre-registration: entries are timestamped and append-only, so whether a hypothesis preceded its results is a fact of the record, not a claim in the paper. That is what makes "we tried 50 things and report the best one" visible in the record instead of invisible in a results folder.

Superseded ideas get marked, never deleted. Research doesn't evolve cleanly, it morphs and branches, and a record that only keeps the current story is the polished paper's hindsight trail. Keeping the whole history means you can revisit abandoned paths when a new tool makes them viable, hand a collaborator what actually happened instead of a reconstruction, and even publish the roads not taken.

Overrides are journaled. The scientist can force past an integrity gate (sometimes you should; a hunch can lead to a breakthrough), but the override itself is recorded along with the exact checks it waived. The failure mode this kills is silent rule-bending: the gap between "we followed the protocol" and what happened, which no one intended and no one can find later. The record stays honest even where you overruled it.

The record self-heals before you publish. Mid-flight mess is tolerated on purpose, because rigor that blocks serendipity would be its own failure; science sometimes lurches forward by stumbling into things. Claims are gated by strength, never existence: a lucky observation enters the record labeled as what it is, and earns its confirmation later. When you are done, the harness walks the record with you and heals what's incomplete, and the heal itself is journaled like any override, so a repaired record never masquerades as an original one. Every result you plan to publish is one you can trust and reproduce exactly.

What standard techniques miss

Each standard tool holds a slice of this. The lab notebook (paper or electronic) holds the narrative, but it is only as complete as the scientist's discipline on their worst week, and it isn't linked to the artifacts: the notebook says what you meant to run, not what ran. The wiki or shared doc holds the write-up, prose about runs rather than the runs themselves. The experiment tracker holds the opposite slice: metrics and configurations logged automatically, with the questions nowhere, so orphan numbers are the default rather than the exception, sometimes lost in flat-file-hell (ie "results-final-v2-final-final.log"). None of them is built to hold the whole causal chain, from a claim in a draft back through the finding, the results, the hypothesis, and the references that motivated it. Pieces of this chain exist across provenance standards and workflow tools, and the deep dives in this series will name them and position relative to them; what none of them holds is the whole chain, attached at the moment of execution.

The system of record is that chain, kept as structure. It is not a better notebook next to the work; it is the work's own trail, with the provenance attached at the moment each step happens, by the machinery that ran the step. This not only allows you to research with confidence, it also means every result you publish carries what a rerun needs, now and in the future.

What the shape buys you

For you, the researcher: put the project down in May, pick it up in September, and start where you left off, because the record holds the state your memory dropped.

For your collaborators: they walk your actual search, dead ends included, instead of your recollection of it.

For your readers and reviewers: the trail from claim back to code, configuration, and environment is there by construction, kept automatically, and the full search is visible, not just the winners.

We believe the pace of science should be set by the ideas. The system of record is where the ideas, results, and findings all live. So you and others can stand on a strong foundation when continuing to push forward into the frontier (Surfing the Frontier Wave that rests between The Bitter Lesson and No Free Lunch).

Why I'm building Talaria Scientific

Fri, 24 Jul 2026 00:00:00 GMT

We're building Talaria Scientific to accelerate computational science research. Talaria is a multi-agent, multi-foundation-model harness: a researcher directs it, and it runs the undifferentiated machinery of computational science validation (the coding, the environment building, the HPC runs, the record-keeping) that currently gets done manually. Below I'll explain what that means, why I'm doing it, and what impact I hope it has.

To learn more about how Distributional became Talaria Scientific check out this post.

The bottleneck was never the ideas

Every computational scientist I know carries around more hypotheses than they will ever get to test. The reviewer suggestion that got explained away instead of implemented. The seminar question that ended with "that's an interesting idea." The paper that raised a "what if?" in the margin. The shower thought that would take a month of setup to check.

Ideas pile up because robustly validating each one is tedious: the data that needs wrangling, the environments that won't build, the baselines that need reimplementing because the original code no longer compiles, the sweeps that die overnight, the checking and rechecking of logs that separates a real effect from a lucky one.

Even with all that, the validation is only half the burden. You also need to keep track of everything: which variants ran, what changed between them, where the project stood when the other project interrupted it, how to reproduce it all now (and later).

Ideas are what moves science forward, but validation is expensive and time consuming, and so validation bottlenecks the science. The pace should be set by the ideas.

Twenty years of the same problem

I've spent my entire career on different versions of discovering and trying to accelerate through bottlenecks in this process. In undergrad and grad school the same problem kept surfacing in every field I touched, from metagenomic assembly at the national labs to quantum mechanics at the Max Planck Institute: whatever the science was, the final step was always parameter tuning. We jokingly called the process "grad student descent": the least tenured person in the group, turning knobs in a high dimensional space by hand to try to get a better result. It burned expert time, it burned expensive compute, and the results were not even good: hand tuning stalls in local optima and brute force aliases over better configurations entirely. Making that step efficient became central to my PhD thesis, using Bayesian optimization to accelerate the tuning.

At Yelp I found the same tuning problem waiting in ads, search, and recommender systems, so I open-sourced my thesis as MOE, the Metric Optimization Engine (github.com/Yelp/MOE). It quickly garnered 1000+ GitHub stars (in 2014!) and spurred dozens of talks around the Bay, showing the problem was pervasive across industry. I turned it into my first startup, SigOpt, where we spent seven years running optimization as a service for demanding modeling teams in trading, tech, and research for everything from hyperparameter tuning to materials design and more. That was the first pillar: making the models better through optimization.

Intel acquired SigOpt in 2020, and I became VP of AI and HPC for Supercomputing, ensuring applications actually performed on the deca-megawatt machines headed to national labs. That was the second pillar: making AI and scientific code practical, working at real scale. It was there that a different problem surfaced. The models had grown powerful, and with power came complexity: non-determinism (the same question could return different answers), non-stationarity (behavior drifted as the weights/code evolved), and chaos (small input changes produced large output changes). Optimization could help you tune a model, but nothing in the stack told you whether you could trust what came out, or if you were just overfitting to some metric or eval. I had spent 15+ years building evals and ways to optimize them efficiently, but never dove into how you can trust them. So in 2023 I founded Distributional to build the third pillar: testing and analytics to make AI systems trustworthy and understandable.

Better models, scalable systems, trustworthy results: you need all three to do great science, and the work demands them together. I worked on the three pillars sequentially and am finally putting them together. Talaria Scientific is that synthesis: everything I've built, aimed back at the problems I have been trying to solve for the last two decades. And the target isn't science in the abstract. It's empowering the world's experts to go further, faster. It takes the grunt work humans are bad at (high-dimensional optimization, large-scale sysadmin, keeping track of everything) into the harness, on the record and under their control, and leaves the work only they can do, the ideas, fully in their hands.

What changed, and what didn't

Over the last three years, agentic coding changed what a strong software engineer can do in a week. The engineers getting the most out of these tools were already excellent: but now they direct agents, review the output, and ensure the architecture and systems align with their vision and taste.

Computational scientists do work with the same shape (code, data, infrastructure, experiments, validation), but their tooling sits roughly where software tooling was three years ago. Most are using AI for pieces of the process: asking a chat app instead of Stack Overflow, pasting snippets back and forth, cobbling it together as they go. There is a large gap between that and directing long-running agentic loops that build and validate entire swaths of the work, and software engineering has already demonstrated that the real acceleration happens when the tool understands the job.

The catch is that the job is different. Doing science is different from doing software engineering. Good unit test coverage is fundamentally different from making sure you don't violate physics. Shipping an app you can iterate on next sprint is fundamentally different from publishing a result that claims some truth about the universe and needs to be defensible forever. An agile board of tickets and pull requests is fundamentally different from scientific exploration and hypothesis testing. A general coding agent understands none of that, and the model companies are incentivized to go after the larger market: their agents are trained and RL-tuned on software engineering processes, apps, and code, not on the computational science process. To truly transform the process you need a tool built for the job.

We still make scientists grind their own mirrors

For a long stretch of astronomy's history, being a great observer meant being a great instrument maker first. William Herschel cast and polished his own speculum mirrors, sometimes sixteen hours at a stretch without taking his hands off the metal, because a telescope good enough for his questions wasn't something you could buy. The telescope was the bottleneck, so telescope craft was the job.

This is roughly how computational science is done today. A researcher's week goes to data wrangling, sysadmin, plotting, reimplementing baselines, and manually validating results. What remains, with what little time left, is the actual science: the smallest slice of the week by volume, and the only part that they alone can do. Science doesn't need to be slow; it has just always been bottlenecked by the tooling available. When mirrors became something you could buy, observation accelerated. When computers became something every lab could have, simulation accelerated. Agentic AI is the next accelerator, and right now scientists are still grinding it by hand.

Muscle and memory

We are building augmentation for the process, not automation of the scientist; a mech suit, not an android. A mech suit is amplification you wear, an android is a replacement you send off to act in your place. Herschel's telescope sat firmly on the suit side of that line by augmenting his eyes: it multiplied his power to see and never chose where to point itself. A mech suit takes the mechanical work off you, faster and better than doing it alone or delegating it to the most junior person in your lab, while every decision about where to walk stays yours. In software, scaffolding like this (the tools, memory, and checks wrapped around the models doing mechanical work) is called a harness, the application layer that sits on top of the foundational model substrate. The oldest version of the idea is why we chose the name: Talaria are the winged sandals of Hermes, which allowed the wearer to travel at the speed of thought. We aim to build tools to let researchers do computational science at the speed of thought.

This muscle of the suit removes the validation schlep, everything that sits between an idea and a result you can trust while adding no science of its own. It's the data wrangling, the environment setup, the baseline reproduction, the parameter sweeps, the manual results checking, plus the bookkeeping that shadows all of it. It is still trustworthy, valid, and reproducible science, but now you don't need to manually perform every step. The grind around the thinking, not the thinking.

A system of record is the suit's memory: every idea, run, result, and decision, kept with its provenance, whether or not it worked out. The lab notebook, inverted: a record the work keeps of itself instead of one the scientist has to remember to keep. It is what lets you put a project down in May and pick it up in September exactly where you left off; what lets a collaborator, a student, or a reviewer walk your actual search instead of your recollection of it; and what makes the trail from claim back to code, configuration, and environment reproducible instead of reconstructible.

Ideas should set the pace of science: Talaria moves the schlep into the harness, raises the bar to trusted and reproducible, and speeds the flywheel for finding the next idea.

The top panel is how computational science is traditionally done: one idea at a time crosses a wide band of schlep by hand, toward a result that arrives and, after enough manual steps and undocumented tweaks, is not even always a result you can trust. The bottom panel is the same span with Talaria: the undifferentiated work runs inside the harness below the dashed boundary, the human lane runs unbroken from idea to a trusted and reproducible result, and the loop from result back to the next idea tightens. That last part compounds: a faster loop doesn't just move one idea through sooner, it spins the whole flywheel faster. The dashed boundary is where the rest of this piece lives: what belongs below it, and why.

The mech suit

In software engineering there is a concept of a "10x engineer," someone who can do the work at the level of 10 other median engineers; these people are becoming "100x engineers" with agentic coding. The goal of Talaria Scientific is simple: take the 10x computational scientist and make them 100x. The gains land exactly where the tooling gap is biggest: the world-class researcher whose domain expertise no model has, and whose week still goes to work no scientist should be doing by hand. This also empowers everyone in the lab to put on the mech suit and explore their ideas, instead of spending years as the hands the work gets delegated to.

I believe that this can't be solved with a completely autonomous scientist, the version where you just point an agent at a problem and come back to perfect results. I believe we are a long way from just writing /goal solve stable fusion containment and then heading out for drinks. That dream is aimed at the wrong target, for two reasons.

The first is that novel research is out-of-distribution by design. A model trained on the world as documented is strongest where the world is well documented, and a discovery is precisely the place where it isn't. General capability keeps expanding over what's documented, but the interesting research keeps living past its edge, there will always be a frontier, by construction (Surfing the Frontier Wave that rests between The Bitter Lesson and No Free Lunch). So the scientist pushes past the edge of the known, and the machine fills in behind: running the validations, holding the provenance, keeping the record honest. The human stays on ideas and taste. The machine takes the schlep.

A boundary follows from this, and we hold it deliberately (it's the dashed line in the figure above). Mechanical work within an agreed budget usually benefits from just running: environment setups, sweeps, baseline reproductions, plots. Anything that smells like a scientific judgment gets proposed to the user, never assumed. No absolutes on either side: the boundary is a dial the scientist owns, not a law of the product.

In practice this helps in two ways. The first is cranking the machine: the suit runs the validation work in parallel, within budget, while you think and ingest results. The second is a better machine to crank: the system of record underneath the work that makes sure you can trust your results.

Optimizing for truth, not for an eval

Train an agent to optimize an eval with nothing pinning down the search or the reporting, and you have built a p-hacking engine, a perfect overfitting benchmaxxer. I saw this time and time again during SigOpt when we helped thousands of experts optimize their systems worldwide, only to realize at the end that they were aiming for the wrong thing. "Publishable" and "true" are different objectives, and an optimizer pointed at the wrong one will find exactly the results that separate them: the spurious findings.

So the harness optimizes for correctness, reproducibility, and truth, and "passes review" stays downstream, never the target. The integrity is structural, not aspirational. Results face adversarial review from agentic reviewers rewarded for killing a pet hypothesis rather than confirming it. And the full search gets recorded, not just the winners, because "we tried 50 things and report the best one" is the campaign-level version of p-hacking that no per-experiment rule catches (just like the famous xkcd).

The component that holds all of this is the system of record, and it was built in an opinionated way: a reference graph at the top (the literature you build on, linked so every claim traces to its source), a triad of hypothesis, results, and findings graphs growing beneath it (what you're asking, what actually ran, and what you now believe, with every result answering a hypothesis and every finding carrying its evidence), and a concept graph overshadowing all of them (the living map of definitions the other graphs share). Each of those structural choices exists to kill a specific way results go wrong, from orphan numbers to reruns that are reenactments instead of reproductions. You can read about the full anatomy and decisions made choice by choice here.

Rigor that blocks serendipity would be its own failure though. Science often advances by stumbling into things. So the design gates the strength of claims, never the existence of discoveries: nothing real gets blocked, it gets labeled honestly until it earns its confirmation. If a tool makes it easier to fool yourself, it makes science slower, not faster, whatever the throughput numbers say. But the harness allows for full, untethered exploration and lets you self-heal your system of record as you go, so when you are done you can trust every result you plan to publish (and can reproduce it exactly).

Where this is headed

Talaria is built for the people doing science in silico: the computational physicists, applied mathematicians, and simulation-driven engineers whose experiments run on solvers and clusters rather than in physical labs. The work that used to dominate supercomputer usage before LLMs. We are starting physics-first on purpose, because that is where the guardrails are strongest: physical invariants, convergence checks, and validated solvers give the harness something rigorous to hold the models against. And the reach is bigger than a tooling niche: better computational science methods push forward materials, aerospace, energy, and the world models behind robotics and autonomy. Each of those fields moves further, faster with better computational science research, and increasing the pace of that research is exactly what the harness is for.

Talaria is in private beta with a small group of computational scientists, starting where my own research lives: PDEs, optimization, and the compute-heavy corners of applied math. The product will be open source by NeurIPS 2026 and always free for open scientific work. How we're building it, and why in the open, gets its own post soon. If validation schlep is what slows your research down, or you just want to watch this get built, get in touch at talariasci.com.

The ideas were never the scarce part. Every computational scientist is carrying a backlog of them: the reviewer's suggestion, the seminar question, the what-if in the margin. Too many of those die of tedium rather than of being wrong. The bottleneck in computational science was never the ideas. If we do this right, it won't be the validation much longer, either.

If you are working on anything this touches (physics models, optimization, world models for robotics and autonomy, materials, aerospace, or anything fluids related), we want to hear from you. Reach out if you want to work with us.