The architecture of the Talaria research harness

Talaria is a multi-agent, multi-foundation-model harness for computational science research: a scientist directs it, and the harness runs the machinery of in-silico hypothesis validation that is often currently done by hand (the coding, the environment and baseline building, the HPC runs, the record keeping).

I have written about why we're building it and about the System of Record at the center of it. This post describes the architecture of the harness itself: the whole system at once, what each piece does, why each piece is shaped the way it is, and what loops back into what. This post should help you decide whether a system like this would help with your research and give you an overview of how it works.

What it is

A scientist directs a driving agent (a frontier coding agent) that runs twenty named actions as skills against a System of Record that acts as a single source of truth for the entire research process. The Research Loop is at the core: ideas are hardened into plans and concrete hypotheses which are implemented and run on available compute, with the results being synthesized into reviewed findings supporting or refuting the hypothesis; this leads to new ideas and the process repeats. The figure below is the complete inventory of those skills and how they interact at a high level, and the rest of the post dives deeper.

The main Research Loop can be thought of as a state machine that builds up the System of Record over time. The Research Loop actions are stages: a stage has a queue of work waiting for it, a session ledger of attempts, and kickback semantics for when a later stage rejects what an earlier one produced. As each stage advances, it writes to its session ledger and updates the System of Record.

The driving agent runs each action as a skill (a versioned, auditable instruction set). The user can read (and edit) any of the skills as they go. Every skill pins its inputs and outputs in the journal with their provenance, so an audit follows the pins to an artifact instead of trusting the model's account of what it did. A record full of reproducible runs and an adversarial review panel are the regression suite that guardrails the process as a whole. The underlying skills, models, and driving agents can be upgraded at any time, with a re-validation layer to ensure alignment, so the system can evolve as more capable models become available. An analogy is a sheet thrown over stacked plates: the sheet is the harness (the skills and gates), the plates are the models the labs keep stacking underneath, and the harness re-drapes over whatever is stacked, while the System of Record underneath it all is the artifact that compounds.

The high-level goals and constraints of the system are as follows:

• Anything the record states must trace back to what produced it: the exact commit, config, seeds, environment, and cost behind every result, so an audit is a lookup instead of an archaeology dig and /repro can reproduce any result by construction.

• Claims and evidence are bound before any computation is run: pre-registration of hypotheses plus immutable results keep a fast loop from becoming a p-hacking engine.

• The scientist owns the judgment and the controls: anything that looks like a scientific conclusion arrives as a proposal, and anything that spends money faces a pre-defined budget gate.

• Exploration stays cheap: the rigor of the machinery scales with claim strength, so a what-if costs a conversation, and only the claims you want to harden and eventually publish go through the full depth of inspection and reproduction.

• Designed for scale up, out, and across by construction: a university cluster, a national-lab allocation, and a heterogeneous set of neoclouds differ by a config block for the runner only (not a rewrite of the code).

• The tools are replaceable and upgradable: models and skills can be swapped as the field moves; the record compounds regardless of the driving agent or underlying models.

The harness is "Multi-agent, multi-foundation-model" in three specific ways: the driving agent itself can use many subagents as it invokes each of the actions above, review panels (for ideas, plans, code, findings) fan out many isolated reviewers as subagents using potentially different underlying models, and the model for the driving agent of the system as a whole is the foundation model of your choice (and can change over time).

What it does

The twenty actions come in five groups: the Research Loop skills are the stages of the hypothesis validation work, extra rigor skills harden the work and extend the record as part of this loop or asynchronously, inspection skills produce artifacts from the record at various levels of depth, maintenance skills keep the machine and record clean, and meta usage helps bootstrap and fix the harness itself. The table below is the action catalog of skills:

Skill	What it does
Research Loop
/idea	captures a thought, with its provenance
/plan	freezes an idea into a pre-registered hypothesis: configurations, seeds, metrics, and what it takes to validate or refute it
/implement	builds and reviews the experiment code that tests the hypothesis
/run	dispatches the work to your compute
/synthesize	proposes findings grounded in the results of the run(s)
Extra Rigor
/interview	hardens an idea through inspection, debate, and reference literature
/lit-search	grounds an idea in the literature; registers references with fidelity notes
/lit-review	curates search results into proposed concepts with references
/concept-search	finds the definitions and methods already in your toolkit
/concept-create	registers a reusable definition, metric, or method
/review-panel	attacks a drafted finding with isolated reviewers
Record Inspection
/status	tells the user what is done, in-flight, and up next
/repro	re-runs a result from its pins and diffs the outcome
/draft	writes content from a subset of the record
/explain	explains anything in the record, grounded in your references and concepts
Maintenance
/validate-graph	checks the graph's invariants
/lint	heals mechanical drift in the graph
/close-out	sweeps the record for loose ends before you share it
Meta Usage
/init	personalizes a fresh instance of the repository
/feedback	packages a bug report, with any crash attached

Every action lands in the record's append-only journal. Every action carries a permission class: the actions that spend money face pre-defined budget gates, and anything that looks like a scientific judgment arrives as a proposal to the user; the only way past a gate is an override signed by the user, which lands in the journal like everything else. Nothing phones home on its own (/feedback packages a report that you send).

The queues and /status are both derived from the record on every task, never stored, so there is only ever a single source of truth. The session ledgers carry the mechanical history of what was done (and the record holds the science), so an interrupted afternoon session resumes from the ledger rather than from memory. The ledgers and the record both live in the user's local instance: a stronger model rereading old sessions for anything an earlier pass missed is allowed by construction.

The System of Record

The System of Record is a single source of truth represented as a research graph of five node types: references (a DAG of what you read: the literature you build on), concepts (a directed graph of what you use: the metrics, baselines, and methods standard to the field), hypotheses (a DAG of what you asked, frozen before you ran), results (a DAG of what ran, immutable from birth), and findings (a DAG of what you now believe, grounded in results). Architecturally, the System of Record is all markdown and JSONL in a git repository, with no database and no server behind it. You can open it in Obsidian and read your own research like a wiki, grep it, or hand the whole directory to a reviewer (human or agent). Large artifacts (ie datasets, checkpoints, rendered plots) live in your own durable store like S3 or R2 and are content-addressed from the run rows; the record itself holds the graph, the journal, and the pins. Because the record is just plain files, every skill can read it, the graph and its history survive any individual tool or session dying, and nothing about the research is held hostage by the product: the agents are replaceable, and the record is inspectable or exportable at any point.

The record's connections come in two kinds, enforced by the harness. Typed edges live in the graph itself: which hypothesis derives from which, which result grounds which finding, which finding cites which reference. Following these typed edges answers what evidence this belief rests on, and what exactly produced it, all the way back to the prompt that kicked it off. Actions are events: every action in the system lands as an append-only journal event (an idea captured, a run dispatched, a result born, a verdict set, a gate refused), and a small rules file decides what each event kicks off next. Mechanical, budgeted work fires automatically (a completed run triggers aggregation and plotting; a materialized result triggers graph validation and reindexing), while anything structural or scientific is proposed to the user (never executed silently). Cascade depth is capped, firings per plan are capped, and past either cap everything switches to proposals for the user, so an event storm decays into a to-do list instead of a bill. The rules file itself can be edited at any time.

How it works: one idea to registered finding, end to end

The entire system is designed as a "'What if?' Machine": A user reads a paper and thinks: what if that method holds on my problem class? Talaria helps answer these questions efficiently and effectively so you can ask more and go deeper into the science instead of being mired in the minutiae.

The path is as follows:

1. /idea writes the "What if?" down with its provenance and any supporting initial context like papers or previous results.

2. /interview debates and refines the idea with the user before anything gets built, because an idea alone often isn't enough for a rigorous plan and certifiable hypothesis. This skill surfaces the prior art (using the /lit-search subskill) to check whether an idea is already in the literature, asks what the cheapest falsifying test would be, and refuses to co-sign the idea until novelty and impact survive the argument. It fields a fresh-context adversarial referee: a reviewer moved to the front of the pipeline to try to refute that an idea is worth exploring. There is also an escape hatch: you can overrule the interview and let the experiments decide at any point (and the override is also recorded).

/lit-search grounds the idea in the published literature: parallel searches over the topics, seed papers, and concepts. Sources are found by walking the citation graph outward from seed papers, ranked by influence, and read into structured reference nodes with notes on how faithful each baseline implementation is to its paper. Behind it, /lit-review curates what the searches registered into proposed concepts, so a metric, a baseline's quirks, and the right confidence-interval method are waiting for the next loop instead of needing to be reverse-engineered again.

3. /plan turns the hardened idea into a hypothesis with a pre-registered contract: which configurations, which seeds, which metrics, and what it takes to validate or refute it, frozen before anything runs.

4. /implement sits between the frozen plan and the compute. It owns the experiment code and its tests, with a self-review and a distinct implement-and-review phase before anything touches a shared cluster. The machinery for stamping an implementation scales with claim strength (exploratory or confirmatory): self-review, an adversarial code-review pass at confirmatory grade (separate from /review-panel), and a smoke test that runs locally. A stamp pins the exact git trees of the declared implementation and of the whole instance (the current record, experiment code, etc).

A dispatch checks its commit against the stamp at the moment it spends money: a committed edit anywhere breaks the match by construction, an uncommitted one already fails the clean-repo gate, and there is no invalidation event to forget to send. A failed run is triaged before more interpretation stacks on top of it, and a failure triaged as a code bug kicks its hypothesis back to /implement rather than getting a quick patch (at exploratory grade the kickback arrives as a proposal you can wave through). The stamp check is grade-scoped: it refuses confirmatory dispatches without a valid stamp and only warns exploratory ones (grade is the record's way of recording claim strength).

5. /run kicks off the computational experiments, but first it faces the gates. The experiment repo must be committed clean; the remote worker runs a materialized checkout of that exact commit, so the committed source behind a run is never ambiguous. The contract must be complete for the claim grade you are running at; a confirmatory registration with holes blocks before a dollar is spent; exploratory runs are more lenient. Finally, the budget must cover the dispatch. Only after all these checks does the sweep fan out on the available compute. A credential-free local backend can run the same dispatch contract on your own machine, so your first exploratory runs work locally before any cloud account is configured. Beyond local compute, SkyPilot is deliberately the only cloud interface in the system: your university Slurm cluster, a Kubernetes cluster, a national-lab allocation, and a commodity neocloud are the same interface, and the gap between them is a backend config rather than a rewrite of your dispatch layer. Every completed run lands as a row in the System of Record carrying the commit hash, the config hash, the seed, the environment fingerprint, the artifact hashes, and the cost (a local run carries the same row, at zero cost).

6. The result is born as an immutable snapshot of the output of a run when the sweep completes, and its coverage is checked against the registration: for example, a sweep that came back with eleven of the twelve promised seeds surfaces as a refusal (not a quiet green). A verdict on the hypothesis can only be measured against exactly one thing: the pre-registered criterion, evaluated on the results of the pre-registered sweep.

7. /synthesize drafts a finding grounded in the results, and /review-panel attacks it: nine reviewer personas in isolated fresh contexts (a skeptic of AI-generated work, a stand-in for the original method's authors, a baseline-fidelity checker, a statistician hunting selection effects, a systems reviewer, a figures referee, and three more), up to three rounds with critiques pooled between rounds, and a judge body that scores the attacks. The isolation buys each reviewer a fresh context, and a claim promoted to confirmatory grade escalates to referees running on different labs' models.

8. The user decides what enters the record as believed after reviewing the results and findings.

9. /repro re-runs a result from its pins, at any later date, and diffs the outcome against the record: bit-exact where the fingerprint matches, within stated tolerance where it does not, and an explicit cannot-reproduce-here when the substrate that ran it is unavailable. A result that moved is a finding in its own right, and it goes back to the panel.

The through-line of all the gates in the system is that the harness will fail loudly if any of the preconditions are violated (and the user must explicitly override on the record to continue). The evidentiary gates bind claim strength (not exploration); the operational guardrails (budgets and clean commits) apply to everything, the way seatbelts do; the user can ignore them if they choose to. Exploratory work runs with light checks and full freedom, mess tolerated on purpose; the machinery ramps up as your claims harden toward something you would publish. You can always explore first and let the record self-heal behind you.

The flywheel

The system is explicitly designed to accelerate a scientist's ability to do their research. Each loop through the machine grows the record: new references with fidelity notes, new baselines that now exist as runnable code, new concepts, new results with their full provenance, new findings, and the superseded paths that tell you where not to dig. The next loop starts where you left off: the lit search begins from references already banked, the interview tests a new idea against findings the record already holds, and baselines built in a previous loop come back ready to run. A faster loop on a compounding record is what this architecture is for: the flywheel gains speed with every pass. On the figure, that is the arc from /synthesize back to /idea (and back again and again). In grad school a single what-if would cost me months; that is the loop this machine rebuilds and collapses toward hours, with the schlep inside the harness and abstracted away, and the user still in complete control.

Past the loop

The rest of the design sits past the single loop. /draft writes content from a subset of the record, from blog posts to papers and talks, with every quantitative claim resolving to its pin and figures regenerated from artifacts rather than pasted. The record's format was chosen so one lab's research graph can merge into another's.

What lives where

Everything you download is the field harness focusing on a specific domain like PDE surrogate methods or Bayesian Optimization. It contains the Talaria core (the Research Loop, the gates, and the System of Record machinery), the field pack (the baselines, metrics, and solvers of the chosen domain), the twenty skills, the docs, and a seed knowledge base (starter concepts and references, plus a worked example record you can read before you run anything). Talaria maintains that whole side of the boundary, and that side is public: the skills get upgraded, the field pack grows baselines, the docs improve, and every instance starts from the same repository.

The instance is the local copy of everything the harness builds around your work, and it is private and yours: your record, your experiment code, your profile and configs, and your papers with their statuses. The record is the research graph and its journal, including the concept graph you accumulate as you go. The boundary between the two is part of the directory layout itself, and the machinery enforces it. An upgrade updates the harness directories and never writes into the instance directories, so a new version of the actions can never rewrite your science. The instance layer is the same set of git trees that the stamps pin, so the boundary you own is also the boundary your runs are attributed to. The seed knowledge crosses the boundary only at initialization: before that it is Talaria-maintained template content, and afterward it is instance content that is modified with your research.

The third region is the published literature, and it is public like the harness. Two paths lead out of the instance. /feedback is the direct one: it packages a bug report with any crash attached, and it needs to be manually sent. There is no private-contribution path, so your unpublished work never enters anyone else's harness, and the only road back into the harness runs through the literature: when a method from your instance ships in a paper, we pull it into the field harness the same way we pull in any other published method from the literature (we read that commons anyway). The larger flywheel at play is that the harness compounds on what the field publishes, and every instance gets better as its field does, which accelerates publishing new results.

What is next

Talaria is in private beta with a small group of computational scientists today. The harness will be open source by NeurIPS 2026, free for open scientific work, always. If you want to judge the architecture properly, future Design Notes posts go component by component. And if you would rather judge it by using it on your own research, request access at talariasci.com.