Building an autoresearch lab I can actually watch

For the past six weeks a machine I built has been doing part of my quant research for me. It sweeps arXiv and the open web for mechanisms, proposes a hypothesis, writes the strategy code itself, backtests it across twelve weekly out-of-sample windows, and tells me which ideas are dead. It has run somewhere around 700 experiments. Nothing has survived holdout.

That reads like a verdict on autoresearch and it is not one. The honest version is narrower and more useful: what I have built is a very good instrument for refuting strategies and a not-yet-good instrument for finding them. Six weeks of failures bought me a fairly precise map of where a loop like this lies to you, and the mistakes that cost me the most time were never the strategies.

My first tests with an off-the-shelf autoresearch pipeline on autopilot produced nothing but a bunch of failures, and the six weeks since have mostly been me finding out why. The answer is that autoresearch lives or dies on its harness. The model proposes, the framework evaluates, and the two have to be isolated hard enough that the model cannot reward-hack its way to a good number. That part is reasonably well understood by now. The piece most setups skip is the one I now think matters most: visibility. A human cannot steer a loop they cannot see, and for weeks I could not see mine.

What it actually is

My previous quant research was much too prone to overfitting: I would find something that looked like an edge and watch it disappear once I looked again. The one good thing that came out of it was the environment — a workflow built exactly to iterate on ideas quickly and test them robustly. This is personal research I do in my own free time, outside the day-job, so the time budget is whatever is left over at the end of a week. Once the environment existed, an unattended pipeline was the only time-effective way to keep the research alive at all.

The result is an agent loop fourteen stages wide. A research stage sweeps arXiv and the open web and writes briefing documents the hypothesizer then has to cite from, verbatim. A hypothesize stage proposes a mechanism and registers its prediction before anything is written — which pairs, which side, expected trade frequency, expected Sharpe. A codegen stage writes the actual strategy, which has to clear a mechanical gauntlet before any expensive run happens: lint, smoke tests, look-ahead checks, determinism, parameter usage. An adversarial review stage tries to refute the code before the backtest ever runs. Only then does the strategy go through twelve weekly out-of-sample windows — walk-forward: fit in-sample, test out-of-sample, one window at a time — and a deterministic evidence gate sorts the result into validated, refuted, or inconclusive.

Everything the loop learns lands in an append-only memory that is injected into every future hypothesis, and when a cycle closes refuted the loop now writes its own seed for the next one and continues the lineage. The graph is the single source of truth for all of it — fourteen nodes and their conditional edges, replayed against 1,674 real cycles with zero violations before I let it drive anything.

On top of this there is a live dashboard that draws that graph as it runs: which stage is executing, what it produced, what failed. I am now building an editor around it so I can inspect and tweak stages mid-cycle, re-run a single stage against real artifacts, and read any past experiment’s actual model turns.

The tradeoffs

Validate the validator before using it

Before I let the loop generate a single hypothesis, I spent the first days auditing the backtester itself. The audit found four real bugs, including a data leak where a forming candle could see its own close before it existed, and a column-index bug in the optimizer that silently rejected every candidate it was handed. Both had been running in my production tooling. If I had skipped this step, everything the lab produced afterwards would have been noise and I would not have known it. The most valuable first product of an auto-research setup turned out to be an audit of its own measurement.

Two weeks later the same trick paid off from the other direction. Porting my production gating logic into the lab so the lab could test what production actually does surfaced a bug in the production version: a bootstrap that sliced the wrong series length and therefore waved through every positive candidate it was supposed to be filtering. The same slice bug sat in several sibling strategies. I found it by making the lab imitate production, not by testing production.

The model proposes, the framework evaluates

The inner loop — the part where the model iterates on a candidate — runs against a frozen evaluator that is hash-pinned: the model can run it as often as it wants and read every number it prints, but it cannot change it, and the pin breaks if anything touches it. There is a sanity gate where a flat signal must abstain, so a strategy cannot win by simply always trading. This is the rule I would give anyone building one of these: the model proposes, the framework evaluates, and the two never touch, because a model that can grade its own homework will.

Cheap model generates, expensive model refutes

Code generation runs on a cheap worker-tier model (DeepSeek). Review runs on an expensive frontier model (Opus). The invariant is deliberate: the cheap model writes, the expensive model refutes. It caught real defects from day one — a wrong divisor in a net-profit calculation that the writer’s own self-check had signed off on, and a wrong parameter that would have silently ruined an experiment. The cost is that review became the most expensive stage in the loop. A refutation before the backtest is still the cheapest mistake the system can make.

The strategist never sees the holdouts

Every “validated” family got re-tested by a human — me — on untouched data the loop had never seen, and the holdout results never went back into the loop’s memory. That firewall keeps the strategist uncontaminated. It also created a blind spot: nothing could tell the loop to stop minting variations of a family that was already dead, so one overnight run piled dozens of siblings into a single family and produced sixteen correlated “validations”. One bet, counted sixteen times. What makes it worse is that the lab’s own research stage flagged the monoculture before I did; it just had no mechanism to act on it. The fix was mechanical rather than judgemental: experiments are clustered by what they actually trade, and a cluster is capped.

In the first version the model also got to pick its own assets, whichever ones it liked the look of. That is selection bias by construction — the model cherry-picks the market where its idea happens to fit. The fix is a fixed base of eighteen assets. I am not entirely sure eighteen is the right number, though; it can be a bit too broad, while the single-asset alternative produced too few trades to say anything statistically significant. That tradeoff is still open.

One language would have been cheaper

The adaptive experiments fit a signal in Go and then trade the JavaScript version, because that mirrors the shape of my production system. Every generated pair has to pass a parity gate proving both implementations behave identically, and that gate is the number one cause of codegen deaths. Asking a model to write the same logic twice roughly doubles the surface it can be wrong on, and the gate catches every one of those mistakes at the cost of the whole experiment — tokens, time, and accuracy. On a rewrite, single-source — one language, no twin — is the first structural fix I would make.

What the failures actually were

Around 45 experiments have ever been marked validated. 44 of those 45 predate the evidence tightening I added in mid-July, and they share a profile: a high Sharpe on twenty to thirty trades. Six families got as far as a real holdout — run by me, by hand, on a month of data the loop had never touched. The holdout record is 0-for-6.

I was already skeptical of the validated verdicts before the holdouts ran; the samples were small and I knew it. The holdouts confirmed it with a consistency that was almost funny. The in-sample ranking did not just weaken, it inverted: the variant with the most filters stacked on it, the best-selected one of the group, performed the worst out-of-sample.

What was tested	In-sample	Holdout
London-session family, 4 variants	all four “validated”	all regressed to zero, ranking inverted
Best statistical profile in the ledger	Sharpe 0.41, profit factor 1.91, 145 trades, p 0.003	Sharpe 0.01, profit factor 1.01, p 0.48
First adaptive experiment	Sharpe 0.65, profit factor 2.65, 52 trades, p 0.005	Sharpe −0.24, profit factor 0.75, net −47
Early “Sharpe 1.71” signal, re-run at scale	Sharpe 1.71 on a single window	Sharpe −0.13, p 0.80

The best profile in the ledger is the one I keep coming back to. Its hit rate held almost exactly — what collapsed was the payoff. The edge had been living in the average win size, the most luck-sensitive component you can build a case on, and nothing in the in-sample statistics was ever going to tell me that.

None of this was the gate being lazy, which took me longer to accept than it should have. I eventually did the power arithmetic, and it is unforgiving. With a 60-trade minimum, the gate can only confirm a per-trade Sharpe of 0.212 or better. The measured median of everything landing on the positive side is 0.188, with a standard error around 0.129. The effect and the error bar are the same size. That equality — not sloppiness — is why 87 experiments in one exposure cell, the same side and the same pair set, produced 41 “validated” verdicts and zero holdout survivors. The answer is not a stricter gate. It is a second, cheaper, much higher-n test of the mechanism — conditional forward returns across every bar, no optimizer, no abstention — kept separate from the expensive test of the strategy. That one is still on the list.

The subtler version of the same problem is a candidate whose “signal” turned out to overlap its own unconditional control 94% of the time. It was not filtering anything; it was picking an entry hour. That generalises into a gate the lab needs and does not have yet: any candidate whose trade count sits within about 10% of its control is re-timing, not signalling.

Then the measurement side finally worked properly and told me the truth in one go. The first full-scale run where the evaluator traded every single window came back with 3,783 trades across twelve windows, a net result of −17,663, and zero positive windows. An honest, unambiguous loss — and the cheap inner evaluator had pointed in the same direction as the real engine, which is the one genuinely good piece of news in the whole scoreboard. The compass works. It has just had nothing good to point at.

Before that, there were weeks where the lab produced almost nothing at all. Its own memory contains the diagnosis it reached at the time — the gate must be suppressing the trades, it wrote — and it was wrong. The real cause was a sizing bug in my own eval kernel that rounded positions down to zero on the exotic pairs. The strategies were never the problem; the measurement was. There was also a counting bug that came close to minting the first false confirmation: adjacent test windows shared a boundary bar, so a strategy triggering on it had the same trade counted twice, and the lab was one fix away from shipping a “confirmed” strategy off double-counted data. That was the week the lab learned that its own measurement can be the liar.

The reason the sizing bug survived for weeks is the point of this whole post. I had too little insight into my own harness, and the loop’s only output was prose. Reading hundreds of lines of model narration about what has been done is a fast way to get mentally exhausted, and it does not show you a bug. A graph does.

Right now the bottleneck has moved again, to the generator. After the last successful close in early August there were 480 consecutive error closes, almost all of them cheap smoke-level codegen defects — wrong parameter ranges, declared-but-unused parameters, references to helpers that were never emitted. Somewhere along the way the lab’s own memory started poisoning its generator: more accumulated learnings meant longer, more constraint-laden hypotheses, which the code generator (which never sees those learnings) could not satisfy, which the review gate then rejected. A feedback loop that only gets worse on its own.

Honest caveats

No human outside my own setup has ever looked at any of this, and that was deliberate. I read the public autoresearch projects — Karpathy’s, Driveline’s Claude Code port, and pi-autoresearch — and the inner eval loop borrows its shape openly from all three: single-file edit scope, a locked scorer, a metric contract read back between model calls, a noise floor derived from multi-seed variance, a sanity gate where a flat signal has to abstain. Past that I did not read much, because my use case is very specific and it had to integrate with my own ecosystem anyway. I think it is best to just try and experiment by yourself, even though that can stack up to costly learnings quickly. I would not understand it as well as I do now had I copied things from other people.

I should also be plain about what this was never for. I never had the confidence in this system to put anything it made into production — it was always research to see how far the model would come, and to learn and iterate on the system. I am not selling signals and I am not giving financial advice, and the scoreboard above is the best proof there is that there is nothing here to sell.

And the honest state of the harness itself: it still is not what it is supposed to be. The eval environment needs hardening, and I still owe the code a deep dive into what the model actually produced. I am thinking about building an entirely new iteration from the ground up, because the previous experiments have messed up what I was going for — mostly accumulated cruft in the pipeline, plus the twin-language design. The memory is not something I am bothered about losing. It was never a production system, it was a very early prototype that needed iterating on either way, and discarding the entire memory would not cost me anything I want to keep.

Where it sits now and what is next

The loop and the dashboard are live, and the active work is the stage editor: the production design is approved and implementation starts now. It makes the pipeline editable and inspectable — per-stage prompt templates applied live, model and temperature knobs, gating which data a stage may touch, running one real cycle, re-running a single stage against real artifacts, and browsing any past experiment’s actual model turns per node. It can also generate new stages itself: describe a stage, and it produces the prompt, config, and graph wiring as a reviewable draft, so I never wire nodes by hand.

If I rebuild — and I probably will — the isolation and the mechanical gates carry forward, hardened further. The cruft gets cut. The Go/JavaScript twin gets another long look, because it makes the models make mistakes too quickly, and every one of those costs tokens, time, and accuracy. And the parked question stays parked: if discovery ever produces a holdout survivor, the lifecycle for running one live is sketched and ready to be built, and not before. The rebuild decision itself waits on that deep dive into what the model actually wrote.

What I would take from it

None of this is really about trading. Hand an agent a loop, and two things decide whether the loop tells you the truth.

The first is isolation, and it is the one everybody already talks about. The model may run the evaluator as often as it likes and may never change it, because the alternative is a system that optimises your scoring function instead of your problem.

The second gets discussed much less. You have to be able to watch the thing work. Every expensive mistake in these six weeks — the weeks of empty output, the double-counted trades, the memory quietly strangling its own generator — was sitting in the artifacts the entire time and completely invisible in the prose. A wall of model output tells you what the loop believes it did. A graph tells you what it did. Most of the effort in agentic coding right now goes into making the model smarter; I am spending mine on being able to see it, because the bug that costs you a month is never the one the summary mentions.