t0ul

A sortie log that survives a restart

· 7 min read · #ai-security, #red-team, #llm, #tooling, #golang

Part of my EVA series, where I build the cockpit I pilot GORED runs from. Last time I streamed a live run to the browser by tailing its trace. This post is what happens when those runs pile up and I want them to survive a restart. Code in t0ul/GORED; write-ups in t0ul/ai-security-engineering.

Intro

Once I could watch a run, I started doing a lot of them. Different models, different prompts, the same target, trying to build a feel for what moved the needle. And every one of them left a trace file in a directory, which was fine, but the cockpit’s sense of what runs existed lived in memory. So the first time I restarted the server to ship a change, my whole afternoon of sorties vanished from the list. The traces were still on disk, but EVA had forgotten they happened.

That is the kind of bug that is annoying once and unacceptable twice, so I gave EVA a sortie log: a small SQLite registry that sits next to the traces and remembers every run across restarts. The interesting part is not that I reached for a database, it is what I chose to put in it, because the obvious answer is wrong and the split I landed on is the thing that makes the rest of the cockpit work.

The EVA leaderboard, models and skills scored side by side The comparison view the registry feeds. Each row is a model paired with a skill, scored on real target bytes, sortable and pinnable. This is the view that made “which model, under which setup” a question I could answer instead of guess.

I. Two homes for a run

The wrong answer is to dump everything about a run into the database: every step, every tool call, every model reply. I did not do that, because the run already has a perfect home for its steps. The per-step detail lives in the gledger trace, the hash-chained log I have leaned on in every post, keyed by one trace id. That is the source of truth for what happened, and it is tamper-evident, which a row in a database I can freely update is not.

So the registry holds only two things: the launch configuration and the final summary. In Go that record is small and deliberately so:

// EVARun is the durable copy of a sortie: the launch config and the final summary.
// The per-step detail stays in the gledger trace at Trace; this is only the index.
type EVARun struct {
	ID       string
	Stage    string
	Base     string
	Model    string
	Trials   int
	Autonomy string
	Status   string
	Started  time.Time
	Elapsed  string
	Trace    string   // path to the hash-chained trace with the real steps
	Prompt   string
	Sums     []core.ContenderSummary
}

Notice Trace is a path, not the content. The registry is an index that says a run with this config happened, ended this way, and its real story is in that file. When I want the steps, I read the trace, which is verifiable. When I want to list a hundred runs or rank them, I read the registry, which is fast. Each does the job the other would do badly.

II. Upsert on every status change

The registry has to be right even when a run is still going, or has crashed, so EVA writes the row on launch and again on every status change rather than only at the end. The whole persistence call is one upsert:

func (s *Store) SaveEVARun(r EVARun) error {
	sums, _ := json.Marshal(r.Sums)
	_, err := s.db.Exec(
		`INSERT INTO eva_runs (id, stage, base, model, trials, autonomy, status, started, elapsed, trace, prompt, sums_json)
		 VALUES (?,?,?,?,?,?,?,?,?,?,?,?)
		 ON CONFLICT(id) DO UPDATE SET
		   status=excluded.status, elapsed=excluded.elapsed, sums_json=excluded.sums_json /* ...and the rest */`,
		/* ...bind the fields... */)
	return err
}

Insert if new, update if the id is already there. So a run shows up the instant it launches, flips to waiting when it pauses at a checkpoint, and settles to captured or failed at the end, and the latest state is always the state on disk. Restart the server mid-run and the row is still there, marked running, which is honest: it tells me a run was in flight when the process died, instead of pretending it never existed.

The database itself is modernc.org/sqlite, a pure-Go SQLite with no C compiler in the loop. I reach for it whenever a tool wants a real local database without dragging cgo into the build, because it keeps the whole thing a single static binary you can copy to a machine and run, which is the same reason the cockpit is one binary in the first place. Listing is a plain query, most recent first, and that list is the sortie log the cockpit opens on.

III. Compare, because the research is A/B by nature

The reason the registry earns its keep is that the questions I actually had were comparative. Not “did this run capture” but “did model A do better than model B on this target,” and “was my sharpened prompt actually sharper than the plain one.” Those are A/B questions, and you cannot answer them from a list you have to squint at.

So two things sit on top of the registry. There is a compare view that puts two runs side by side, their configs and their outcomes next to each other, so a model-versus-model or prompt-versus-prompt question is a glance instead of a memory test. And there is the leaderboard in the screenshot above, where each row is a model paired with a skill, scored on real target bytes, sortable and pinnable. Both of them read the summaries the registry already stores, so they cost almost nothing to build on top of what persistence gave me for free. This is the view that turned “which model, under which setup, for this kind of bug” from a thing I half-remembered into a thing I could sort a column on.

IV. Triage reads the trace, not a second copy

One decision here I would defend hard. When the run page shows you the findings and the exact request that captured a flag, the repro curl you can paste into a terminal, it does not read those from the database. It parses them out of the trace. I was tempted to store findings as their own rows, because that is the normal thing to do, and I am glad I did not. If the findings lived in a second place, that second place could drift from the trace, and then the cockpit would be showing you a finding the tamper-evident record does not actually support. By deriving triage from the trace every time, the thing you act on is always backed by the verifiable record, not by a copy of it I might have written wrong.

The registry does carry a little scar tissue from growing, and I will own it rather than hide it. I added a models column after the fact, for researcher sorties that run several models at once, and rows written before that column existed scan it as a null string instead of blowing up. That is the honest cost of letting a schema evolve while you are still learning what a run needs: you handle the old shape gracefully instead of pretending the first version was final.

Conclusion

So a run lives in two homes on purpose. Its steps are in the hash-chained trace, which is the truth and cannot be quietly rewritten. Its config and summary are in a tiny SQLite registry, which is the index that survives a restart and makes a hundred runs listable, comparable, and rankable. The views that answer my real questions, compare and the leaderboard, ride on the registry, and triage reads the trace so what I act on is always backed by the record. One database for speed, one trace for truth, and no third copy to drift.

Next I will open the door I have been pointing at since the first post: the factory, where the tools and skills a run uses are authored. It is the one part of the cockpit I deliberately left read-only, and the reason why is the whole point of the post.

← Back to the blog