Can a local LLM pentest your own app?
Part of my GORED series, where I point small local language models at my own web apps and measure, honestly, whether they can break them. This first post is all setup: the weapon, the target, the ruler, and the two rules. The harness is t0ul/GORED and these write-ups live in t0ul/ai-security-engineering. Every number I quote comes from a kept, hash-verified trace.
Intro
I had a question that would not leave me alone. We keep reading that language models can hack now. There are demos, there are leaderboards, there are papers with impressive single numbers. So I wanted to know the version of that claim that actually matters to me, a person who ships small web apps: can a model I run on my own laptop, with no cloud, no frontier API, find the security bugs in an app I wrote, the way a human reviewer would?
I will tell you now that the honest answer is mostly no, and this series is the long version of why. That took me a lot of hours to establish, and a chunk of those hours were me fooling myself with numbers that looked like a yes. So before I show you a single result, I want to spend one whole post on the setup, because the setup is the thing that makes the rest trustworthy. A pentest benchmark you can accidentally cheat is not a benchmark, it is a mirror.
This post has no findings in it. It has the rules of the game.
EVA, the cockpit I built to run all of this. The preflight panel on the left is the whole argument in miniature: the oracle reads real bytes, containment is armed, and the model is the thing being tested.
I. The weapon is the model, not the tool
The framing I settled on is “the model is the weapon.” That sounds dramatic, but it is actually a constraint I put on myself, and it is the hardest one to hold.
A pentest harness is a pile of tools: an HTTP client, a fuzzer, a scanner, a thing that forges a token, a thing that floods a login. It is very easy to build a harness where the tools do the attacking and the model just picks which tool to run. You can get eye-popping scores that way. You can get a clean sweep. And you will have measured almost nothing about the model, because the exploit lived in the tool the whole time.
So the rule is: the model has to be the thing that finds and crafts the attack. If a tool already knows the vulnerable endpoint and the working payload, then a “capture” with that tool is the tool’s capture, not the model’s. I keep both kinds of run in the later posts, because the gap between them turns out to be the whole story. But I label them, every time, so the two never get quietly merged.
II. The target is my own app, on purpose
I am not trying to pop someone else’s box. The target is RiskRancher, a small app I wrote and maintain, plus a couple of public teaching apps when I need a vulnerability I did not plant myself. Using my own app matters for three reasons.
First, I know the ground truth. I can say with confidence what is wrong with it, because I did a security review of the app myself, item by item, and that review is the ruler I measure against. There are fifteen items on it.
Second, there is no legal or ethical gray zone. It is mine, it runs on my machine, and the whole thing happens inside a virtual machine with its network egress denied, so nothing I am testing can call home even if it wanted to.
Third, and this is the subtle one: using my own app is also a trap I had to watch for. If I design the harness while staring at my app’s bugs, I can leak the answers into the harness without meaning to, by naming the right endpoint in a tool description or routing the model straight to the vulnerable field. That is the second rule, below.
III. The ruler, and what counts as a point
The score is recall against that fifteen-item review. If the review found fifteen things, how many of them can the model reproduce? That is the number I care about, and it is the number I keep honest.
A point only counts when all of this is true:
- The model fired the attack itself, not a tool that already knew the answer.
- An oracle confirmed the finding by reading the real server’s bytes, not the model’s claim about what happened.
- The finding was found cold, meaning the brief did not hand the model the vulnerability class and the endpoint on a plate.
That last condition is the brutal one, and most of this series is really about it. It is easy to score well when you tell the model “the login is injectable, go.” It is a completely different thing to say “here is an app, find a way in.” The distance between those two sentences is exactly the distance between a tool and a pentester, and it is where the local models fall off a cliff.
IV. The two rules that keep me honest
Everything above compresses into two rules. I wrote them on the wall because I kept breaking them by accident.
No cheating. The oracle decides on the real server’s bytes. Never the model’s self-report, never a hardcoded expected string, and never a deterministic detector that I quietly count as if the model had found it. If I run a scanner that always flags a missing header, that header is a legitimate hygiene finding, but it is the scanner’s finding, and calling it the model’s recall would be a lie. The first time I caught myself doing a version of this, pooling a tool’s deterministic result into the model’s score, I had to throw out a result I was proud of. That story is in the next post.
No hardcoding. Nothing specific to my app is compiled into the harness. Endpoints and fields come from generic recon. Tool descriptions use placeholders, not my real routes. The classifier that decides “this looks like an auth endpoint” uses generic keywords, not a lookup table of RiskRancher’s paths. If I bake the answer in, the model is not solving anything, it is reciting.
Hold both of those and the experiment is science. Break either and it is theater. Most published pentest demos, if you read them closely, are quietly breaking one of them, and I say that as someone who broke them too.
Conclusion
So that is the frame. A small local model as the weapon, my own app as the target, a fifteen-item review as the ruler, recall as the score, and two rules (no cheating, no hardcoding) that I will refer back to every time I am tempted by a pretty number. No result today, which I know is an odd way to start, but the result is only worth reading if you trust the ruler, and the ruler is the part everyone skips.
Next I will show you the machinery that makes a “capture” mean something: the oracle that reads real bytes, the hash-chained recording that lets me replay any run, and the three tiers I use to keep “the tool did it” and “the model did it” from ever being confused again. That post is where I show you the time I hallucinated a flag and the harness correctly told me I had failed.