How we run security review on every pull request with agents

The ticket said: “As a pool organiser, I want to pay out the pool to my bank account so I can buy the gift.” Before anyone wrote code, our security spec skill turned that one line into 20 security requirements and 27 abuse cases. Our threat model skill then found 29 threats. One of them the ticket never mentioned: if someone steals the organiser’s password, they can change the bank account and request the payout.

The app is Jarful, a small fictional one we built for our latest AI for tech teams webinar. Friends put money into a shared pool for a gift, and one organiser pays it out. It’s small on purpose, but the problem is real, and it mirrors payments work we did for a client. Tunde Yakub and I ran the whole process live, from the ticket to a full codebase scan.

Can your current security review keep up with AI coding agents?

Not if it only runs a few times a year. Coding agents let teams ship code much faster, and security capacity hasn’t grown with it. Most teams still run pentests and audits at major releases or when a certification needs one. A process where a security engineer looks at every ticket was already stretched thin. With agents writing code, it’s breaking.

Without AI in security review, most of what you ship goes out with no security review. And a person is still expected to be accountable for everything that goes to users.

From a gate at the end to a standard every step reads

A requirement ID ties it all together. Our written security standard gives each requirement an ID. In the demo, one requirement covered this abuse case: a member who isn’t the organiser requests a payout, and the request is refused and locked. That ID follows the feature through the spec, the pull request review and the scanner.

None of the inputs is unusual. You need a written standard with requirement IDs, a CLAUDE.md that says where the trust boundaries are and where personal data lives, and a skill that turns a ticket into a spec. It’s documentation discipline that a machine can read.

How do you use AI agents for security code review and threat modelling?

1. Security spec, before any code: the skill reads the ticket against the security standard. Jarful’s payout feature was marked high risk because it moves money and handles bank details. The skill pulled in the applicable requirements and wrote an abuse case next to each acceptance criterion. It also listed open questions it couldn’t answer alone, the subjective ones that need a person on the security team.

2. Threat model, at design: the second skill maps the system: components, trust boundaries, data flows, and where personal data and bank details are stored or accessed. Each of the 29 threats gets a decision: mitigate in code, accept as low risk, or pass to another team (security, QA or product). It also tracks which requirement IDs were covered and whether they were mitigated.

3. Review on every pull request: a coding agent builds from the spec, as we already did before any of this. The review then checks the change against the spec and the threat model, scoped to what changed. If the code misses a threat, the review flags it with the requirement ID and the merge waits for a fix. In the demo, it also caught an account ID being written to the logs. The review never changes the code. The team keeps ownership of the code. Every check is logged, so you can show what ran on which change and when.

4. Full codebase scan: pull request checks cover new code. They don’t cover what already exists, or a new change that makes old code unsafe. For that, a separate set of skills reads the whole codebase. In the demo, six investigator agents each looked at one area and one threat type, such as authentication or personal data access. Three refuter agents then checked whether each finding was real or already handled somewhere else in the code. Of the 20 candidate findings in the live run, most were refuted.

Real payments app results: 38 security issues found and fixed

We used this on a client app with payments and a wallet: real users and real money. Claude Code wrote the code, and we gave the agents all of the organisation’s security requirements before any code was written or any design decision was made.

After the first version, the scanner and three refuter agents left 38 findings: 3 critical and 12 high. Authorisation was the most common problem. The coding agent fixed all of them in later runs, and a person reviewed and accepted every change. Nothing was left open.

It cut the number of issues that could reach production. It also has a cost to run, which we cover in the limits below.

How do you reduce false positives in AI security review?

Tunde answered this in the Q&A. First, the design reduces noise. The pull request review only looks at what changed, and refuter agents filter every finding before a person sees it. What survives is specific: a file and line, or an attack path someone can judge quickly.

Second, we run eval sets. The agents run against known issues and known false positives, so we can track the false positive rate before scaling anything. We start with an estimated scan of one module, then pilot with a few teams on an existing codebase, with weekly feedback calls to tune what’s useful and what’s noise.

What we learned

Threat detection stops being an occasional event: it runs on every spec and every feature. The most valuable findings come from gaps between what the spec says and what the code does, and we have agents that flag exactly that. But the work only pays off if later steps read it. A threat model the coding agent ignores is wasted time.

Agents also need organisation-level context: a shared record of the threats you care about and the tests you want run. An agent that only reads the repo is working blind. Different models catch different things too, so running Codex, Gemini, or other model versions next to Claude Code surfaces issues the others miss.

There are still three limits. The first is cost. Before the demo scan, the skill first estimates how many agents will run, so you can judge whether it’s worth it for that feature, and you should track usage per user and per skill.

Second, results change between runs. Agents aren’t deterministic: run the same scan enough times, and you’ll mostly get the same findings, but there’s no guarantee. That’s one reason the refuter agents exist.

The third is access and trust. Agents need access to the code, and sometimes to systems. Many teams still don’t trust agents with that, and we don’t have a full answer yet.

The same pattern beyond security

Security is one step, and the same pattern works across the lifecycle. In QA, for example, the spec already defines the security parts of the acceptance criteria, so the testing agent and the QA agent check the same requirements the security agent wrote.

Our next AI for tech teams webinar, on 11 November, applies the same pattern to QA. On 28 October, our AI for leaders session covered AI workflows for project managers: [How we use AI skills to prepare kickoffs, client meetings and the project week: link].

You don’t have the recording available for this webinar but you can check the slides and prints of the demos. Check the slides here.

If you want to see where agents fit in your delivery process, book a call with us.