Security Review at the Speed of Agents

How we built an agentic system that continuously threat models and reviews our service catalog, straight from the code and at the speed engineering now ships

We built an internal agentic security review system we call Prowler, to continuously threat model and assess our production service catalog from the code that ships. It gives the security team evidence-backed findings without slowing development. 

In one afternoon, one engineer with a coding agent can land what used to take a week-long sprint, including new endpoints, new internal services, or a refactor that quietly moves a trust boundary. The capabilities that unlock this speed also make vulnerabilities cheaper for attackers to find and faster for them to exploit. 

Shifting security left has always been the right strategy, because a flaw caught at design review is the cheapest one to fix. The problem is that a design-stage threat model describes intent, not what actually ships, and coding agents widen this gap because production code is more rapidly diverging from original designs. Reviewing deployed code directly has often depended on manual penetration testing, which is too resource-intensive to perform continuously across every service.

We run static analysis everywhere, and it catches what its rules describe: unparameterized queries, dangerous function calls, hardcoded secrets. The bugs that actually hurt us in pentests and bug bounties are different: context bugs, where the code looks fine on its own and the flaw only shows up once you know how the pieces connect. Here are two handlers, recreated from the kind of code we review:

// CODE ARTIFACT #1 (recreated example; not production code)

func (s *Server) GetStatement(ctx context.Context, req *pb.GetStatementRequest) (*pb.Statement, error) {
	acct, err := auth.AccountFromContext(ctx) // account derived from the session
	if err != nil {
		return nil, err
	}
	return s.store.GetStatement(ctx, acct.ID, req.StatementId)
}

func (s *Server) GetTransferReceipt(ctx context.Context, req *pb.GetReceiptRequest) (*pb.Receipt, error) {
	if _, err := auth.AccountFromContext(ctx); err != nil { // authenticated? yes
		return nil, err
	}
	return s.store.GetReceipt(ctx, req.ReceiptId) // fetched by caller-supplied ID alone
}

Both authenticate and pass code review. The second one would return any user’s receipt to any authenticated caller, unless receipts are scoped somewhere else entirely, such as in the store layer, a gateway policy, or the way IDs are issued. Whether this is a critical bug or acceptable behavior depends on facts outside this file. Finding it means answering two questions: 

1. Can an attacker choose receipt_id, and 

2. Is ownership enforced anywhere along the path? 

Answering those questions requires a reviewer to read the authentication framework, handler, and data layer together as a whole system.

Authorization issues can also manifest as skipped validation steps, unhandled failure signals, or inconsistent rule enforcement across handlers. Static rules frequently fail to capture these contextual issues.

The rest of this post is what that system looks like in practice, and what it found.

We set three requirements for the system:

1. Signal Quality: The system must minimize false positives to ensure findings remain actionable for engineers.

2. Auditability: Because we operate in financial services, every finding that reaches a human must carry the evidence trail that produced it. The reviewer needs to be able to challenge the machine rather than take its word.

3. Developer Velocity: A deep assessment takes tens of minutes, so it could not be a CI gate. It had to run continuously in the background on its own schedule.

Within those constraints, we started with an open-source pattern: a team of specialized security agents (a threat modeler, a scanner, an exploitability tracer, a reporter) coordinated by an orchestrating agent, pointed at one service at a time. In the early versions, the whole pipeline was one long instruction document that an orchestrating LLM executed: tracking state, merging findings, tallying verdicts. It worked, and a version ran in production. But the model would occasionally merge two findings that weren't duplicates, or drift a severity between runs. The rules were all written down: checkpoints, fingerprint rules, quality gates. And it followed them almost always. 

That was the problem: we'd asked an LLM to be a state machine.

That led to the principle the current system is built on: Code handles orchestration; the LLM handles judgment. 

If a step has one right answer, such as deduplication, vote tallying, or report assembly, ordinary tested software handles it. The LLM handles steps that need judgment: modeling threats, hunting bugs, tracing exploitability, and verifying claims. Everything that crosses the boundary is validated against a typed schema, and output that doesn't conform is retried against the exact error. This removed silent drift and made runs resumable because every stage had checkpoints.

# CODE ARTIFACT #2 (recreated example; not production code)

STAGES = [
    init_run,                 # code: run dir, checkpoint manifest
    build_profile,            # code: endpoints, auth rules, data access
    agent(threat_model),      # judgment
    agent(scan_registry),     # judgment: known classes
    agent(scan_specialist),   # judgment: open-scope logic bugs
    dedup_candidates,         # code: one case per suspect
    agent(trace_paths),       # judgment: reachability + PoC
    agent(verify, votes=N),   # judgment: independent, blind votes
    tally_and_cap,            # code: majority vote, severity caps
    report_and_diff,          # code: fingerprints, page only on new
]

The stage list above shows where the effort went: most of what follows the two scan stages exists to check their output. Models generate findings much faster than anyone can verify them, and if verification doesn't keep pace, the backlog just moves to the humans reading the output.

Prowler runs against the service catalog on a continuous schedule. A scheduler ranks services weekly based on exposure tier, recent change velocity, and time since the last assessment. High-velocity services are prioritized so that recent code changes are reviewed quickly. For each selected service, the stages run in order.

Profile, then model. An extractor builds a structured snapshot of the service in seconds, including endpoints, authentication rules, data access patterns, and external calls. Then, a threat-modeling agent turns that snapshot into a ranked context about external reachability, trust boundaries, and the flows that deserve scrutiny. Threat models are cached and carried forward between runs, so knowledge of a service accumulates. Every run re-derives the model from what shipped, keeping the threat model aligned with the implementation.

Scan with two different approaches. A registry-driven agent hunts known vulnerability classes using curated guidance about how each class appears in our environment. That guidance includes patterns the platform already protects so the agent doesn't flag them. In parallel, an open-scope specialist reasons about the service's business logic without relying on the registry. We added the specialist because a closed list has the same ceiling as static analysis: anything outside the list is invisible. Simply expanding the list would dilute the focused guidance on our most critical and common bug classes. The open-scope pass therefore scans for issues that don't fit an existing guideline. When it repeatedly finds the same bug pattern, that pattern can become a registry entry. The registry is plain, reviewable YAML, with one entry per vulnerability class and guidance about patterns the scanner should not flag.

# CODE ARTIFACT #3 (recreated example; abbreviated registry entry)
id: authz-object-level
class: broken object-level authorization
severity_ceiling: critical
hunt_for:
  - handler fetches by caller-supplied ID with no binding to the
    authenticated principal
  - ownership check present, but only after side effects have fired
safe_patterns:        # platform-enforced; do not flag
  - data-access methods that bind queries to the authenticated account
  - routes protected by a separately verified elevated-access policy
evidence_required:
  - the fetch call site AND the auth-context derivation, both cited

Trace what an attacker can actually do. Deduplicated candidates get an exploitability trace. An agent follows the path from the public edge, across service boundaries, to the flagged code and determines whether someone on the internet with a valid user session can trigger it. If so, the agent emits a proof-of-concept request that gives the triage team stronger evidence.

# TRACER OUTPUT (recreated example; not production data)

Finding: IDOR: GetTransferReceipt
Endpoint: GET /api/v1/receipts/{receipt_id}
Auth: Bearer token required (any valid user session)
Parameter taint: receipt_id; user_controlled (URL path, no ownership check)
PoC: curl -s 'https://api.example.com/api/v1/receipts/<VICTIM_RECEIPT_UUID>' \
       -H 'Authorization: Bearer <ATTACKER_SESSION_TOKEN>'

Verify adversarially. With the trace evidence attached, each candidate goes to an independent verifier that starts from the assumption that the scanner is wrong. It re-derives the claim from source, cites the call sites it read, and looks for explanations that would make the finding a false positive, such as compensating controls, policy enforcement, feature gates, or dead code. High-severity findings receive multiple independent votes, and code tallies the majority. Verifiers can also conclude that the bug is real but the claimed attack path is wrong. That verdict sends the finding back for analysis instead of producing a misleading report or discarding a real issue. Each vote is a structured, auditable artifact that becomes part of the evidence trail.

// CODE ARTIFACT #4 (recreated example; one verifier's vote)
{
  "finding_id": "authz-object-level-03",
  "verdict": "confirmed",        // confirmed | refuted | real_but_wrong_path

  "call_sites_read": [
    "example/handler.go:41",
    "example/store.go:118",
    "example/auth_context.go:77"
  ],
  "false_positive_checks": {
    "policy_enforcement": "absent; route not present in the relevant policy",
    "feature_flag_gate": "absent",
    "compensating_control": "none found on the traced path"
  },
  "reasoning": "Handler forwards req.ReceiptId to store.GetReceipt without binding to the auth context; the store applies no ownership filter."
}

Put together, this is what one of those nightly runs looks like from the operator's seat:

Two moments from real runs show what this looks like in practice.

In one assessment, the registry scanner looked very similar to the entry above. An account-data service routed its account-scoped queries through one small helper whose job was to restrict results to the caller's own account. The scanner flagged the helper because, on two of its code paths, it accepted whatever account identifier the caller supplied without checking it against the authenticated user. On its own, that reads like a minor finding in one function.

Deduplication grouped these paths into a single finding targeting the root cause. Exploitability tracing showed that 15 endpoints relied on that helper for access control. Because the helper failed to validate identifiers, a request could bypass the intended scoping on any of those endpoints. A single fix in the helper function secured all 15.

In another assessment, the open-scope specialist caught a bug with no established pattern. In a change-management flow, changes must pass review and record an approval before they take effect. The server modeled the workflow as a sequence of steps, but the handler read the caller's progress, including whether the review had passed, from the request body. It did not consult the server's record. A caller could assert that the review step had happened, jump to execution, and apply the change without a passed review or recorded approval.

# CODE ARTIFACT #5 (recreated example; not production code)

def apply_approved_change(request):
    state = request.json["workflow_progress"]    # progress supplied by the caller
    if state["review_passed"]:                   # the caller vouches for itself
        record_signoff(state.get("signoffs", []))
        execute_change(request.user)             # server record never consulted

Because the handler contains no dangerous function calls and no identifier fetched without a check, traditional static analysis would not flag it. For findings like these, engineers receive tickets with evidence attached. The automated loop ends at triage, where a person reads the finished finding and decides what to fix. The security team's focus shifts from performing individual assessments to reviewing output and tuning system rules. We are also testing an automated triage agent that re-evaluates backlog findings, which may further shift engineers toward decision-making rather than manually sorting findings.

Prowler replaces a review model built around moments in time, which means threat models refresh automatically from source code, and security assessments run continuously across critical services.

The full loop (rank, scan, verify, file) has run nightly without human intervention for the past several months, assessing production services and filing real findings.

Over 90% of the services in our two highest-criticality tiers have now been assessed. For most, this was their first deep security assessment.

The verification machinery also lets us grade ourselves. Another independent agent re-derives the findings, arguing each one for and against. So far, 92% have been confirmed at root cause, and the findings human reviewers had already dismissed were rejected here too. An experimental fix agent has started proposing remediation pull requests for confirmed findings, which go back to the owning engineering team to review and fix. 

The strongest evidence so far has come from convergence with human testing. In one case, Prowler found the same flaw that our pentest team had proved independently, with no coordination between the two efforts. In another, an issue raised through our internal reporting channels was already filed as a recent finding by Prowler.

One limitation is that these results do not measure recall. Measuring recall requires a corpus of bugs we missed. Findings from penetration tests and bug bounty reports serve as ground truth, helping us build test cases to improve future detection.

Prowler fills the gap between dependency scanning, secret detection, and the creative adversarial thinking of a human pentest. The environmental context it accumulates includes cached threat models, platform safety patterns, and knowledge refined by triaged tickets.

The architecture enabled autonomous operation. Most of the reliability came from two design choices. We split the work the way a security team does (into threat modeling, hunting, tracing, and verification) because each stage has different failure modes. 

We also moved orchestration out of the LLM. When an LLM held the pipeline's state, a silent misstep created a human troubleshooting session and prevented unattended operation. 

The living threat model has lasting value. We set out to catch vulnerabilities and also built a current, machine-resolution threat model of every service. While individual findings are resolved, the underlying models reduce computational overhead on subsequent runs and improve verification accuracy.

A system's context defines its boundary. The clearest example of this came from a finding we closed. Prowler flagged an endpoint that accepted a caller-supplied identifier and returned the data scoped to it without binding that identifier to the authenticated caller. The verifier confirmed the missing check but declined to confirm exploitability. Triggering it required two unguessable identifiers and a pre-existing relationship between them, and our codebase contained no visible way to obtain a valid pair. The verifier recorded the evidence that would change its verdict, recommended adding the ownership check as defense in depth, and the finding was closed as not exploitable in practice.

Weeks later, a human penetration test confirmed the finding. The missing evidence existed in another repository, which the pipeline could not read. 

The scanner found the bug, and the verifier documented its uncertainty without overstating the evidence. The final outcome was still wrong because the system lacked relevant context. The roadmap therefore prioritizes expanding what the pipeline can read. We also retain findings that aren't exploitable today because future code or previously unavailable context may complete the attack path.

We are building an attack-path memory system based on these context boundaries. Security teams often close low- and medium-severity findings, especially those that are not currently exploitable, because a human team cannot carry hundreds of unresolved possibilities. Prowler preserves them in a central findings store. Each finding becomes a structured record with its evidence, identity, and history across runs. When code changes anywhere in the catalog, the system can ask what new attack path may now be possible. A PR review that knows the service carries an open low or medium finding could flag the change that worsens it, such as a new endpoint that exposes sensitive data through a flaw that previously returned nothing sensitive. This would allow us to anticipate vulnerabilities that are one code change away before they become fully exploitable.

Supporting this system requires broader pipeline integration. Prowler is the find layer, and every finding moves through discovery, triage, and remediation.

Design review remains essential because catching a flaw while it's still a diagram is usually the cheapest fix. Continuous review supports it by staying close to the source of truth: the code that ships.

Thank you to the teams across our Security organization who helped build, tune, and operate Prowler. 

Vikram Ashtaputre is on the Application Security team at Robinhood, where he focuses on securing the codebase, keeping new features from widening its attack surface, and scaling application security reviews with agentic workflows.  

“Prowler” is a proprietary internal workflow solution developed by employees of Robinhood. It is not affiliated with or related to any external technology tools.