SFU Cybersecurity
Research Assistant
Overview
AutoSec reads a Java codebase the way an attacker would: it finds a flaw, proves the flaw is real, then closes it. Four LLM agents — Finder, Exploiter, Patcher, Verifier — run in sequence over LangGraph, and my job is to turn that pipeline from something that works on a good day into something that produces evidence a paper can stand on.
What happened
The pipeline chains four agents. The Finder runs IRIS, an LLM-augmented CodeQL, to surface candidate vulnerabilities. The Exploiter tries to actually trigger them. The Patcher writes a fix, and the Verifier rebuilds the project to confirm the exploit is dead and the existing tests still pass. I own the end-to-end runs across the 120 Java projects in CWE-Bench-Java — the scans, exploits, and patches the evaluation is built on all come out of runs I babysit from start to finish.
The part I find most interesting is watching real threat modelling emerge inside the Exploiter. Instead of trusting a static warning at face value, it's handed read-only tools and a source-to-sink data flow and asked to reason about attack surface: it groups traces that converge on the same sink into distinct flows, tags each program point as source, intermediate, or sink, then writes a proof-of-vulnerability test and runs it in a container against both the vulnerable and the fixed commit. A static finding stops being a guess and becomes something the system has demonstrably triggered.
When I started, a full benchmark run couldn't survive its own startup. I rebuilt the reliability contract so it runs unattended: cached scan results are reused by default, with a fallback that re-extracts a project's source when the working tree has gone missing so the build always has something to compile, plus flags to force a fresh analysis or inject cached stages. I also chased down the quieter failures — shell-quoting the analysis command, sharing one typed output shape across the agents — that were breaking long runs halfway through.
The frontier now is patching. Detection and exploitation have become reliable enough to trust; the Patcher is the harder, less-mature stage, still working under uniform hard-coded constraints, and the next step is teaching it to reason per vulnerability instead. We're writing the whole system up as a manuscript aimed at ACM CCS, with the emphasis on reliability, verification, and auditability — results you can actually re-run.