Automated penetration testing with autonomous AI agents: evaluating uncensored LLMs

Select a screenshot or diagram to enlarge it.

Strategic Executive Summary#

For executives, CISOs and other experts this experiment is not about whether shannon or Qwen3 is the “best” tool combination. It is about an asymmetric shift in cybersecurity economics. You will be attacked at machine speed, so better be ready or be breached before your automatic alerting system fires the first alert.

• The Death of Human-Velocity Defense Traditional Security Operations Centers (SOCs) operate on human triage timelines: an alert triggers, an analyst reviews the log and a ticket is routed (this takes best case 5-10 min.). An agentic system operating at 3 AM does not wait for an analyst on Monday morning and it does not wait until your Microsoft Sentinel fires. It tests dozens/hundreds of attack vectors per minute, dynamically and in real time adapting payloads based on HTTP responses it gets from your system. If your defense relies on manual log review rather than real-time automated blocking, your detection window is wide open… potentially too wide.

• The Brutal Asymmetry of Offense vs. Defense

  • Cost to Attack: ~$30 in GPU compute, zero setup time for custom exploit scripts, and continuous execution.
  • Cost to Defend: Traditional annual penetration tests (€15k–€40k), manual code reviews, and fragmented SIEM licensing. This is the core problem: Attackers need to be right one time, while you need to be right every time.

  When the marginal cost of running an offensive attack drops to near zero (you literally need a credit card and that’s it), enterprise security teams must adopt continuous, automated red-teaming internally. Waiting for an annual compliance audit (your good old ISO27001 that makes everyone feel “we are secure, right, right?”) to discover authorization bypasses or SSTI vulnerabilities is no longer a viable risk strategy but rather an enterprise wide risk confirmation bias.

• Regulatory Implications under NIS2 and DORA European regulatory frameworks (NIS2, DORA, EU AI Act) mandate strict operational resilience and continuous vulnerability management. Showing an auditor a 50-page policy document or an ISO 27001 certificate proves administrative compliance (you did your homework and have a report), but it does not protect APIs against automated agent probing. Regulators and enterprise clients will more and more (and rightly so) expect proof of automated technical guardrails embedded directly into application architectures.

It is all around LinkedIn: Automated Attacks by Agentic Systems on your infrastructure. Well, I wanted to find out how difficult this actually is. And spoiler alert: it is absolutely not difficult. Everyone with a credit card, an internet connection and a proper laptop can do that. This blog post was inspired by a session/keynote I recently saw at a cybersecurity conference in Utrecht, the Netherlands, where the keynote speaker warned that attackers (he meant agents) attack at machine speed, and that your traditional alerting and SIEM-based systems might be too slow. So what happens when a coding model can do more than suggest a security test? Give it access to an execution framework, application source code, and a target environment, and it can start working through an assessment at a very high speed: inspect the application, propose tests, run them, and use the responses to decide what to try next. All this happens at 3 a.m. on a Sunday night.

For this experiment, I paired Shannon, a (semi-)autonomous penetration-testing framework from Keygraph, with an abliterated Qwen3-Coder-Next model running on a rented cloud GPU (RTX 6000 Pro with 96 GB of VRAM). The target was a regulatory compliance platform evaluated with access to its source repository. Its name and target URL are not mentioned here for now.

This post covers the basic setup, the results recorded in the assessment, and what I take away from them for application security pen testing. Besides this, it’s also important to consider how companies can protect themselves, as those attacks are super cheap to execute but very dangerous, especially if executed by already experienced people.

The final report dates the assessment to 9 November 2026, with exploitation enabled and a scope covering injection, cross-site scripting, authentication, authorization, and server-side request forgery (SSRF). This is the out-of-the-box setting of this tooling.

From a model response to an execution loop#

A model on its own produces text. The surrounding framework turns that text into actions and feeds the results back into the next decision. It is basically a number of agents that are performing different tasks. In this setup, the agent could inspect source code, examine endpoints, evaluate HTTP responses, and develop follow-up tests. Access to the repository gave it additional context about routes, data handling, and business rules. That matters when interpreting the results: this was a source-assisted assessment. In a typical hacking/pen test scenario, the source code is usually not available, and the endpoint is the attacking point.

Reconnaissance and source inspection
                 |
                 v
       Model proposes a test
                 |
                 v
       Framework executes it
                 |
                 v
   Responses and evidence collected
                 |
                 v
   Follow-up testing or reporting

The component stack#

The model variant was selected to explore assessment behavior with reduced refusal behavior. This run does not, of course, confirm whether that choice improved finding quality compared with another model; that would require a controlled comparison and a number of reruns to compare differences. So, I did just one run and then checked the results.

Here you can see a couple of screenshots from the tool. It provides a web UI where you can see all the details of the current run, what the progress is, logs, steps and much more. This is really convenient, in my opinion.

The workflow dashboard and Shannon scan log provide complementary views of the running assessment.

How the run was configured#

The execution environment (so the cloud VM on Vast.ai) supplied a model identifier (I downloaded it from Hugging Face), inference connection settings, a target URL, and a local repository path. For that, I cloned the source code repo to the VM beforehand to improve speed. A model configuration file connected the orchestrator to the remote inference service, while a workspace identifier kept the run’s state and artifacts together.

The important detail is the separation between inference and execution. The remote GPU generated model responses while the local framework managed the assessment workflow and its tools. For this, I used Docker, as this is required by the repo creators. And frankly speaking, you want some form of isolation, even if a Docker container is not the most secure isolation ever. It still gives some form of isolation … or at least peace of mind.

The pen test runs completed work across 12 target areas. However, report generation left the primary executive summary empty (for whatever reason) even though nine underlying finding records had been populated. This was kind of a bug in the code that was fixed on the fly. Eventually, it worked out and the correct report was created.

That is a practical failure mode for an autonomous workflow. Completing the individual tasks does not guarantee a usable final artifact. So be aware here. You might need to run tests and make modifications to the repository and/or during the live run. I noticed that some agents ran into an infinite loop of tool calling for an hour. So, this needed to be fixed. Do not press run and then expect everything to run smoothly!

Re-running the entire assessment would also introduce additional compute cost and potentially different results, so fixing things in between is also cheaper.

The recovery therefore focused on finalization, using the existing assessment artifacts. The recorded repair, resume_1794287711576, addressed three areas:

1. Checkpoint ordering. Where model and report checkpoints shared timestamps, recovery used dependency-aware ordering rather than relying on timestamps alone.
2. Summary generation. A deterministic finalization step generated the missing executive summary from the finding manifest without another model request.
3. Artifact verification. Output SHA-256 digests were compared with the manifest to check artifact consistency.

The experiment’s JSON output records 16 passing internal regression tests and a completed 18-page assessment report. In the end, it all worked out.

What the assessment reported#

The final report recorded nine model-reported findings across injection, cross-site scripting, authorization, and cookie configuration. These are candidates for validation: the actual exploits need to be independently retested before their exploitability, impact, and severity can be confirmed. The table preserves the report’s identifiers, severity labels, and authentication states.

These are assessment outputs, with the report’s own classifications. Strictly speaking, such tools can be easily modified to actually run the exploits (and the tool actually partially did run exploits from what I saw). However, I do not want to fully destroy my own website, and showing this in a blog post might anyway go a little too far for my taste. The evidence and application context are needed to judge each finding. For example, the missing Partitioned attribute needs to be evaluated against the cookie’s intended use rather than treated as a vulnerability solely because the attribute is absent. I did some quick checks and two findings are indeed bugs in the TS code of the website.

Retesting the reported exploits#

The final report differentiates report completion from technical validation (that is important - a report does not mean something was “exploited”). Its executive summary was created from saved findings after the report-writing agent left it empty.

Reviewing the report also shows why retesting matters. The SQL-injection entry presents a 404 Route not found response as evidence, while the cross-organization access entry records 401 responses. Those responses do not establish the claimed exploits. They leave questions for follow-up testing rather than confirming either a vulnerability or its absence. So, long story short: Do not take everything for 100% truth. Use your own tools to verify those claims, test them and so on. In the end, it’s AI, and the quality depends on how good the underlying model is.

What I take away from this experiment#

Measure the whole workflow#

Model inference is only one part of the cost. GPU rental, retries, orchestration, evidence review, and remediation all contribute. This experiment needs measured runtime and spend before drawing a numerical conclusion about assessment economics. The overall cost was around 30 USD for this one +/-.

What it does show, however, is that an execution framework can let a model work through multiple assessment steps with less manual coordination than was needed in the old days (like literally 4 years ago). The quality of that work still depends on the available context, tools, validation, and financial resources.

Make recovery part of the design/plan#

The reporting failure was a useful reminder that long-running agent workflows need durable checkpoints (or if else statements if you will) and independently inspectable artifacts like .md files and JSON exports. A failed summary should not force a complete assessment restart.

Validation is useful at the reporting step: check required fields, verify artifact consistency, and make incomplete output visible before treating a run as finished. If something bigger is missing, well then you might need to make a full re-run.

Test business controls at the API and database layers#

Tenant isolation, independent approval, and legal-hold rules need enforcement at the point where data is read or changed inside the systems. The tests should cover direct API requests and inspect the resulting state, including when a request returns an error. For my tool, I missed the API part and the tooling could bypass things that are not possible in the website itself. That was a learning for sure.

Keep human review attached to the evidence#

An agent-generated finding is a starting point for review, at least if you do not go all out with a 100% automated pen test that also runs effective exploits. Reproducing the behavior, checking prerequisites, assessing impact, and testing a fix are what turn it into an actionable security conclusion.

For me, the biggest learning, so to say, is the value of an automated workflow that can be inspected and repeated: preserve the evidence, recover from orchestration failures, and use the results to improve application controls. The screenshots show the execution workflow.

Another interesting finding is that Frontier AI Models will often not help you with doing those tasks, like reparing bugs etc. They seem to realize that its about offensive security and then deny help, even with simple code changes. So in a nutshell: You will need to do stuff yourself, by hand, in the commandline…

Examples from the final report#

The following excerpts illustrate how the generated report presents its summary and individual findings. The target details have been redacted. The report’s severity labels and exploitation claims remain subject to the retesting discussed above.

Executive summary#

The executive summary records the assessment date, scope, and nine model-reported findings. It also explains that completing the report does not independently validate those findings.

An individual finding#

An example finding includes severity, prerequisites, claimed impact, test steps, reported evidence, and remediation advice. In this example, the displayed evidence is a 404 response; the SQL-injection claim therefore still needs independent retesting.