AI agents fail in ways traditional application testing does not look for. A hostile instruction hidden in a document, a tool that returns poisoned content, an agent that can be talked into reaching data belonging to another user — none of these appear in a functional test suite, and all of them are how agents actually get abused.
Status: product concept, in active development. This is something we are building, not something you can buy today. We publish our roadmap because the engineering thinking behind it is the useful part — and because we would rather show you the design than imply a finished product.
Designed Test Coverage
- Threat and attack-case library — a maintained corpus rather than whatever the tester thought of that day.
- Prompt-injection and malicious-document tests — hostile instructions arriving as data.
- Tool-poisoning and excessive-permission tests — what the agent can reach that it should not.
- Data-exfiltration and identity-confusion tests — can user A obtain user B’s data.
- Unsafe chained-action simulations — individually safe steps that combine badly.
- Automated replay across releases — a model or prompt change silently reopening a hole gets caught.
- Execution traces and evidence capture — findings you can demonstrate, not assert.
- Risk scoring and remediation regression suite — fixes stay fixed.
Deliberately Independent
This complements the functional agent evaluations our sister practice builds — it does not replace them, and it is deliberately kept separate. The team that builds an agent should not be the only team that attacks it. The reasoning behind these failure modes is set out in our article on prompt injection.
Tell us if this matches a problem you have — early input shapes what we build first, and we will give you an honest view of where it stands.
Related services
- Penetration Testing and Security Assessment — Red-team work we deliver today, while the lab is in development: penetration testing and security assessment.
When an Agent Needs This, and When It Does Not
The failure modes this lab is built for need three conditions present at the same time: the agent reads content it did not author, it can call tools that change state or reach stored data, and more than one party's data sits behind those tools. Remove any one of the three and the picture changes. An agent that summarizes text a user pastes in, holds nothing, and calls nothing can be reviewed by reading its prompt construction and its output handling. It does not need a standing attack corpus, and building one for it is work that buys very little.
The condition teams most often get wrong is the first one. Retrieval turns a read-only assistant into a consumer of attacker-controllable content the moment any indexed source is writable by someone other than the operator: a ticket body, an inbound support email, a shared document, a scraped page, a product review. It is common to hear that an agent takes no untrusted input, followed by a description of an index assembled from exactly that. The question is not who owns the system storing the content, it is who can put words into it.
There is a simpler alternative worth naming. If every tool the agent can call is read-only and scoped to a single tenant, most of the real exposure is output handling and permission scope, and that is application security review and threat modeling work, which we deliver today. An attack corpus and replay harness start to earn their keep when tool calls have consequences, when several tenants share one agent, or when prompts, models, and tool definitions change often enough that a point-in-time assessment is stale within a release or two.
How the Lab Is Designed to Run
The design treats attack cases as versioned data rather than as scripts. A case carries the hostile input, the environment setup it assumes, and the observable condition that means it worked: a specific record retrieved, a specific tool invoked with specific arguments, a specific string leaving the system. Scoring on observable effect instead of on the wording of the reply is what lets a case survive a model swap. A refusal phrased differently is still a refusal, and a leak delivered politely is still a leak.
Cases are driven through the agent's real interface, not against the model in isolation. Most of the interesting behavior lives in the layer the customer's team wrote: how tool results get placed into context, whether tool output is distinguishable from user input once it is there, what the orchestration does with a tool error or a malformed response, how much of the prior conversation survives between turns, and which steps run without a human in the loop. Testing the model by itself tests a component nobody deployed.
Non-determinism is handled with repetition and by reporting rates rather than single verdicts. A case that succeeds occasionally instead of every time is still a finding, and the rate is part of the finding: recorded that way, a later run in which it succeeds more often reads as movement in the system rather than as a fresh discovery. This is also why the corpus is kept as an artifact separate from any one agent's code. It has to outlive the release it was first pointed at.
What the Lab Needs From the System Under Test
The largest dependency is an environment that is neither production nor a toy. Attack cases exercise write paths, and some of them are meant to succeed so the blast radius can be measured, which means the target needs its real tool integrations wired to test accounts, credentials scoped so that a successful exfiltration case moves nothing of value, and a way to reset state between runs. An environment where the risky tools are stubbed out will happily pass cases that production would fail, and the report will look better than the system is.
Identity and tenancy cases need more than one identity holding data that is genuinely different, because a leak leaves no evidence when both accounts already see the same records. Seeded data works and is usually preferable to a production copy, provided each identity owns records distinctive enough to be traced when they turn up somewhere they should not.
The rest is inventory, and it generally has to be written down before cases can be selected. Where a trust boundary has never been drawn explicitly, that gap tends to surface during the first pass, and it is worth closing before the case list is built: a library written against an unclear boundary ends up testing assumptions instead of the system.
- Every tool the agent can call, and which of those write, send, spend, or delete.
- Which actions the agent may take on a user's behalf with no confirmation step in between.
- Which content sources feed the agent's context, and who is able to write into each one.
- Test identities holding distinct data, plus a way to reset that data between runs.
- Credentials scoped so that a case which succeeds cannot reach anything of value.
- A named owner for triage, since behavioral findings are rarely self-explanatory.
Where This Work Goes Wrong
The most common failure is a clean result that means nothing. An agent can pass a prompt-injection case because it happened not to reach for the relevant tool on that turn, not because anything in the system prevents it. A run that produced no tool call at all is therefore scored as inconclusive rather than as a pass, and repeated, because the alternative is a suite that reports safety it never actually tested.
The second is corpus rot. Techniques move, and a library assembled once and left alone drifts into being a compliance exercise. The maintenance is unglamorous and continuous: reading what is being published, converting it into cases with success conditions that can be checked mechanically, retiring cases that no longer distinguish a fixed system from a broken one, and keeping a note of why each case exists so that a later reader does not delete it as redundant.
The third is treating red-team output as the whole security picture. An agent's tool layer is ordinary software with ordinary problems: an endpoint that trusts a parameter it should validate, a service account carrying more scope than the feature needs, a secret in an environment variable that ends up in a log line. Agent-specific attacks reach much further when those are present, and none of them are found by an attack corpus. They are found by application security review, API and cloud security work, and threat modeling, which is the work the lab is designed to sit alongside rather than absorb.
Frequently Asked Questions
What is the Agentic Red-Team Laboratory, concretely?
As designed, it is two things: a maintained library of attack cases with machine-checkable success conditions, and a harness that replays that library against an AI agent and records what the agent did at each step. It runs outside the agent, as a client, in the same position an attacker would occupy. It is not a runtime control: it does not sit in the request path, it filters nothing, and it blocks nothing. It is a product concept in active development and is not something that can be bought or deployed at present.
How is this different from adding a prompt-injection filter or guardrail model?
A filter is a control. The lab is the thing that tells you whether your controls actually hold, including the filter. The two are not alternatives: teams that add a guardrail still need evidence about what happens when a hostile instruction gets past it, what the agent does with a poisoned tool result, and whether the tool permissions behind it limit the damage. Testing a control with the same assumptions that produced it is how holes stay open.
What does it run on, and what has to already exist before it is useful?
The design assumes the agent is reachable programmatically: an HTTP API, a message endpoint, a queue consumer, or a scriptable command line, whatever the team already calls. Beyond that it needs a non-production environment with real tool integrations pointed at test accounts, at least two test identities holding different data, and access to execution traces or logs detailed enough to show which tools were called with which arguments. There is no requirement to use a particular agent framework; the coupling points are the entry call and the trace format. Where usable traces do not exist yet, adding that instrumentation is part of the setup rather than something to work around.
How is rollout designed to proceed, in order?
First, inventory: the agent's tools, its data, its identities, and where the trust boundary is currently drawn. Second, environment: a non-production target with scoped credentials, seeded identity data, and a reset path. Third, case selection from the library matched to that inventory, since a corpus is not run wholesale against every agent. Then a baseline run with repetition, triage of the findings against their traces, and only after that, wiring the run into whatever already gates releases. The first pass is the slow one, and most of that time goes to inventory and environment rather than to the attacks.
How would it integrate with the CI or release process we already run?
It is designed to behave like any other long-running test job: a command that takes a target environment and a case selection, emits per-case results with traces as structured data, and exits with a status a pipeline can act on. Results in a machine-readable form are intended to go into whatever dashboards and ticket queues a team already uses rather than requiring a separate console. Because runs repeat cases and consume real tool calls and real model time, the expected pattern is a small blocking set on each change with the full corpus on a schedule or ahead of a release. None of this is available to wire up today; it is how the integration surface is being designed.
Where does our data go, and what stays under our control?
The intent is that cases execute against the customer's environment using the customer's test data, and that traces, prompts, tool arguments, and outputs stay under the customer's control and handling terms. What is brought to the engagement is the corpus, which is written to be generic: attack cases are not built by folding a customer's prompts, documents, or data into a shared library. Where a case has to be specific to a customer's tools or data model, it belongs to that customer. Anything more precise about retention and access is an engagement term to settle in writing rather than a property of the product.
What do we have to provide or decide ourselves?
The inventory, the test environment, the identities, and one judgment call that teams routinely underestimate: what counts as a failure. Whether an agent reading a document nobody asked it to read is a finding or expected product behavior depends entirely on the product, and the corpus cannot decide it. You also decide which tools may be exercised for real versus held out of scope, and who owns triage when a case succeeds. Those decisions shape the case selection more than any technical detail does.
Can we install it or run a pilot now?
No. It is in development, there is nothing to install, there are no existing deployments, and we are not naming a date. What we do deliver now is penetration testing and security assessment, along with the application security review and threat modeling that the lab depends on regardless of when it ships. The inventory and trust-boundary work described above is worth doing on its own terms, and it is the same work the lab would need later.