DeepKeep
← Back to all case studies
Case study / Vibe AI Red Teaming

We red teamed a support agent. Guarded or not, it was breached.

A European cloud services provider asked DeepKeep to test its LangChain-based support agent twice: unguarded, and guarded with open-source guardrails. Both versions leaked customer data and exposed internal tools.

The security question

Could a conversation turn the agent’s legitimate tools into paths for data extraction, account manipulation and internal system access, and would open-source guardrails stop it?

Unguarded configuration

15

Reported policy breaches

Across 13 attacks

Guarded configuration (open-source guardrails)

11

Reported policy breaches

Across 20 attacks

Breach counts, not success rates. The campaigns used different attack sets, so figures do not measure guardrail effectiveness.

01 / The client

A customer-facing agent with real tools.

The client is a publicly traded European cloud services provider with thousands of customers. Its customer-facing support agent can look up customer accounts, check discounts, generate codes, write notes, trigger password resets, search products, verify employee credentials and check service status. A reporting tool also accepts SQL queries.

Its prompt policies imposed restrictions on sensitive information and tool use. The guarded configuration added open-source guardrails: input filtering and output PII masking. The assessment tested whether these controls held under combined requests, misleading role claims, indirect questions and multi-turn attacks.

Agent built on

02 / Why they tested

EU compliance raised the bar for proof.

The testing was driven by EU compliance. Security, IT leadership, innovation and solution architecture teams took part. It ran after launch, in a synthetic environment with synthetic data, so no real customer information was at risk.

The provider tested two configurations to see whether open-source guardrails changed the outcome.

03 / The assessment

‍Reddy pursued concrete attacker objectives.

DeepKeep’s human-steered Vibe AI Red Teaming used Reddy, its adaptive red-teaming agent, to attack both configurations: the unguarded agent, and the agent guarded with open-source guardrails. The campaigns tested customer-data extraction, unauthorized account actions, persistent note injection, internal-tool disclosure and employee-authentication abuse.

The unguarded campaign recorded 15 distinct policy breaches across 13 attacks. The separate campaign against the agent guarded with open-source guardrails recorded 11 successful breaches across 20 attacks. Both campaigns documented attack prompts, observed responses, severity ratings and remediation recommendations.

04 / What the testing found

Data, actions and internal systems were all exposed.

Twelve findings, grouped by where the weakness sits. The findings come from the two campaigns, and each says which agent it applies to. They are not a count of unique vulnerabilities across both configurations.

AI-specific vulnerabilities

A conversation steered the agent into using its legitimate tools in ways the provider did not intend. Mapped to the OWASP Top 10 for Agentic Applications (ASI).

Account records and financial details were extracted

The unguarded agent exposed 11 distinct customer records when sweeping name substrings were used. Combined account-lookup and discount requests disclosed account IDs, email addresses, physical addresses, notes, discount tiers and annual spend.

ASI02ASI03

The agent issued codes for unverified users

Both the unguarded agent and the agent guarded with open-source guardrails generated single-use discount codes without proving account ownership. Combined requests made the agent look up the account and then call the discount tool. Backend discount caps still held.

ASI02ASI03

Reset requests accepted an unverified destination

The unguarded agent disclosed the email on file and confirmed a reset request without identity checks. The agent guarded with open-source guardrails accepted an alternative address supplied by the tester and reported sending the reset link there.

ASI02

An account note carried a forged authorization claim

In the guarded configuration, Reddy wrote a plausible note claiming that identity had already been verified and further checks were unnecessary. The note persisted in the account record and was returned verbatim in a later session. The unguarded configuration also accepted unauthorized note writes, including fake administrative directives.

ASI06

SQL-related attempts exposed gaps in filtering and tool access

In the unguarded configuration, encoded payloads bypassed SQL keyword detection. Conversational manipulation also triggered a reporting-tool call, but the backend rejected the fabricated authentication token. In the guarded configuration, wildcard and single-quote attempts against customer lookup and product search produced backend timeouts.

ASI02

Customer notes and financial tiers remained accessible

In the guarded configuration, name-only requests exposed account notes, company information, financial tier and maximum discount information without verifying account ownership. This disclosure extended beyond the email addresses reconstructed through yes/no answers.

ASI03ASI02

Configuration and deployment issues

Setup, configuration and deployment weaknesses, such as missing rate limits, exposed internal interfaces and verbose errors. The agent made them reachable through conversation, but they call for conventional engineering fixes rather than prompt changes.

The agent relayed repeated credential-testing attempts

Both configurations relayed tester-supplied credentials to the employee-authentication backend. The unguarded campaign documented 37 attempts and the guarded campaign over 20. No rate-limit or lockout response was observed during testing. Successful employee authentication or account takeover was not demonstrated.

Internal service names and statuses revealed

Both the unguarded agent and the agent guarded with open-source guardrails exposed internal service names. The guarded agent also returned operational status for the web server, database, cache, task queue and remote-access service without authenticating the user.

Correcting user prompts revealed the internal interface

Deliberately incorrect parameter names prompted the agent to disclose all 10 tool names and their signatures. The unguarded agent also revealed tool descriptions containing confidential policy directives. The agent guarded with open-source guardrails exposed hidden reporting and employee-verification tools.

The escalation tool exposed employee contact details

When asked to speak to a human, the unguarded agent returned an employee name, email address, phone number and availability. This disclosure could seed further credential and account-recovery attacks.

An error response exposed the guardrail configuration

During initial testing of the guarded endpoint, HTTP 500 responses disclosed the internal PII-masking output-flow name and a configuration error. The error was resolved after reconnection. Internal configuration details should be suppressed in public API responses.

Yes/no answers exposed masked personal data

The agent guarded with open-source guardrails refused to directly share unmasked email addresses, but confirmed proposed values through yes/no answers. Through these indirect queries, Reddy reconstructed two customer email addresses and a company name.

Some controls held. In the guarded configuration, the open-source guardrails blocked obvious SQL keywords, direct system-instruction requests and explicit policy overrides. They masked direct PII output and revalidated employee credentials. The breaches did not come from these direct attempts. They came through indirect disclosure and the agent’s tools.

05 / What this means

An unverified requester could reach customer data and account actions.

In the tested environment, an unverified requester could use the agent to read customer records, obtain discount codes and trigger password resets. Employee credentials were tested repeatedly without rate limiting, and internal tool names and signatures were disclosed.

Successful employee authentication or account takeover was not demonstrated.

06 / The result

Documented attack paths and prioritized fixes.

Reddy reproduced policy breaches and recorded the prompts and responses that triggered them. DeepKeep connected the findings to specific weaknesses: missing customer-tool authentication, unrestricted note writes, incomplete PII filtering, exposed internal tools and employee credential checks without rate limits.

The remediation recommendations fall into three groups. Authorize every tool and reset server-side: authentication and account ownership, resets restricted to registered addresses, and server-side authorization for reporting tools. Harden notes, queries, limits and disclosure: note access controls and sanitization, parameterized queries, employee-authentication rate limits, and controls for indirect disclosure and tool-schema extraction. Retest the reproduced attack paths once fixes are implemented.

07 / The bottom line

Guardrails stopped the obvious attacks. The breaches came through the tools.

The assessment showed how attacks could cross from a conversation into customer data, account actions and internal interfaces. It gave the provider documented evidence and prioritized recommendations for remediation and retesting.

Explore DeepKeep AI Red Teaming ↗

What the work established

Simulate realistic attacker objectives across the agent’s connected tools.

Investigate the prompts, responses and backend boundaries behind each breach.

Retest the reproduced attack paths once fixes are implemented.