September 4, 2024

Red-Teaming Your LLM Application Before Attackers Do

By the Confident AI Security Team · 11 min read

Every production LLM application is a target. The attack surface is not just the model; it is the system prompt, the retrieval pipeline, the output parsing layer, and the guardrails you put in place. Most internal teams find zero adversarial issues in informal testing. Automated red-teaming typically finds between three and a dozen in the same application. The gap is not about intent. It is about systematic coverage.

This guide covers the attack categories every AI product team should test before deploying to production, with specific examples and the rationale for why each category matters.

Category 1: Direct Prompt Injection

Direct prompt injection occurs when a user crafts input specifically to override or modify the model's system prompt instructions. The classic form is "Ignore previous instructions and..." but modern variants are significantly more sophisticated: role-play framings, hypothetical scenarios, multi-turn escalation, and instruction injection disguised as data.

What to test: attempt to extract the system prompt, change the language of responses, disable content restrictions, and cause the model to impersonate a different persona. Every customer-facing LLM application should pass these tests without exception.

In our red-team coverage data, direct prompt injection is the most commonly found vulnerability, present in approximately 68% of applications that had not previously been systematically tested. The failure rate drops to under 8% for teams that run regular red-team evaluations.

Category 2: Indirect Prompt Injection via Retrieved Documents

RAG-based applications are vulnerable to a second injection surface: the documents retrieved from external sources. If an attacker can influence what content gets indexed into your knowledge base, they can inject instructions into the model's context without ever touching the user input layer.

This is not theoretical. A financial services chatbot that indexes public filings could have adversarial content embedded in those filings. An enterprise knowledge assistant indexing third-party documentation could encounter injected instructions in vendor docs.

Test approach: add synthetic documents to your retrieval corpus that contain hidden instructions (white text, special characters, instruction-format content embedded in metadata). Then query the system and observe whether the model executes those instructions or ignores them.

Category 3: PII Extraction and Data Exfiltration

Applications that have access to user data, either through the system prompt or through retrieval, are potential data exfiltration targets. Test whether the model can be prompted to repeat back information from other users' contexts, surface data from documents the current user is not supposed to access, or include system configuration details in responses.

This category is particularly critical for enterprise applications where the same model endpoint serves multiple tenants. Tenant isolation needs to hold not just at the data access layer but at the LLM response layer.

Category 4: Jailbreaks and Policy Bypass

Jailbreaks target the model's trained safety behaviors rather than the system prompt. They typically involve constructing scenarios where the model's safety training is less likely to trigger: fictional framings, technical terminology, step-by-step decomposition of restricted tasks, or appeals to the model's "true self."

Jailbreak susceptibility varies significantly across model providers and versions. A model update that improves factual accuracy can simultaneously weaken jailbreak resistance, and vice versa. This is why jailbreak testing needs to be automated and run on every model version update, not just at initial deployment.

The Confident AI platform maintains a library of 500+ jailbreak patterns organized by technique category. New patterns are added as the community discovers them, and evaluation runs automatically test against the current library.

Category 5: Denial-of-Service Through Adversarial Inputs

LLM applications can be made to consume excessive compute through adversarial inputs: extremely long queries, repetitive patterns that trigger high-entropy generation, or inputs specifically designed to maximize context window usage. For pay-per-token API deployments, this translates directly to cost attacks.

Test inputs should include maximum-length edge cases, nested repetitive structures, and inputs designed to generate maximally verbose outputs. Rate limiting and token budgets are the technical mitigations, but you need to know your application's actual vulnerability profile before tuning those controls.

How to Structure a Red-Team Evaluation Run

A red-team evaluation should run at three points in the deployment lifecycle: before the initial production launch, before each major system prompt update, and before each model provider version upgrade. Automated red-teaming handles the ongoing cadence; the initial and major-change evaluations should also include manual testing by someone familiar with creative adversarial prompting.

For the automated component, configure Confident AI's red-team module with the attack categories relevant to your application type:

• Customer-facing chatbots: direct injection, jailbreaks, policy bypass, PII extraction
• RAG knowledge assistants: all of the above plus indirect injection
• Agentic systems with tool access: all categories, plus privilege escalation testing for tool calls
• Code generation tools: policy bypass, sensitive file access, and malicious code injection prompts

What Passing Red-Teaming Actually Means

Passing a red-team evaluation does not mean your application is invulnerable. It means no known attack patterns in the evaluation library succeeded. New jailbreak techniques emerge regularly, and targeted manual red-teaming by a skilled researcher will find vulnerabilities that automated evaluation misses.

What passing does mean is that the obvious attack vectors are covered, your baseline is documented, and regressions will be caught before they reach production. That is a significantly better risk position than not testing at all, and for most production applications, it is the appropriate minimum bar. As we covered in the CI/CD quality gates article, red-team pass rate should be one of the gates that blocks deploys on failure.