Red Teaming AI

How Offensive Security Principles Apply to Machine Learning Systems

8 MIN Read

If you wouldn’t deploy an application without testing it for vulnerabilities, why are you deploying AI systems without doing the same?

INTRODUCTION

Introduction

Red teaming has always been about thinking like an adversary. The value of offensive security work — penetration testing, adversarial simulation, red team exercises — comes from applying attacker logic to systems that defenders have built and tested from a defender’s perspective. The adversary finds things the defender missed, not because the defender is incompetent, but because attacker and defender look at the same system through fundamentally different lenses.

That same dynamic applies to AI systems, and the gap between what standard functional testing catches and what adversarial testing reveals is at least as wide for AI as it is for traditional applications. Wider, arguably, because AI systems have properties — emergent behaviors, sensitivity to input framing, complex decision boundaries — that standard testing methodologies were not designed to probe.

AI red teaming is an emerging discipline that applies offensive security principles to the evaluation of AI systems. It is not a single technique or a standard methodology. It is a practice that draws on the tradecraft of traditional red teaming and extends it with AI-specific knowledge, tools, and attack patterns. This article is a practitioner’s introduction to that practice.

02 / AI SECURITY

What AI Red Teaming Is — and What It Is Not

AI red teaming is sometimes conflated with AI safety testing or model evaluation. These are related but distinct activities:

AI safety testing evaluates whether a model produces harmful, biased, or unintended outputs under normal use conditions. It is concerned with the model’s alignment with intended behavior and human values.

AI security testing evaluates whether a model and its surrounding system can be exploited by an adversary to achieve unauthorized outcomes. It is concerned with how an attacker could manipulate the system to their advantage.

AI red teaming encompasses both dimensions but from an adversarial perspective — actively attempting to find exploitable weaknesses across safety, security, robustness, and misuse vectors, using the creativity and persistence of an attacker rather than the structured checklists of a compliance evaluation.

The distinction matters because safety testing and security testing catch different things. A model can pass safety evaluation with high marks and still be vulnerable to prompt injection attacks that bypass every safety control. A model can be resistant to adversarial prompting and still expose training data through carefully crafted extraction queries. Comprehensive evaluation requires both lenses.

03 / AI SECURITY

The Four Dimensions of AI Red Teaming

1. Safety Testing

Safety red teaming probes the model’s ability to resist producing harmful outputs. This includes testing guardrail robustness — attempting to bypass content filters through jailbreaking techniques, role-playing scenarios, hypothetical framings, and encoded inputs. It also includes testing for bias, discriminatory outputs, and the model’s handling of edge cases that safety training may not have covered adequately.

2. Security Testing

Security red teaming evaluates the model and its surrounding system against adversarial attack techniques. The primary focus areas are prompt injection vulnerabilities in the application layer, training data extraction through carefully crafted queries, model inversion attacks that reveal information about training data, and the security of the infrastructure that serves the model.

3. Adversarial Robustness Testing

Robustness testing evaluates how the model performs when inputs are deliberately perturbed to cause misclassification, unexpected behavior, or degraded performance. In image classification models, adversarial examples — images modified in ways imperceptible to humans that cause the model to misclassify them with high confidence — are the canonical robustness challenge. In language models, adversarial robustness testing examines how the model responds to inputs designed to evade detection systems or cause the model to behave inconsistently.

4. Misuse Testing

Misuse testing evaluates the extent to which the AI system can be weaponized for purposes it was not intended to serve. This includes testing whether the model can be prompted to generate content that facilitates harmful activities, whether its capabilities can be repurposed for offensive use cases, and whether the application provides capabilities that create risk in aggregate even if individual outputs appear benign.

04 / AI SECURITY

Tools and Methodologies

AI red teaming has developed a growing toolkit of both purpose-built tools and adapted traditional security tools:

Garak. An open-source LLM vulnerability scanner that automates probing of language models for a range of known vulnerabilities including prompt injection, jailbreaking susceptibility, data leakage, and hallucination. Garak provides a structured baseline assessment and surfaces issues for deeper manual investigation.

PyRIT (Python Risk Identification Toolkit). Microsoft’s open-source framework for AI red teaming, designed to help practitioners identify risks in generative AI systems through automated and semi-automated testing. PyRIT supports both safety and security evaluation dimensions and integrates with common AI development workflows.

Manual adversarial prompting. Automated tools find known vulnerability patterns. Manual adversarial prompting finds unknown ones. Experienced red teamers apply creativity, domain knowledge, and attacker intuition to construct inputs that automated scanners would never generate. This remains the highest-value component of any AI red team engagement.

Traditional application security tools. AI applications are still applications. API security testing, authentication testing, authorization testing, and infrastructure assessment using standard application security tools remain relevant and necessary alongside AI-specific techniques.

05 / AI SECURITY

Building an AI Red Team Program

An ad hoc AI red team assessment is more valuable than no assessment. A structured AI red team program is significantly more valuable than a one-time engagement. Building that program requires decisions about scope, cadence, composition, and integration with the broader security program.

Who should be involved. Effective AI red teaming requires a combination of offensive security expertise and AI domain knowledge. Practitioners who understand traditional attack techniques but have no AI background will miss model-specific vulnerabilities. Practitioners with deep AI knowledge but no adversarial security mindset will miss exploitation paths that an attacker would identify immediately. The most effective AI red teams combine both skill sets.

Scope and prioritization. Not all AI systems present equal risk. Red team assessments should be prioritized based on the sensitivity of the data the system processes, the access it holds, the decisions it informs, and the consequences of successful exploitation. High-risk AI systems warrant comprehensive red team engagement before deployment and on a recurring basis thereafter.

Cadence. AI systems evolve more rapidly than traditional applications. Model updates, prompt changes, and integration modifications can introduce new vulnerabilities between scheduled assessments. Red team cadence should account for the pace of change in the AI systems being evaluated.

Feedback loops. Red team findings should feed directly into model hardening, prompt engineering, policy updates, and security control improvements. The value of red teaming is not the report — it is the improvement it drives. Organizations that conduct AI red team assessments without a structured process for acting on findings are getting a fraction of the available value.

06 / AI SECURITY

Where Offensive Security Skills Transfer — and Where They Diverge

Practitioners with traditional offensive security backgrounds bring significant transferable skills to AI red teaming: the adversarial mindset, methodology discipline, threat modeling intuition, and the ability to chain findings into impact narratives that resonate with decision-makers.

The areas of divergence are equally important to understand:

• AI attacks are often probabilistic rather than deterministic. A prompt injection attempt may succeed on some model invocations and fail on others. Red teamers accustomed to reliable exploit execution need to adjust their expectations and methodology.
• The attack surface is semantic rather than syntactic. Traditional injection attacks exploit parsing failures. Prompt injection exploits meaning — the way the model interprets and acts on natural language. This requires a different intuition about what constitutes a payload.
• Evidence collection and reproducibility work differently. AI model behavior can be difficult to reproduce consistently, which creates challenges for documenting findings in the format that traditional pen test reports expect.
• The remediation vocabulary is different. Traditional pen test findings map to configuration changes, patch deployments, and code fixes. AI red team findings may map to prompt engineering changes, fine-tuning interventions, architectural redesigns, or access control modifications — remediation categories that the development team needs to understand.

KEY TAKEAWAYS

Key Takeaways

Recap: AI red teaming extends offensive security tradecraft into a new domain. The mindset transfers completely. The techniques require deliberate development.

• AI red teaming covers four dimensions: safety testing, security testing, adversarial robustness, and misuse testing
• Automated tools like Garak and PyRIT provide structured baseline coverage; manual adversarial prompting finds what automation misses
• Effective AI red teams combine offensive security expertise with AI domain knowledge — neither alone is sufficient
• AI red teaming should be a continuous program, not a one-time assessment, given the pace of change in AI systems
• Traditional offensive security skills transfer strongly to AI red teaming; the divergences are specific and learnable

Security / Strategy / Trust

Security isn't just technology.
It's trust.

Exploring identity, security strategy, AI security, and the human side of cybersecurity.