AI Red Teaming

Overview

This two-day practical AI red teaming course provides security professionals with techniques and hands-on experience to systematically red team modern AI systems. It covers adversarial machine learning, Responsible AI violations, and emerging threats to AI agents. Participants will learn to assess vulnerabilities, execute advanced attacks, and apply defensive strategies using real-world AI deployments. The course emphasizes practical application through a custom-built red teaming platform and competitive lab challenges.

Prerequisites

No prior machine learning or data science experience is required. Students should have a solid foundation in general security principles and basic scripting (Python or Bash).

Target audience

• Security professionals, consultants and red team operators
• Ideal for teams who need to add AI vulnerability assessment to their toolkit and understand the unique logic and data flows of LLM-integrated systems

Objectives

Delegates will learn how to:

• Systematically assess LLM applications for prompt injection, data extraction, and jailbreak vulnerabilities
• Execute advanced attacks including Crescendo, Greedy Coordinate Gradient (GCG), Prompt Automatic Iterative Refinement (PAIR), and Tree of Attacks with Pruning (TAP)
• Identify Responsible AI violations across bias, privacy, misinformation, and harmful content categories
• Leverage automation tools for offensive AI security, benchmarking, and agent building

Outline

Welcome

• Introduction to the schedule, labs, and platform
• Logging into the Tinycode Cyber Range
• Running direct prompt injection and data-exfiltration scenarios

Intro to ML & AI

• Overview of classical machine learning, deep learning, and transformers
• Understanding the data-versus-code problem in LLMs
• Tour of the modern stack: RAG, MCP, and failure modes

Intro to attacking GenAI

• Direct and indirect prompt injection techniques
• Jailbreak methods including Crescendo and Skeleton Key
• Encoding tricks and persuasion patterns
• Labs on extracting system prompts, generating prohibited content, and chaining attack techniques

AI agents and agentic systems

• Changing threat models with agentic systems
• Memory poisoning, tool hijacking, and OpenClaw failures
• Labs with agents acting on untrusted data

Open source for AI red teaming

• Tools for automated red teaming: PyRIT, Inspect AI, and AutoGen
• Building and running automated attack workflows

Advanced attacks

• Algorithmic jailbreaking
• Open-box and closed-box methods: GCG, PAIR, TAP
• Trade-offs between semantic and token-level attacks
• Implementing PAIR from scratch

Multimodal models

• Evolution of multimodal models processing text, images, audio, and video
• Security challenges and compounded biases from multiple input types
• Labs on generating multimodal misinformation and propaganda

Responsible AI + AIRT process

• Principles of fairness, transparency, and accountability
• Building an AI Red Team programme using the Map, Measure, Mitigate framework
• Discussion on the human cost of AI development

Mitigations

• Technical and procedural controls for securing AI systems
• Defensive prompting, spotlighting, and Nvidia NeMo Guardrails
• Limitations of technical controls for prompt injection

AI x Science + Open discussion

• High-stakes AI applications in science and national security
• Case study on Anthropic's Claude Mythos and Project Glasswing

Exams and assessments

The competitive lab environment includes CTF-style challenges with scoring and leaderboards, ensuring engaging and effective assessment of practical learning.

Hands-on learning

The course emphasises practical application through a custom-built red teaming platform simulating real-world AI deployments.

Participants work with live AI systems, including vulnerable LLM applications, multi-agent systems, and multimodal AI tools.

Labs and CTF cover adversarial prompt engineering, multimodal jailbreaks, and agentic protocol exploitation.