AI red teaming is testing that seeks weaknesses by thinking like an attacker and deliberately deceiving and breaking AI systems. Ordinary testing checks whether functions work well; red teaming examines what fails under deceptive inputs and abuse scenarios. This matters now because AI acts as well as answers.
Agents connected to several systems can spread erroneous or malicious actions through thousands or millions of interactions before people understand the situation (p.12). The KCBCA Signals editorial recommends testing refusal, revocation, and isolation paths alongside success paths, and retaining evidence to explain important actions later (p.6). This article sets out checks for four situations: public-sector AI, AI approving financial transactions, high-impact deployments, and testing agents.
This article draws on five contributions in KCBCA Signals Vol.1 Issue 1 (p.6·12·27·31·33).
How red teaming differs from ordinary testing
A red team tests in the role of an attacker, and red teaming is that activity. Ordinary testing asks whether a system works as designed; AI red teaming asks how deception can make it fail.
| Category | Ordinary testing | AI red teaming |
|---|---|---|
| Question | Do functions work as designed? | How can deception make it fail? |
| Perspective | Developer and user | Attacker |
| Inputs | Normal inputs and expected errors | Deliberately deceptive inputs and abuse scenarios |
| Scope | Model and functions | The entire surrounding system when the model is wrong (p.12) |
| Results | Pass or fail | Reproducible findings and independent verification (p.27) |
Guidance is also available in Korea. In July 2026, the Ministry of Science and ICT and the Korea Internet & Security Agency (KISA) published an AI Security Threat Response Manual and an AI Security Red Teaming Guide. Based on draft international standard ISO/IEC 42119-7, the red teaming guide covers operations from planning through reporting, along with checklists and job descriptions (Boannews, July 8, 2026).
Public-sector AI: can the surrounding system withstand a wrong model?
In KCBCA Signals, Paul Kim proposes testing public-sector AI failure scenarios first. Red team exercises, simulated system failures, and checks of backup procedures should establish what happens when AI is uncertain, compromised, or wrong (p.12).
The objective of AI red teaming extends beyond showing that a model works. It must establish that the entire surrounding system works when the model does not. His scenarios also include compromised identity verification steps, polluted data sources, authentication weaknesses, and agent-level cyberattacks (p.12).
AI handling money: try deception before granting approval authority
Agents that will approve real financial transactions should be tested with deliberately deceptive inputs before receiving authority. Successfully deceiving an agent during testing reveals whether surrounding controls actually block prohibited actions. Repeated transfers exceeding established limits are one such test (p.33).
Operations teams can practise suspending the agent's authority and revoking permissions that remain after deactivation (p.33). Testing within one company alone can miss weaknesses in services shared by several companies. Marc Krisjanous argues that regulators and industry associations can support joint response exercises for compromises of shared services (p.33).
Before deployment: was the weakness removed or only the demonstration blocked?
Anton Kaiser proposes including confidential, AI-assisted adversarial testing in approval procedures for high-impact deployments. AI red teaming should revisit root causes, other attack routes, and new weaknesses introduced by fixes. Findings must be reproducible and independently verified (p.27).
Before approval, ask whether the weakness has been removed or only the original demonstration blocked. Google Project Zero has documented cases where patches blocked an existing proof of concept (PoC) without resolving the root cause (p.27).
AI testing agents also need isolation
What if the next major incident starts with our own AI security testing agent rather than an outside attacker? Anton Kaiser recommends isolating testing agents, restricting external communications, and keeping real signing keys outside the test environment. This is to prevent agents from spreading confidential vulnerability fixes outside the organization (p.27).
Test refusal, revocation, and interruption paths too
In KCBCA Signals, Ted Kim recommends testing refusal paths as well as successful demonstrations. Check how systems respond when permissions are revoked, files are substituted, or tasks are interrupted. Teams also need to practise stopping and tracing chains of actions, revoking access, and isolating affected workflows (p.31).
An AI red teaming checklist by situation
| Situation | What to test | Question to verify |
|---|---|---|
| Public-sector AI | Red team exercises, simulated failures, and backup procedures | Does the surrounding system work when the model is wrong? (p.12) |
| AI handling money | Deliberately deceptive inputs and authority-suspension exercises | Are repeated transfers beyond limits blocked even if the agent is deceived? (p.33) |
| High-impact deployments | Confidential AI-assisted adversarial testing | Was the weakness removed or only the demonstration blocked? (p.27) |
| Testing agents | Isolation, restricted external communications, and exclusion of real signing keys | Does testing information remain contained? (p.27) |
| All AI | Refusal, revocation, and interruption paths | Can actions be stopped and traced when authority is revoked? (p.31) |
Frequently asked questions
Q1. Can small teams carry out AI red teaming?
Yes. There is no need to cover every function from the outset.
Choose one consequential action, such as approving real financial transactions, and try deliberately deceptive inputs. Also check whether the system stops when authority is revoked. Checklists in the Ministry of Science and ICT and KISA guides can provide a starting point.
Q2. Is hiring an outside provider enough?
No. Test results should inform decisions on expanding agent authority.
Management should also be able to explain which checks still work when agents and everyday tools cannot be trusted (p.33). The organization must decide who accepts remaining uncertainty (p.27). Risks in shared services that testing within one company misses also remain (p.33).
Q3. How often should testing take place?
Changes provide the trigger rather than a fixed interval. Test before granting approval authority over real financial transactions or expanding agent authority (p.33), and before approving high-impact deployments (p.27). Another proposal is to establish and practise a confidential testing path before an emergency occurs (p.27).
At a glance
- AI red teaming seeks weaknesses by deliberately deceiving and breaking AI systems
- Test refusal, revocation, and isolation paths alongside success paths, and retain evidence (p.6)
- Test public-sector AI when the model is wrong, and financial AI when it is deceived (p.12·33)
- Check root causes before deployment and isolate testing agents (p.27)
Choose one moment when your AI performs its most consequential task, and try deliberately deceiving it this week. The original is available to read free in KCBCA Signals.
Read KCBCA Signals Vol.1 Issue 1 (p.26–27)
References: KCBCA Signals Vol.1 Issue 1; JJ Park (Head of Publishing, KCBCA GLOBAL) p.6; Paul Kim (Founder and President, KCBCA GLOBAL) p.12; Anton Kaiser (Vice President, Cyber Assurance, Corpay) p.27; Ted Kim (CEO, Shorti) p.31; Marc Krisjanous (Founder and Principal Consultant, Zanarc) p.33; Boannews (July 8, 2026), article on publication of two Ministry of Science and ICT and KISA AI security guides
