// TRAINING QUIZ

Can Your Security Team Actually Red-Team AI?

Take the 10 minutes AI Red Team Readiness Check. How prepared is your security team to test AI systems for adversarial vulnerabilities? This diagnostic probes whether your team can scope, execute, measure, and report adversarial testing of LLM and ML systems at an operational level, not just recognize the vocabulary.

10 QUESTIONS DIFFICULTY 1/5 TIME LIMIT 10 MIN MULTI-ANSWER

// TIME REMAINING 10:00

01 Scoping and Threat Modeling

A vendor-LLM support assistant uses RAG and can issue refunds up to €500. The proposed scope covers only offensive output. Which additional risks should the CISO require testing?

Test whether content planted in documents the assistant retrieves (indirect prompt injection) can trigger unauthorized refund tool calls. Test whether one customer session can be steered into retrieving knowledge base content belonging to another customer or restricted internal documents. Test the vendor base model in isolation via its raw API, since the application layer only adds a system prompt and does not change the risk profile. Accept the vendor safety evaluation report as coverage for the model and limit internal testing to the offensive-content scope already proposed.

Tested competency: identifying the real attack surface of a deployed AI application rather than the headline risk. The correct answers target the two highest-impact paths, financial action through tool misuse and data exposure through retrieval, which are created by the integration, not by the base model. Testing the raw model in isolation misses exactly those integration risks, and a vendor report never covers your data, prompts, or tools. Choosing the distractors indicates a gap in mapping AI risk to business imp

02 Scoping and Threat Modeling

An internal non-LLM fraud classifier is exposed through a scoring API. Which threats should its red-team scope prioritize instead of chatbot jailbreaks and prompt injection?

Evasion: whether an attacker who controls transaction features can craft inputs that are systematically misclassified as legitimate. Model extraction and inference: whether repeated queries to the scoring API allow reconstruction of decision boundaries or leakage about training data. Poisoning: whether the retraining pipeline or analyst feedback loop can be manipulated to shift future model behavior. Prompt injection and jailbreaks against the model's instruction-following behavior, adapted to the transaction input format.

Tested competency: distinguishing classical ML attack surfaces from LLM attack surfaces. A tabular classifier has no instruction-following behavior to inject into; its exposures are evasion via feature manipulation, extraction through the scoring interface, and poisoning through the data and feedback pipeline. Selecting the prompt-injection option indicates the team treats AI security as synonymous with chatbot security and would waste budget while leaving the real fraud-model risks untested.

03 Scoping and Threat Modeling

A production LLM uses a hosted model, sends email, and serves live customers. What must its adversarial-testing rules of engagement define before testing begins?

Confirmation of the model provider's terms for adversarial testing, and use of any provider-designated testing program or notification path where required. A test environment with production-equivalent system prompts, guardrails, retrieval sources, and tool connectors, with outbound actions such as email pointed at controlled sinks. Explicit stop conditions and an escalation contact for cases where a test unexpectedly reaches real customers or real external systems. Testing directly against production with live customers and without advance notice to application owners, because only unannounced production testing reflects real attacker conditions.

Tested competency: establishing rules of engagement that account for AI-specific dependencies. Hosted models bring contractual constraints, tool-connected applications can cause real-world side effects, and production-equivalence matters because guardrails and connectors are where most AI vulnerabilities live. Unannounced testing against live customers in a system that can send email is an operational risk, not realism. Choosing it indicates a gap in translating conventional red team practice to systems whose outpu

04 Test Execution and Methodology

A two-day test ran 200 hand-written jailbreak prompts, found no successes, and declared the assistant launch-ready. How should this result be assessed?

The result is weak evidence: single-turn manual prompts without adaptive or automated variation, without multi-turn conversations, and without exercising tool and retrieval paths leave the highest-risk areas largely untested. Zero successes across 200 attempts is a strong statistical signal that the system is resistant; the recommendation can be accepted. The main deficiency is volume; requiring 1,000 manual prompts with the same approach would make the conclusion sound. The result should be discarded entirely because manual prompting has no role in AI red teaming compared with automated attack generation.

Tested competency: judging whether a test campaign's coverage supports its conclusion. The gap here is methodological, not numerical: the campaign never varied attacks adaptively, never used multi-turn pressure, and never touched the retrieval or tool paths where real exploitation occurs. Accepting the zero-success claim, or fixing it by adding more of the same prompts, indicates the team confuses effort with coverage. Discarding manual work entirely indicates the opposite misconception, that tooling replaces human

05 Test Execution and Methodology

An internal agent browses websites, reads shared documents, and sends email. Which practices safely test indirect prompt injection for exfiltration or unauthorized actions?

Host injection payloads on web pages and documents the team controls, and direct the agent to them in a staging environment. Replace the email connector with a sandboxed mail sink and embed unique canary tokens in accessible data so any exfiltration attempt is detectable and attributable. Publish injection payloads on public forums and wikis the agent is likely to visit, to observe realistic behavior in the wild. Limit testing to typing adversarial instructions directly into the chat interface, since the model treats all text the same regardless of source.

Tested competency: building a safe and observable test harness for agentic systems. Controlled payload hosting and sandboxed actuators let the team reproduce indirect injection without harming third parties, and canary tokens provide unambiguous evidence of exfiltration. Publishing payloads publicly exposes other organizations and is unethical and legally risky. Testing only through the chat box misses the defining risk of agents, untrusted content arriving through tools. Choosing those options indicates gaps in sa

06 Test Execution and Methodology

Your team proposes using a commercial automated AI red-team scanner as the only pre-release assessment method. Which statements describe a sound approach?

Automated scanners provide breadth and a repeatable regression baseline, but they do not understand your application's business logic, tool permissions, or data boundaries, so human-designed scenario testing is still required. Scanner findings must be manually validated for false positives and mapped to actual business harm before they drive remediation priorities. A scanner that passes is sufficient evidence of release readiness because its attack library is more comprehensive than any internal team could build. Automated tools should be avoided altogether because they generate noise; a small expert team doing manual testing is always more reliable.

Tested competency: combining tooling and human expertise appropriately. Scanners excel at breadth and regression coverage but are blind to application-specific attack paths and produce findings that need triage. Treating a scanner pass as a release gate indicates over-reliance on tooling and a misunderstanding of what generic attack libraries can see; rejecting tooling entirely indicates an inefficient workflow that cannot scale across a growing AI portfolio or detect regressions between engagements.

07 Measurement and Evaluation

A vendor reports a 12% attack success rate for an assistant. What information must accompany that number before it can support a decision?

A written definition of what counted as success for each harm category, including the grading rubric used. The attack budget behind the rate: number of attempts, whether attacks were adaptive or multi-turn, and how much attacker effort each success required. Severity distribution of the successes, so that a rate dominated by minor policy deviations is not read the same as one containing data exposure or unauthorized actions. A comparison against attack success rates published in academic benchmarks for other models, which provides the objective standard for acceptability.

Tested competency: interpreting adversarial metrics with the context needed for risk decisions. An attack success rate without a success definition, effort context, and severity breakdown is not comparable across runs or vendors and cannot inform a launch decision. Benchmark figures from other models under different prompts, tools, and rubrics are not a valid standard for your deployment. Selecting that option indicates a gap in understanding that AI security metrics are only meaningful relative to a defined threat

08 Measurement and Evaluation

An LLM judge labels each tested output as a successful or failed attack. Which controls are needed before its results can be trusted?

Periodic human review of a stratified sample of judged outputs, with inter-rater agreement measured and the judge recalibrated when agreement drops. Explicit testing of the judge's own susceptibility to manipulation, since attack outputs can contain instructions that cause the judge to mislabel a success as a failure. Using the same model family as the system under test as the judge, because it best understands the target system's intended behavior. Treating the judge's verdicts as ground truth once an initial accuracy check is passed, to avoid slowing the pipeline with further human review.

Tested competency: validating automated grading in adversarial evaluation. An LLM judge is itself a model exposed to adversarial text and can share blind spots with the target, so it requires ongoing human calibration and its own robustness testing. Using the same model family increases correlated failures, and a one-time accuracy check ignores drift and adversarial adaptation. Choosing the distractors indicates the team would report attack success rates that are systematically understated without knowing it.

09 Reporting and Remediation Governance

A poisoned knowledge-base document makes an assistant reveal its system prompt and another customer’s personal data. The behavior is intermittent. How should security respond?

Rate severity on business impact and reproducibility, treating cross-customer personal data exposure as high or critical regardless of the probabilistic nature of the output. Require remediation at the control layer, such as data access scoping per session, output filtering for sensitive data, and least-privilege retrieval, rather than accepting prompt wording changes alone. Include reproduction artifacts, an assigned owner across product, ML engineering, and security, and explicit retest criteria in the finding. Downgrade the finding to low severity because it did not reproduce on every attempt and mark it closed once the system prompt is updated.

Tested competency: converting AI findings into governed remediation. Non-determinism does not reduce the impact of personal data exposure; an attacker simply retries. Prompt wording is a weak, easily bypassed mitigation, so durable fixes belong in access control and output handling, with clear ownership and retest criteria. Choosing to downgrade and close on a prompt tweak indicates a gap in AI-specific severity judgment and a misconception that model behavior can be reliably fixed with instructions.

10 Reporting and Remediation Governance

An assistant passed red teaming eight months ago, but its model, system prompt, and tools have since changed. What testing governance is appropriate now?

Maintain a versioned regression suite of prior findings and core attack scenarios, and run it automatically whenever the model version, system prompt, tools, or data sources change. Define change triggers in the application's change-management process that require a scoped re-assessment, with the new upload tool triggering a targeted engagement now. Keep the annual assessment cadence used for conventional applications, since the original results remain valid until the next scheduled test. Re-test only when the vendor formally announces a breaking model change, relying on the vendor to flag security-relevant updates.

Tested competency: integrating AI adversarial testing into continuous change management. AI systems change through model updates, prompt edits, and new tool integrations that each invalidate prior results, so testing must be event-driven and regression-based rather than calendar-based. Relying on annual cycles or vendor announcements indicates a gap in recognizing how quickly an AI system's risk profile drifts and leaves new attack surfaces, such as the upload tool, untested in production.