AI Incident Response: When Your AI System Leaks Data

TL;DR — AI incident response requires a plan tailored to AI-specific threats: prompt injection, data exfiltration through model outputs, hallucination-induced data leaks, and model misuse. Five severity tiers from S0 (platform-wide outage) to S4 (minor policy violation). Key steps: detect (behavioral monitoring, output filtering), contain (kill switches, model rollback, session revocation), investigate (audit trail analysis, root cause), remediate (patch, retrain, update filters), and review (post-incident report). The NIST AI RMF MANAGE function (MANAGE 4.1-4.3) requires incident response plans, recovery procedures, and incident communication. Microsoft's April 2026 guidance notes that AI breaks traditional IR models because the root cause is a probability distribution, not a line of code.

Traditional incident response follows a well-established playbook: detect the breach, contain it, investigate, remediate, and review. The steps are clear because traditional systems are deterministic — a vulnerability is either exploited or not, and the root cause is a specific line of code or configuration.

AI systems break this model. A Microsoft Security blog post from April 2026 puts it directly: "A model may produce harmful output today, but the same prompt tomorrow may produce something different. The root cause is not a line of code; it is a probability distribution shaped by training data, context windows, and user inputs."

AI incidents are also harder to detect. A prompt injection attack doesn't trigger a firewall alert. A hallucination that reveals sensitive training data looks like a normal model response. Data exfiltration through a model output doesn't show up in database access logs.

This post covers how to build an AI incident response plan that addresses these unique challenges.

What Makes AI Incidents Different?

The EchoLeak vulnerability of 2025 demonstrated this: a zero-click prompt injection flaw enabled data exfiltration without any user interaction. The attack didn't exploit a network vulnerability or a database — it exploited the model's context and permissions. Traditional security tools didn't catch it because there was no unauthorized access in the traditional sense.

What Are the AI Incident Types?

Prompt Injection

Adversarial inputs that manipulate model behavior. The model receives instructions embedded in data (documents, web pages, messages) that override its system prompt or user instructions.

Detection: Output content monitoring, behavioral anomaly detection (agent taking unexpected actions), user reports.

Containment: Disable the affected agent session, block the malicious input source, update prompt filters.

Data Exfiltration

The model is tricked into revealing sensitive data through its outputs. This can include training data (memorization attacks), context window data (other users' queries in a shared session), or connected system data (database records accessed through tools).

Detection: Output PII scanning, data access pattern monitoring, unusual query patterns.

Containment: Disable model access to the affected data source, revoke tool credentials, purge affected conversation history.

Hallucination Data Leak

The model fabricates information that happens to be real sensitive data. This is particularly dangerous in RAG systems where the model has access to a corpus of sensitive documents — a hallucination might reproduce a real document's contents that the user shouldn't have access to.

This is covered in detail in our guide on AI hallucination data leaks.

Detection: Output validation against ACLs, fact-checking pipeline, user reports of seeing unauthorized data.

Containment: Disable the affected retrieval path, audit ACL enforcement, re-index affected documents with corrected permissions.

Model Misuse

Users or agents use the model for purposes outside its intended scope. An agent designed for customer support is used to generate phishing emails. A model trained for code completion is used to write malware.

Detection: Usage pattern monitoring, output content classification, rate limiting alerts.

Containment: Suspend the offending user or agent, update usage policies, implement output content filters.

Training Data Poisoning

Malicious data injected into the training set causes the model to behave incorrectly in specific situations. This is the hardest to detect because the model appears to function normally until the trigger condition is met.

Detection: Model performance regression monitoring, output distribution shift detection, adversarial testing.

Containment: Rollback to a known-good model version, retrain on verified data, implement data provenance checks.

How to Define Incident Severity?

How to Build an AI Incident Response Plan?

Response Plan Architecture

Incident Response Playbook

from dataclasses import dataclass
from datetime import datetime
from enum import IntEnum

class Severity(IntEnum):
    S0 = 0  # Platform outage
    S1 = 1  # Active data exfiltration
    S2 = 2  # Suspected data leak
    S3 = 3  # Performance degradation
    S4 = 4  # Minor issue

@dataclass
class AIIncident:
    incident_id: str
    severity: Severity
    incident_type: str  # prompt_injection, data_exfiltration, etc.
    detected_at: datetime
    affected_model: str
    affected_tenants: list[str]
    description: str
    status: str = "open"

class AIIncidentResponder:
    """Coordinates AI incident response across the platform."""

    def handle_incident(self, incident: AIIncident) -> dict:
        """Main entry point for incident response."""
        response = {
            "incident_id": incident.incident_id,
            "actions": [],
            "status": "contained",
        }

        # Step 1: Immediate containment for S0/S1
        if incident.severity <= Severity.S1:
            response["actions"].append(self._kill_switch(incident))
            response["actions"].append(self._notify_security_team(incident))

        # Step 2: Containment for S2
        elif incident.severity == Severity.S2:
            response["actions"].append(self._revoke_sessions(incident))
            response["actions"].append(self._block_source(incident))

        # Step 3: Investigation
        response["actions"].append(self._collect_audit_trail(incident))
        response["actions"].append(self._analyze_root_cause(incident))

        # Step 4: Remediation
        response["actions"].append(self._remediate(incident))

        # Step 5: Post-incident review
        response["actions"].append(self._schedule_post_incident_review(incident))

        return response

    def _kill_switch(self, incident: AIIncident) -> dict:
        """Disable the affected model or agent immediately."""
        return {
            "action": "kill_switch",
            "target": incident.affected_model,
            "timestamp": datetime.utcnow().isoformat(),
            "detail": "Model disabled platform-wide"
        }

    def _revoke_sessions(self, incident: AIIncident) -> dict:
        """Revoke all active sessions for affected tenants."""
        return {
            "action": "revoke_sessions",
            "tenants": incident.affected_tenants,
            "timestamp": datetime.utcnow().isoformat(),
        }

Kill Switch Design

Every AI platform needs a kill switch — a mechanism to disable AI components without taking down the entire platform. This is the AI equivalent of pulling the network cable during a traditional breach.

kill_switches:
  global_model_disable:
    trigger: "S0 or S1 incident"
    action: "Disable all model inference endpoints"
    impact: "All AI features unavailable"
    recovery: "Manual re-enable after incident review"

  per_tenant_disable:
    trigger: "S2 incident affecting specific tenant"
    action: "Disable AI features for affected tenant only"
    impact: "Other tenants unaffected"
    recovery: "Automatic after remediation verified"

  agent_session_revocation:
    trigger: "Agent behaving anomalously"
    action: "Revoke all active agent sessions and JIT tokens"
    impact: "Agent stops mid-task"
    recovery: "User re-initiates session after investigation"

  tool_access_disable:
    trigger: "Tool being misused via prompt injection"
    action: "Revoke tool credentials, block at API gateway"
    impact: "Agent loses specific tool access"
    recovery: "Re-grant after tool filter update"

Kill switches should be testable, fast (under 30 seconds to execute), and logged. They should also be accessible to on-call engineers without requiring admin access to the model infrastructure.

How to Detect AI Incidents?

Detection is the hardest part of AI incident response. Traditional security tools (firewalls, IDS) don't catch AI-specific incidents. You need AI-specific detection mechanisms.

Detection Layers

class AIIncidentDetector:
    """Multi-layer AI incident detection."""

    def __init__(self):
        self.pii_scanner = PIIScanner()
        self.behavior_monitor = BehavioralMonitor()
        self.access_monitor = DataAccessMonitor()

    def check_output(self, output: str, user_context: dict) -> list[AIIncident]:
        """Check model output for potential incidents."""
        incidents = []

        # Layer 1: PII detection in output
        pii_found = self.pii_scanner.scan(output)
        if pii_found:
            incidents.append(AIIncident(
                incident_id=f"inc-{datetime.utcnow().timestamp()}",
                severity=Severity.S2,
                incident_type="data_exfiltration",
                detected_at=datetime.utcnow(),
                affected_model=user_context.get("model", "unknown"),
                affected_tenants=[user_context.get("tenant_id", "unknown")],
                description=f"PII detected in model output: {pii_found}"
            ))

        # Layer 2: ACL violation check
        if not self._check_output_acl(output, user_context):
            incidents.append(AIIncident(
                incident_id=f"inc-{datetime.utcnow().timestamp()}",
                severity=Severity.S1,
                incident_type="hallucination_data_leak",
                detected_at=datetime.utcnow(),
                affected_model=user_context.get("model", "unknown"),
                affected_tenants=[user_context.get("tenant_id", "unknown")],
                description="Output contains data user should not have access to"
            ))

        return incidents

How to Conduct a Post-Incident Review?

The NIST AI RMF MANAGE function (MANAGE 4.3) requires that "incidents and errors are communicated to relevant AI actors, including affected communities. Processes for tracking, responding to, and recovering from incidents and errors are followed and documented."

Post-Incident Review Template

## Post-Incident Review: [Incident ID]

### Summary
- **Incident type:** [prompt_injection / data_exfiltration / hallucination_leak / ...]
- **Severity:** [S0-S4]
- **Duration:** [detection to containment]
- **Impact:** [users affected, data exposed, business impact]

### Timeline
- [timestamp] — Incident detected by [detection layer]
- [timestamp] — Containment action taken: [action]
- [timestamp] — Root cause identified: [cause]
- [timestamp] — Remediation applied: [action]
- [timestamp] — Incident closed

### Root Cause Analysis
- **What happened:** [detailed description]
- **Why it happened:** [prompt injection / ACL bypass / model behavior / ...]
- **Why it wasn't prevented:** [gap in detection / missing filter / ...]

### Data Impact
- **Data exposed:** [types of data, number of records]
- **Users affected:** [count, tenant IDs]
- **External exposure:** [was data sent outside the organization?]

### Remediation
- **Immediate fix:** [what was done to contain]
- **Permanent fix:** [what was done to prevent recurrence]
- **Model changes:** [rollback, retrain, filter update]

### Preventive Measures
- [ ] [new detection rule added]
- [ ] [filter updated]
- [ ] [ACL enforcement improved]
- [ ] [kill switch tested]
- [ ] [team training updated]

How to Align with NIST AI RMF?

The NIST AI RMF MANAGE function directly addresses incident response:

Organizations implementing the NIST AI RMF should integrate their AI incident response plan into the MANAGE function's risk treatment processes. This ensures incident response is not a separate activity but part of the continuous risk management cycle.

AI Incident Response Checklist

• [ ] Documented AI incident response plan with severity tiers (S0-S4)
• [ ] Kill switch for global model disable (under 30 seconds)
• [ ] Per-tenant disable capability
• [ ] Agent session revocation mechanism
• [ ] Tool access disable at API gateway
• [ ] Output content filtering (PII, harmful content)
• [ ] Behavioral monitoring for agent anomalies
• [ ] Data access pattern monitoring
• [ ] User feedback channel for reporting AI issues
• [ ] Model performance regression monitoring
• [ ] Audit trail for every model invocation, tool call, and output
• [ ] Post-incident review template and process
• [ ] Incident communication plan (internal teams, affected users, regulators)
• [ ] Model rollback capability (revert to previous model version)
• [ ] Regular tabletop exercises for AI incident scenarios
• [ ] Integration with zero-trust AI agent architecture for session revocation
• [ ] Alignment with NIST AI RMF MANAGE 4.1-4.3
• [ ] Approval gates for sensitive actions to prevent misuse
• [ ] Documented escalation paths for each severity tier
• [ ] Training for on-call engineers on AI-specific incident types

FAQ

What is an AI incident response plan?

An AI incident response plan is a documented procedure for detecting, containing, and remediating security incidents involving AI systems. Unlike traditional IR plans, it addresses AI-specific incident types: prompt injection, data exfiltration via model outputs, hallucination-induced data leaks, model misuse, and training data poisoning. The plan defines severity tiers, escalation paths, containment procedures, and post-incident review processes.

How is AI incident response different from traditional IR?

Traditional IR deals with deterministic systems — a vulnerability is either exploited or not. AI systems are probabilistic — the same prompt can produce different outputs on different runs. Root cause analysis is harder because the "bug" is in a probability distribution, not a line of code. Additionally, AI incidents can involve data leaks through model outputs (hallucination reveals training data) rather than traditional data exfiltration paths.

What are the most common AI incident types?

The five most common AI incident types are: (1) prompt injection — adversarial inputs manipulate model behavior, (2) data exfiltration — model outputs reveal sensitive training or context data, (3) hallucination data leak — model fabricates sensitive information that happens to be real, (4) model misuse — users or agents use the model for unintended purposes, and (5) training data poisoning — malicious data corrupts model behavior.

How quickly should you contain an AI incident?

Severity 1 incidents (active data exfiltration, prompt injection causing harm) require immediate containment — disable the affected model or agent within minutes. Severity 2 incidents (suspected data leak, model producing harmful outputs) require containment within 1 hour. Severity 3 incidents (performance degradation, minor policy violations) require investigation within 4 hours. The key is having automated kill switches that can disable AI components without taking down the entire platform.

What should an AI incident post-incident review include?

A post-incident review for AI incidents should include: incident timeline and detection method, root cause analysis (was it prompt injection, data poisoning, model behavior, or configuration error), data impact assessment (what data was exposed and to whom), containment actions taken and their effectiveness, remediation steps (model rollback, prompt filter updates, ACL changes), and preventive measures to avoid recurrence. Document everything for regulatory compliance.

Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →