How to Implement PII Redaction Before AI Ingestion
TL;DR — PII redaction before AI ingestion detects and removes personally identifiable information from text before it reaches your AI system. Three-layer detection: regex for formatted PII (SSNs, credit cards), NER for unstructured PII (names, addresses), and LLM judge for context-dependent cases. Reversible tokenization preserves model quality while ensuring raw PII never enters vector stores, embeddings, or model training data. Microsoft Presidio is the open-source foundation. This is the technical implementation of GDPR's data minimization principle.
A PII redaction policy written in a company wiki is not a control. It is a liability waiting to surface in your next HIPAA or GDPR audit. Enterprise AI pipelines in regulated industries require system-level, self-hosted PII detection that generates immutable audit logs mapped to NIST AI RMF (PredictionGuard 2026).
When personal data enters an AI system without redaction, it becomes part of the vector store, the embeddings, and potentially the model's outputs. Once embedded, PII is extraordinarily difficult to remove — you cannot "untrain" a model on specific data points. The solution is redaction at the ingestion layer: detect PII before it enters the system, replace it with placeholders, and store the mapping in a secure vault.
This post covers the architecture, detection techniques, and implementation patterns for PII redaction in enterprise AI pipelines. For the broader data protection strategy, see how to keep enterprise AI data private.
Why PII Redaction Belongs at the Ingestion Layer
PII can enter your AI system at four points:
| Entry point | Example | Risk |
|---|---|---|
| Document ingestion | HR files, contracts, medical records indexed for RAG | PII stored in vector database, retrievable by any user |
| Prompt input | Employee pastes customer data into a chat interface | PII sent to external LLM provider, subject to their retention |
| Training data | Datasets containing customer records used for fine-tuning | PII baked into model weights, unrecoverable |
| API responses | AI generates output containing PII from training data | PII leaked to unauthorized users |
Redaction at the ingestion layer — before data reaches the vector store or model — is the only point where you have full control. Once data is embedded or used in training, removal requires re-embedding or retraining the entire model.
This is also a GDPR requirement. Article 5's data minimization principle requires processing only data that is adequate and necessary. The CNIL recommends pseudonymization before sending data to LLMs: "good AI doesn't need to know who is concerned to draft a summary" (CNIL 2026).
Three-Layer PII Detection Architecture
No single detection technique catches all PII. A production pipeline requires three layers, each compensating for the others' weaknesses:
Layer 1: Regex + Checksum Validation
| PII type | Pattern | Validation |
|---|---|---|
| SSN (US) | \d{3}-\d{2}-\d{4} |
Area number validation |
| Credit card | Luhn algorithm | Checksum verification |
| RFC 5322 pattern | Domain validation | |
| Phone | E.164 format | Country code check |
| IBAN | ISO 13616 pattern | Checksum verification |
| Passport | Country-specific patterns | Format validation |
Regex catches formatted PII with near-zero false positives. It misses unstructured PII — names, addresses, job titles — that have no consistent format. Use regex as the first layer: fast, precise, and cheap.
Layer 2: Named Entity Recognition (NER)
NER models detect PII that regex cannot. Microsoft Presidio is the open-source standard, combining regex recognizers with NER models and supporting custom recognizers for organization-specific patterns (Microsoft 2026).
| NER model | What it detects | Strengths |
|---|---|---|
| spaCy (en_core_web_lg) | PERSON, ORG, GPE, DATE | Fast, lightweight, good baseline |
| DeBERTa-v3 NER | Person, organization, location | Higher accuracy on complex text |
| Presidio (built-in) | 50+ PII types | Combines regex + NER + custom |
| Flair NER | Named entities in context | Strong on ambiguous entities |
NER has higher recall but lower precision than regex. It may flag "Apple" as an organization when the text refers to the fruit. This is why Layer 3 exists.
Layer 3: LLM Judge
A secondary LLM validates flagged tokens for context-dependent PII that regex and NER cannot resolve. For example:
- "The patient was diagnosed with diabetes" — the diagnosis is PHI, not the word "patient"
- "She transferred $50,000 to account 123456" — the amount and account are financial PII
- "Meeting with the CEO about the merger" — the merger is confidential business information
The LLM judge has the highest accuracy but also the highest latency and cost. Use it selectively — only for tokens flagged by Layers 1 and 2 with low confidence scores.
Reversible Tokenization: Redaction Without Quality Loss
The key to preserving AI model quality is reversible tokenization. Instead of deleting PII, replace it with structured placeholders:
| Original text | Redacted text |
|---|---|
| "John Smith applied for a loan" | "[PERSON_1] applied for a loan" |
| "Contact john@example.com" | "Contact [EMAIL_1]" |
| "SSN: 123-45-6789" | "SSN: [SSN_1]" |
A token vault maps each placeholder to its original value:
- [PERSON_1] → John Smith
- [EMAIL_1] → john@example.com
- [SSN_1] → 123-45-6789
When the AI system responds, the vault rehydrates placeholders with original values. The model sees the semantic structure without accessing the raw PII. Studies show less than 5% quality degradation for most tasks (DigitalApplied 2026).
Tokenization strategies
| Strategy | How it works | Use case |
|---|---|---|
| Sequential placeholders | [PERSON_1], [PERSON_2] | General-purpose, preserves entity count |
| Salted hash | SHA-256(token + salt) | Deterministic, no vault needed |
| HMAC | HMAC-SHA256(token + key) | Deterministic, keyed, verifiable |
| FPE (FF1) | NIST 800-38G format-preserving encryption | Reversible, preserves format (SSN looks like SSN) |
| Vaulted tokenization | Random token + vault lookup | Most secure, fully reversible |
For most enterprise AI pipelines, sequential placeholders with a vault are sufficient. For regulated data requiring format preservation (e.g., credit card numbers that must look like credit card numbers), use FPE.
PII Redaction Pipeline Architecture
The pipeline operates in two phases:
Ingestion phase: Documents enter through connectors (Slack, Google Drive, email). The parser extracts text. The detection layer identifies PII. The replacement layer swaps PII for placeholders. Only redacted text enters the vector store and embedding model.
Query phase: When a user asks a question, the prompt goes through the same redaction pipeline. The LLM processes redacted text. The response is rehydrated with original values from the vault before being returned to the user.
This architecture ensures PII never touches the embedding model, the vector store, or the external LLM API. For preventing sensitive data from leaking into AI prompts, this is the ingestion-layer control that complements the gateway-layer DLP.
Implementation With Microsoft Presidio
Microsoft Presidio provides the detection and anonymization layers. Here is a production pipeline pattern:
| Component | Role | Technology |
|---|---|---|
| Analyzer | Detects PII entities in text | Presidio Analyzer + spaCy NER |
| Anonymizer | Replaces detected entities with placeholders | Presidio Anonymizer |
| Vault | Stores placeholder-to-value mappings | Redis or encrypted PostgreSQL |
| Custom recognizers | Organization-specific PII patterns | Regex + context words |
| Audit log | Records every detection and redaction event | Immutable log to SIEM |
Presidio supports custom recognizers for organization-specific PII — internal project codenames, employee ID formats, proprietary identifiers. Add these alongside the 50+ built-in recognizers.
PII Redaction and Compliance
| Regulation | Requirement | How redaction helps |
|---|---|---|
| GDPR Article 5 | Data minimization | Removes unnecessary PII before processing |
| GDPR Article 25 | Data protection by design | Redaction is a built-in privacy control |
| GDPR Article 32 | Security of processing | Pseudonymization is a recognized security measure |
| HIPAA | De-identification of PHI | Removes 18 HIPAA identifiers before AI processing |
| CCPA | Consumer privacy rights | Reduces PII exposure to third-party AI providers |
| EU AI Act Article 9 | Risk management | Redaction is a risk mitigation control for high-risk AI |
For AI risk assessment, PII redaction is a control that reduces both inherent and residual risk scores for data exfiltration and privacy violation threats.
FAQ
What is PII redaction before AI ingestion?
PII redaction before AI ingestion is the process of detecting and removing personally identifiable information from text before it enters an AI system — whether a vector database, LLM prompt, or training pipeline. It uses regex, named entity recognition (NER), and LLM-based validation to identify and replace sensitive tokens.
What tools are used for PII redaction in AI pipelines?
Microsoft Presidio is the open-source standard, combining regex recognizers with NER models. For production pipelines, layer Presidio with DeBERTa or spaCy NER models, and add an LLM judge layer for context-dependent PII. Format-preserving encryption (FPE) handles reversible redaction.
How does reversible PII redaction work?
Reversible redaction replaces sensitive tokens with placeholders (e.g., [PERSON_1], [EMAIL_1]) before sending text to the LLM. A vault maps placeholders back to original values. When the LLM response comes back, the vault rehydrates placeholders with real values, preserving the model's ability to understand context.
Does PII redaction affect AI model quality?
Minimal impact when done correctly. Redaction preserves sentence structure and semantic meaning — the model sees 'PERSON_1 did X' instead of 'John Smith did X.' Studies show less than 5% quality degradation for most tasks. The tradeoff is privacy compliance vs. marginal quality loss.
Is PII redaction required by GDPR?
GDPR's data minimization principle (Article 5) requires processing only data that is adequate and necessary. PII redaction is the technical implementation of this principle for AI systems. The CNIL recommends pseudonymization before sending data to LLMs. For high-risk processing, a DPIA will identify redaction as a required control.
Want a self-hosted AI company brain that does all of this out of the box?
Book a demo →