PII detection and masking in AI pipelines

The executive article "Data privacy in the age of AI" addresses the decision: which data can be exposed to a model, under what legal basis, and who is accountable if it leaks. This piece addresses the implementation: how a pipeline detects, masks and — when authorized — re-identifies personal data before and after inference.

The problem isn't finding obvious PII

National ID numbers, emails and phone numbers are easy. The real risk is what slips through unnoticed: a proper name in free text, an address in a notes field, a card number split across two columns, or an internal ID that, joined with another dataset, re-identifies someone. A PII protection pipeline needs layers, not a list of regex patterns.

Layer 1 — Regex and structured patterns

Regex is still the first line of defense for fields with a known format: national ID, tax ID, credit card, email, phone, postal code, license plate. It's fast, deterministic and cheap at volume.

EMAIL:    [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
CARD:     \b(?:\d[ -]*?){13,19}\b
SSN-like: \b\d{3}-?\d{2}-?\d{4}\b

The problem with regex alone: false positives (an 11-digit order number looks like an ID) and false negatives (an unformatted ID buried in text). Regex should never be the only layer for data feeding an automated decision.

Layer 2 — Checksum validation

Documents such as national tax IDs and credit cards carry a check digit. Applying the checksum algorithm after the regex match removes most false positives at no model cost:

def luhn_valid(number: str) -> bool:
    digits = [int(d) for d in number if d.isdigit()]
    if len(digits) < 13:
        return False
    total = 0
    for i, d in enumerate(reversed(digits)):
        if i % 2 == 1:
            d *= 2
            if d > 9:
                d -= 9
        total += d
    return total % 10 == 0

The same applies to other national ID formats with their own check-digit algorithms. A regex candidate that fails checksum is discarded or downgraded in confidence before it ever reaches the NER layer.

Layer 3 — NER for free text

Names, addresses, job titles and other PII without a fixed format require named-entity recognition. An NER model (spaCy, a fine-tuned transformer, or a managed service) runs over free-text fields — ticket descriptions, call transcripts, notes fields — and flags spans with an entity type and confidence score.

Engineering caveat: NER carries far higher latency and infrastructure cost than regex. Best practice is to run regex+checksum first across the full volume, and reserve NER for fields already identified as free text plus audit sampling over structured fields, rather than running the heavy model over everything.

Reversible vs. irreversible pseudonymization

Once detected, PII must be handled according to its downstream use:

Practical rule: if the use case doesn't need the original value, don't use a reversible technique. Reversible pseudonymization is extra responsibility — key vault, access control, a log of who re-identified what and when.

Masking before inference

The most important decision point in the pipeline is: does masking happen before the data leaves the company's trust boundary, or after? For calls to third-party models (LLM-as-a-service, third-party API), the correct answer is always before. The typical flow:

1. 1.Raw data enters the pipeline.
2. 2.The detection layer (regex + checksum + NER) flags PII spans.
3. 3.The masking layer replaces the spans with a reversible token (if the use case requires linking the response back to the original record) or a fixed label.
4. 4.The masked text is what goes to the model — internal or third-party.
5. 5.If the model's response needs to be re-linked to the original record, re-identification happens afterward, inside the trust boundary, never at the external provider.

Controlled re-identification

Re-identification should not be a function available to any service consuming the pipeline. Treat it as a privileged operation:

• Explicit authorization by role — not by possession of a token.
• Logging of every re-identification: who, when, which record, what justification.
• Expiration of the token-to-original-value link once the data's retention period ends.

Minimization and retention

Detecting PII is not an excuse to keep it "just in case." Two engineering principles:

• Minimize at ingestion: if a field isn't needed for the use case, it never enters the pipeline — filtered at ingestion, not at output.
• Retention with automatic expiration: each pseudonymized record carries an expiration date tied to the purpose that justified its collection. A periodic purge job removes both the data and the re-identification link.

Handling data-subject rights (GDPR/LGPD)

A well-designed pipeline answers three requests without manual intervention on every record:

• Access: locate all pseudonymized records associated with a data subject, given their original identifier.
• Correction: update the original value without breaking the link to tokens already issued.
• Erasure: delete the original value and invalidate all associated tokens, while keeping a trace that an erasure occurred (without retaining the data itself).

This is only feasible if the tokenization index is designed from day one with the data-subject key as a searchable field — not bolted on after the first access request arrives.

What to do on Monday

• Audit free-text fields in production and run an NER sampling test to measure how much unstructured PII is currently going unnoticed.
• Add checksum validation to any existing ID/card detector — it's the lowest-effort change with the biggest reduction in false positives.
• Check whether any third-party LLM call is sending unmasked fields; if so, that's the top priority fix before anything else on this list.
• Document, for each PII type in use, whether the applied pseudonymization is reversible and who is authorized to re-identify it.

Further reading

Executive track:

• Data privacy in the age of AI: GDPR and LGPD in practice — the business view of this same topic.