PII Anonymization¶

OpenMed's deidentify() API supports five redaction methods for detected PII entities:

This document focuses on replace, which was upgraded in v1.3.0 to a full Faker-backed obfuscation engine.

The new replace engine¶

method="replace" no longer picks from a small hardcoded list. It builds a per-document Anonymizer (see openmed.core.anonymizer) that delegates to Faker with custom providers for clinical IDs.

from openmed import deidentify

deidentify(
    "Paciente Pedro Almeida, CPF: 123.456.789-09",
    method="replace",
    lang="pt",
    locale="pt_BR",          # default for pt is pt_PT; override per call
    consistent=True,         # same input -> same surrogate within doc
    seed=42,                 # cross-run reproducibility
)

Locale resolution¶

lang (an ISO 639-1 code OpenMed uses everywhere) maps to a Faker locale via LANG_TO_LOCALE:

Pass locale= explicitly to override per call (e.g. pt_BR to generate CPF/CNPJ surrogates instead of Portuguese NIF/VAT).

Determinism¶

Three modes:

• Random (default). Every call samples fresh surrogates. Good for audits or when you want visible variability.
• consistent=True. Same (canonical_label, original_value) pair resolves to the same surrogate within the call. "John Doe" appearing twice in one document gets one surrogate.
• seed=<int> (implies consistent=True). Same seed across runs produces the same surrogate stream — useful for snapshot tests and regression fixtures.

Determinism uses hashlib.blake2b over (seed, canonical_label, original), so different originals always get different surrogates.

Format preservation¶

Phone numbers, dates, and emails preserve the structure of the original:

deidentify("Call (415) 555-1234", method="replace", consistent=True, seed=1)
#  -> "Call (XXX) XXX-XXXX"  (digit groups, separators, country code position)

deidentify("Born 01/15/1970", method="replace", consistent=True, seed=1)
#  -> "Born MM/DD/YYYY"      (separator and ordering kept)

deidentify("Email: john@hospital.org", method="replace", lang="en")
#  -> "Email: alice@hospital.org"  (domain kept, local part faked)

Clinical ID checksums¶

OpenMed reuses the existing checksum validators in openmed.core.pii_i18n so every surrogate ID passes the same validator that detection uses:

Extending¶

from openmed import register_clinical_provider, register_label_generator
from faker.providers import BaseProvider

# Add your own checksum-bearing ID format
class HospitalAccountProvider(BaseProvider):
    def hospital_account(self):
        return f"HACC-{self.numerify('########')}"

register_clinical_provider(HospitalAccountProvider)

# Override the generator for a canonical label
def my_first_name(faker, original, *, locale):
    return faker.first_name() + "-test"

register_label_generator("FIRST_NAME", my_first_name)

Privacy-filter family¶

OpenMed ships three privacy-filter families, all the same OpenAI Privacy Filter architecture (gpt-oss-style sparse-MoE transformer with local attention, sink tokens, RoPE+YaRN, tiktoken o200k_base), differing only in their training data:

All run through the same extract_pii() / deidentify() API — only the weights differ:

extract_pii(text, model_name="OpenMed/privacy-filter-mlx-8bit")
extract_pii(text, model_name="OpenMed/privacy-filter-nemotron-mlx-8bit")
extract_pii(text, model_name="OpenMed/privacy-filter-multilingual-mlx-8bit")

deidentify(text, model_name="OpenMed/privacy-filter-nemotron",
           method="replace", consistent=True, seed=42)
deidentify(text, model_name="OpenMed/privacy-filter-multilingual",
           method="replace", consistent=True, seed=42)

Backend selection. On Apple Silicon with MLX importable, the MLX artifact runs natively via PrivacyFilterMLXPipeline. Elsewhere, the call substitutes the corresponding PyTorch model via transformers and emits a one-time UserWarning explaining the swap. The fallback is family-aware — an MLX-only Nemotron request on Linux substitutes OpenMed/privacy-filter-nemotron, and an MLX-only multilingual request substitutes OpenMed/privacy-filter-multilingual, so the user gets the same training distribution they asked for.

Either way the output entity dicts have the same shape so the rest of the pipeline behaves identically. Smart-merging (regex-based span construction) is skipped for this family — the model already does Viterbi-constrained BIOES decoding internally.

The dispatch lives in openmed.core.backends.create_privacy_filter_pipeline. To register a new fine-tune that should fall back to its own PyTorch repo on non-Mac hosts, add a row to _TORCH_FALLBACK_BY_FAMILY in that module. If a fine-tune introduces a genuinely different architecture (not just new weights), it would also need a new MLX model class and family branch in openmed.mlx.models.build_model — but a same-architecture fine-tune needs neither.

Backwards compatibility¶

replace outputs are no longer drawn from the prior hardcoded list (["Jane Smith", "John Doe", "Alex Johnson", "Sam Taylor"], etc.). Any test that asserted on those exact strings must either:

• pass consistent=True, seed=<value> and update the expected output, or
• assert non-equality with the original instead of equality with a hardcoded surrogate.

Other methods (mask, remove, hash, shift_dates) are unchanged.