Per-Language PII De-identification¶

OpenMed's PII detection and de-identification are multilingual. Built-in language packs live in openmed.core.pii_i18n.SUPPORTED_LANGUAGES. Terminology maintainers and translators should also use the localization glossary for stable product and privacy terms. The optional Indic family can serve the built-in Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu codes. Every code documented here wires up:

• a default PII model from DEFAULT_PII_MODELS, used when you pass lang= without an explicit model_name= (an env:OPENMED_INDIC_NER_MODEL entry means weights are optional and must be configured by the caller), and
• a Faker locale from LANG_TO_LOCALE (openmed/core/anonymizer/locales.py), used by method="replace" to generate locale-aware surrogates.

Pick a language by passing its ISO 639-1 code to extract_pii() or deidentify():

from openmed import deidentify

deidentify("Paciente Pedro Almeida, CPF 123.456.789-09", lang="pt", method="mask")

For the redaction methods (mask, remove, replace, hash, shift_dates), locale resolution, determinism, and cross-document surrogate vaults, see PII Anonymization.

Automatic language and script routing¶

The staged privacy pipeline can choose a document pack automatically while preserving exact language decisions for every script run:

from openmed.core.pipeline import Pipeline

route = Pipeline(lang="auto").stage2_language_script(
    "Patient stable. 患者发热。 रोगी स्थिर है।"
)
print(route.lang, route.model_name)
print(route.metadata["runs"])

The core fallback is deterministic and dependency-free. It combines Unicode script runs with each LanguagePack's candidate priority and context hints; for example, adjacent kana selects Japanese for Han runs, while standalone Han prefers Chinese and Marathi lexical markers disambiguate Marathi from the otherwise Hindi-first Devanagari route. Install openmed[lid] to enable the lazy, on-device pycld2 adapter for ambiguous runs. The adapter and its CLD2 implementation are Apache-2.0, import only when routing is first requested, and do not download or bundle model weights.

Thai and Hebrew script runs select th and he. Myanmar, Khmer, Lao, Sinhala, Thaana, Tibetan and other unregistered scripts are reported with source="stdlib:unknown-script", reason="unsupported_script" and low confidence. Pipeline(lang="auto") rejects letter-bearing unsupported runs before model inference. If a caller has independently qualified a fallback, it can opt in with LanguageRouter(fallback_pack="en") passed as language_router=; this is not a claim of language support. Numeric-only and punctuation-only runs do not trigger that rejection. Routing diagnostics expose only scripts, codes, counts and offsets, never note text.

Deterministic detection in both deidentify() and staged Pipeline folds all Unicode decimal digits to ASCII and removes invisible bidi formatting controls in an offset-mapped matching view. Validators see the folded values; redaction spans still cover the original characters. Persian and Urdu matching also folds declared Arabic keyboard kaf/yeh variants (and Urdu heh variants), without rewriting the output or changing Arabic-language normalization. Diagnostics contain normalization counts, not original or folded identifiers.

Numeric Gregorian dates accept ., / and - separators and use the shared LOCALE_DATE_ORDER table for shifting and date surrogates. Valid year-first forms remain year-first; ambiguous numeric dates without a known locale are masked with [DATE_SHIFTED] rather than guessed. Hebrew attached prefixes are not included in the redacted date span. This does not add calendar conversion or model-quality qualification. The robustness operators bidi_in_value and native_digit_substitution can exercise these matching boundaries with a repeatable seed.

Built-in and optional languages¶

Chinese segmentation and Han-script routing use the dedicated zh registry entry. Being listed above does not by itself mean a code is model-backed: the rows whose model column reads env:OPENMED_INDIC_NER_MODEL or user-supplied ship no bundled weights and require a caller-supplied model. Malayalam, Punjabi, Polish, Russian, Tamil, and Urdu retain explicit public placeholder routes for compatibility, but those routes are not claims of dedicated trained weights. Codes absent from the table entirely (for example lv, sk, ms, tl, and fi) are not model-backed either. Several of them still have validator-backed national-ID coverage (openmed.core.pii_i18n.NATIONAL_ID_ONLY_LANGUAGES); see PII Anonymization for the ID providers. Urdu defaults to ur_IN and Faker's installed en_IN backend for general surrogate data, with a one-time approximation warning. Callers can still select ur_PK explicitly for CNIC validation.

The optional Indic NER adapter never downloads a checkpoint automatically. Set OPENMED_INDIC_NER_MODEL to a user-supplied local path or model repo, or pass an explicit model. When it is unset, registry lookup returns no optional model and the Naamapadam-style suite reports a structured skip reason.

openmed.core.pii_i18n.USER_SUPPLIED_MODEL_LANGUAGES holds the two codes that are registered for script routing, surrogate locales, deterministic patterns, and the public REST/MCP enums while claiming no bundled default model. The Malayalam route has a different fallback contract:

• ml now has a built-in multilingual fallback. The optional Indic NER adapter remains available when the caller supplies its weights explicitly.
• ne reads user-supplied. It has no adapter and no env-var path, so omitting model_name raises ValueError: Language '<code>' has no bundled OpenMed PII model; pass an explicit model_name (user-supplied-model languages: ne).

The user-supplied column value is a registry placeholder rather than a loadable repository. Passing it back as model_name raises the same error as omitting it, so a value copied from openmed_list_pii_languages never becomes a download attempt. SUPPORTED_LANGUAGES includes registered fallback packs such as pa but excludes the route that requires user-supplied weights.

Indian-English and code-mixed clinical notes¶

Latin-script Hinglish routing remains explicit. Set code_mixed=True to add the Roman-Hindi rule bank while keeping the English model route. When token tags are omitted, OpenMed's stdlib-only fallback derives exact offset/label records locally; no model weights or network access are required:

from openmed import deidentify

result = deidentify(
    "Patient Ravi ka aadhaar 246778325484 hai.",
    lang="en",
    code_mixed=True,
)

Callers with a locally hosted token classifier can pass lid_model=. The hook receives the input and offset-only token spans and must return one of hi, en, ne, univ, or other for every span. Explicit token_language_tags= still take precedence. Audit metadata retains offsets, labels, and hashes rather than token surfaces.

For lang="hi", lang="ml", or lang="te", a note containing both Latin and the language's Indic script automatically activates the India clinical route. OpenMed segments the note into offset-preserving script runs, adds bounded context to each run so PERSON and LOCATION spans can cross a script boundary, sends Latin windows to the registered English clinical model, and sends Indic windows to the language's registered model. Caller-supplied model IDs or local model paths are used for every window instead; OpenMed does not select an unregistered third-party model automatically.

The documented first-party fallback is OpenMed/privacy-filter-multilingual. Applications that choose that fallback must supply it explicitly as model_name; the route does not silently switch models after an inference error. Telugu replacement still uses the documented en_IN Faker locale approximation and can emit its existing one-time warning.

Indian-English prescription abbreviations such as Tab., Cap., OD, BD, TDS, HS, and SOS are normalized locally before entity merging. Source text and offsets remain unchanged. All shipped fixtures are synthetic; any restricted clinical corpus or separately trained weights remain user-supplied or out of process.

Hausa (ha) is available through that deterministic pattern-pack path with a native ha_NG surrogate locale. Boko context cues cover dates, ages, Nigerian NINs, and Nigerian/Nigerien phones. Ajami coverage is intentionally limited to numeric patterns—phones, NINs, and dates written with Western or Arabic-Indic digits—and does not claim Ajami lexical recognition or transliteration.

Yoruba (yo) is available through that deterministic pattern-pack path with a native yo_NG surrogate locale. It covers Nigeria NINs and +234 phones with tone-marked, decomposed, or unmarked context cues, and replacement spans expand to whole base-plus-diacritic clusters so dot-below and tone marks cannot be orphaned.

Igbo (ig) is available through that deterministic pattern-pack path with a native ig_NG surrogate locale. It recognizes Nigeria NINs, +234 phones, dates, and ages in Igbo and English-Igbo clinical text, including decomposed or unmarked context cues. Replacement spans expand to whole base-plus-diacritic clusters so dot-below vowels cannot be split or left behind.

Worked examples¶

Each example de-identifies synthetic, non-PHI text with method="mask". The exact placeholder tokens come from the chosen model's own entity labels, so they can vary by model (see choosing a method); the canonical labels below are illustrative.

Amharic — am¶

• Model: OpenMed/privacy-filter-multilingual · locale am_ET (en_KE is the documented Faker approximation)

Before: ስም፡ ሰላም ተስፋዬ። ስልክ፡ +251 911 234 567።
After:  ስም፡ [NAME]። ስልክ፡ [PHONE]።

Arabic — ar¶

• Model: OpenMed/OpenMed-PII-Arabic-SnowflakeMed-Large-568M-v1 · locale ar_EG

Before: المريضة ليلى حسن، الهاتف +20 10 1234 5678
After:  المريضة [NAME]، الهاتف [PHONE]

Assamese — as¶

• Model: OpenMed/privacy-filter-multilingual · locale as_IN

Before: অৰুণ গুৱাহাটীত জীৱন চিকিৎসালয়লৈ গ'ল।
After:  [PERSON] [LOCATION] [ORGANIZATION] গ'ল।

Bengali — bn¶

• Model: OpenMed/OpenMed-PII-Bengali-mSuperClinical-Large-279M-v1 · locale bn_BD

Before: অরুণ কলকাতায় আনন্দ হাসপাতালে গেলেন।
After:  [PERSON] [LOCATION] [ORGANIZATION] গেলেন।

Danish — da¶

• Model: OpenMed/privacy-filter-multilingual · locale da_DK

Before: Patient Anna Nielsen, CPR 170885-1234
After:  Patient [NAME], CPR [ID]
### Czech — `cs`

- Model: `OpenMed/privacy-filter-multilingual` · locale `cs_CZ`

```text
Before: Pacient Jan Novák, rodné číslo 751116/0008
After:  Pacient [NAME], rodné číslo [ID]

German — de¶

• Model: OpenMed/OpenMed-PII-German-SuperClinical-Small-44M-v1 · locale de_DE

Before: Patientin Anna Müller, Steuer-ID 86095742719
After:  Patientin [NAME], Steuer-ID [ID]

Greek — el¶

• Model: OpenMed/privacy-filter-multilingual · locale el_GR

Before: Ασθενής Γιώργος Παπαδόπουλος, ΑΜΚΑ 16117508024
After:  Ασθενής [NAME], ΑΜΚΑ [ID]

English — en¶

• Model: OpenMed/OpenMed-PII-SuperClinical-Small-44M-v1 · locale en_US

Before: Patient John Doe was seen on 03/14/2025; call 555-0142
After:  Patient [NAME] was seen on [DATE]; call [PHONE]

Spanish — es¶

• Model: OpenMed/OpenMed-PII-Spanish-SuperClinical-Small-44M-v1 · locale es_ES

Before: Paciente Maria Garcia, DNI 12345678Z
After:  Paciente [NAME], DNI [ID]

Persian — fa¶

• Model: OpenMed/privacy-filter-multilingual · locale fa_IR

The multilingual checkpoint is a compatibility placeholder until dedicated Persian weights are published and evaluated. The built-in pack adds RTL Persian patterns, Eastern-digit normalization, Jalali dates, and checksum-valid Iranian national-ID surrogates.

Before: بیمار با کد ملی ۱۲۳۴۵۶۷۸۹۱ در تاریخ ۱۴۰۵/۰۵/۲۹ مراجعه کرد.
After:  بیمار با کد ملی [ID] در تاریخ [DATE] مراجعه کرد.

French — fr¶

• Model: OpenMed/OpenMed-PII-French-SuperClinical-Small-44M-v1 · locale fr_FR

Before: Patient Jean Dupont, NIR 1 84 12 76 451 089 46
After:  Patient [NAME], NIR [ID]

Gujarati — gu¶

• Model: OpenMed/privacy-filter-multilingual · locale gu_IN

Before: રોગી શ્રી નરેશભાઈ. પિન કોડ ૩૮૦૦૦૧. આધાર ૨૪૬૭ ૭૮૩૨ ૫૪૮૪.
After:  રોગી શ્રી [PERSON]. પિન કોડ [ZIPCODE]. આધાર [ID_NUM].

Kannada — kn¶

• Model: OpenMed/privacy-filter-multilingual · locale kn_IN (en_IN Faker backend)

Before: ಕೃತಕ ಟಿಪ್ಪಣಿ: ಶ್ರೀ ಕೆ. ಎಸ್. ರವಿ ಅವರು, ಆಧಾರ್ ೨೪೬೭ ೭೮೩೨ ೫೪೮೪.
After:  ಕೃತಕ ಟಿಪ್ಪಣಿ: ಶ್ರೀ [PERSON] ಅವರು, ಆಧಾರ್ [ID_NUM].

Malayalam — ml¶

• Model: OpenMed/privacy-filter-multilingual · locale ml_IN (en_IN Faker backend); an optional local Indic NER model can be supplied explicitly.

Before: രോഗി ശ്രീ പുതുശ്ശേരി രാമൻ, ആധാർ ൨൪൬൭ ൭൮൩൨ ൫൪൮൪.
After:  രോഗി ശ്രീ [PERSON], ആധാർ [ID_NUM].

Marathi — mr¶

• Model: OpenMed/privacy-filter-multilingual · locale mr_IN

Before: रुग्णाचे नाव सौ. वैशाली सुरेश देशमुख. जन्मतारीख १४ फेब्रुवारी १९८५.
After:  रुग्णाचे नाव [PERSON]. जन्मतारीख [DATE].

Nepali — ne¶

• Model: user-supplied · locale ne_NP

Before: सुमन थापा काठमाडौंको जीवन अस्पताल गए।
After:  [PERSON] [LOCATION]को [ORGANIZATION] गए।

Odia — or¶

• Model: OpenMed/privacy-filter-multilingual · locale or_IN

Before: ଅରୁଣ ଭୁବନେଶ୍ୱରରେ କଳିଙ୍ଗ ହସ୍ପିଟାଲକୁ ଗଲେ।
After:  [PERSON] [LOCATION] [ORGANIZATION] ଗଲେ।

Punjabi — pa¶

• Model: OpenMed/privacy-filter-multilingual · locale pa_IN (approximated with Faker's en_IN backend; warns once)
• Built-in Gurmukhi patterns cover Punjabi dates, Indian mobile numbers, Aadhaar, Punjab/Chandigarh PIN codes, and honorific-anchored full names. Honorifics accept one to eight horizontal spaces, tabs, or non-breaking spaces without shifting the name span. These are bounded synthetic regression checks, not a trained Punjabi model or clinical-quality claim. Replacement names preserve ਸਿੰਘ and ਕੌਰ gender suffixes. A configured Indic adapter may still be selected explicitly.

Before: ਅਰੁਣ ਅੰਮ੍ਰਿਤਸਰ ਵਿੱਚ ਜੀਵਨ ਹਸਪਤਾਲ ਗਿਆ।
After:  [PERSON] [LOCATION] ਵਿੱਚ [ORGANIZATION] ਗਿਆ।

Tamil — ta¶

• Model: OpenMed/privacy-filter-multilingual (placeholder) · locale ta_IN

The formerly configured dedicated checkpoint is not in the public model catalog. Treat the placeholder as compatibility routing, not qualified Tamil model coverage; configure and evaluate explicit weights for production use.

Before: அருண் சென்னையில் காவேரி மருத்துவமனை சென்றார்.
After:  [PERSON] [LOCATION] [ORGANIZATION] சென்றார்.

Urdu — ur¶

This rules-and-surrogates pack uses a named default-model placeholder, not verified trained Urdu coverage. Synthetic fixtures test source offsets and bidi-safe masking, not clinical accuracy. Horizontal non-breaking spaces in honorific names remain part of the original source span.

• Model: OpenMed/privacy-filter-multilingual · locale ur_IN

Before: مریض محترمہ فاطمہ خان، آدھار ۲۴۶۷ ۷۸۳۲ ۵۴۸۴۔
After:  مریض [PERSON]، آدھار [ID_NUM]۔

Hebrew — he¶

• Model: OpenMed/privacy-filter-multilingual · locale he_IL

Before: מטופל דוד לוי, טלפון 054-1234567
After:  מטופל [NAME], טלפון [PHONE]

Hindi — hi¶

• Model: OpenMed/OpenMed-PII-Hindi-SuperClinical-Large-434M-v1 · locale hi_IN

Before: रोगी अनीता शर्मा, फोन +91 9876543210
After:  रोगी [NAME], फोन [PHONE]

Indonesian — id¶

• Model: OpenMed/privacy-filter-multilingual · locale id_ID

Before: Pasien Budi Santoso, NIK 3201234567890123
After:  Pasien [NAME], NIK [ID]

Italian — it¶

• Model: OpenMed/OpenMed-PII-Italian-SuperClinical-Small-44M-v1 · locale it_IT

Before: Paziente Marco Rossi, codice fiscale RSSMRC80A01H501U
After:  Paziente [NAME], codice fiscale [ID]

Japanese — ja¶

• Model: OpenMed/OpenMed-PII-Japanese-BigMed-Large-560M-v1 · locale ja_JP

Before: 患者 佐藤 花子、電話 +81 90 1234 5678
After:  患者 [NAME]、電話 [PHONE]

Korean — ko¶

• Model: OpenMed/OpenMed-PII-Korean-NomicMed-Large-395M-v1 · locale ko_KR

Before: 환자 김민수, 전화 010-1234-5678
After:  환자 [NAME], 전화 [PHONE]

Dutch — nl¶

• Model: OpenMed/OpenMed-PII-Dutch-SuperClinical-Large-434M-v1 · locale nl_NL

Before: Patiënt Eva de Vries, BSN 123456782
After:  Patiënt [NAME], BSN [ID]

Norwegian — no¶

• Model: OpenMed/privacy-filter-multilingual · locale no_NO

Before: Pasient Ingrid Hansen, fødselsnummer 12035101460
After:  Pasient [NAME], fødselsnummer [ID]

Polish — pl¶

This is a rules-and-surrogates pack with a named default-model placeholder, not a claim of trained Polish model coverage. Its synthetic leakage tests do not establish clinical accuracy.

• Model: OpenMed/privacy-filter-multilingual · locale pl_PL

Before: Pacjent, PESEL 85031512344, adres ul. Przykładowa 12.
After:  Pacjent, PESEL [ID_NUM], adres [STREET_ADDRESS].

Portuguese — pt¶

• Model: OpenMed/OpenMed-PII-Portuguese-SnowflakeMed-Large-568M-v1 · locale pt_PT (pass locale="pt_BR" for Brazilian CPF/CNPJ surrogates)

Before: Paciente Pedro Almeida, CPF 123.456.789-09
After:  Paciente [NAME], CPF [ID]

Romanian — ro¶

• Model: OpenMed/privacy-filter-multilingual · locale ro_RO

Before: Pacient Ion Popescu, CNP 1960101221144
After:  Pacient [NAME], CNP [ID]

Russian — ru¶

• Model: OpenMed/privacy-filter-multilingual · locale ru_RU
• The multilingual model name is a routing placeholder until dedicated Russian weights ship; deterministic Cyrillic patterns remain available offline.

Before: Пациент Иван Петров, СНИЛС 112-233-445 95
After:  Пациент [NAME], СНИЛС [ID]

Swedish — sv¶

• Model: OpenMed/privacy-filter-multilingual · locale sv_SE

Before: Patient Anna Andersson, personnummer 510312-1140
After:  Patient [NAME], personnummer [ID]

Swahili — sw¶

• Model: OpenMed/privacy-filter-multilingual · locale sw

Before: Jina: Amina Hassan. Nambari ya kitambulisho 12345678
After:  Jina: [NAME]. Nambari ya kitambulisho [ID]

Telugu — te¶

• Model: OpenMed/OpenMed-PII-Telugu-SuperClinical-Large-434M-v1 · locale en_IN (Faker has no Telugu locale — en_IN is a documented approximation)

Before: రోగి రమేష్ కుమార్, ఫోన్ +91 9876543210
After:  రోగి [NAME], ఫోన్ [PHONE]

Thai — th¶

• Model: OpenMed/privacy-filter-multilingual · locale th_TH

Before: ผู้ป่วย สมชาย ใจดี โทร 081-234-5678
After:  ผู้ป่วย [NAME] โทร [PHONE]

Turkish — tr¶

• Model: OpenMed/OpenMed-PII-Turkish-SuperClinical-Small-44M-v1 · locale tr_TR

Before: Hasta Ayşe Yılmaz, TCKN 10000000146
After:  Hasta [NAME], TCKN [ID]

isiXhosa — xh¶

• Model: OpenMed/privacy-filter-multilingual · locale xh_ZA (zu_ZA is the documented Faker approximation)

Before: Igama lesigulane: Xolani Qwabe. Inombolo yesazisi 7903116001080
After:  Igama lesigulane: [NAME]. Inombolo yesazisi [ID]

Chinese — zh¶

• Model: OpenMed/OpenMed-PII-Chinese-BigMed-Large-560M-v1 · locale zh_CN

Before: 患者王芳，电话 13800138000
After:  患者[NAME]，电话 [PHONE]

The dedicated registry entry is used by default for Chinese PII detection, while segmentation and exact-offset guarantees remain unchanged.

isiZulu — zu¶

• Model: OpenMed/privacy-filter-multilingual · locale zu_ZA

Before: Igama lesiguli: Nomcebo Dlamini. Inombolo kamazisi 8001015009087
After:  Igama lesiguli: [NAME]. Inombolo kamazisi [ID]
### Ukrainian — `uk`

- Model: `OpenMed/privacy-filter-multilingual` · locale `uk_UA`

```text
Before: Пацієнтка Олена Коваль, РНОКПП 2974281300
After:  Пацієнтка [NAME], РНОКПП [ID]

Vietnamese — vi¶

• Model: OpenMed/OpenMed-PII-Vietnamese-SuperClinical-Small-44M-v1 · locale vi_VN

Dates parse day-first (17/08/1985 and ngày 8 tháng 12 năm 1970), mobile numbers follow the +84/0[3,5,7,8,9] plan, and postal codes are the current five-digit format. CCCD detection requires an explicit CCCD or căn cước công dân cue and is validated structurally — the 12-digit contract from Article 12 of the 2023 Law on Identification, with no public checksum.

Before: Bệnh nhân Nguyễn Minh Anh, CCCD 001203123456
After:  Bệnh nhân [NAME], CCCD [ID]