Skip to content

FHIR Bulk Data privacy gateway

OpenMed provides a local-first FHIR Bulk Data 3.0.0 gateway for supported R4 resources. It reads NDJSON one resource at a time, applies the existing FHIR de-identification policy, and commits each output file atomically. Synthetic offline exports are useful for validating the workflow without a FHIR server.

Local export

from openmed.interop.fhir.bulk import BulkDataGateway, BulkGatewayConfig

gateway = BulkDataGateway(
    BulkGatewayConfig(
        input_dir="./synthetic-export",
        output_dir="./deidentified-export",
        policy="hipaa_safe_harbor",
        method="replace",
        schema_policy="fhir_hipaa_safe_harbor",
        date_shift_secret=date_shift_secret,
        max_buffered_resources=1,
    )
)
report = gateway.export(job_id="synthetic-run")
print(report.to_dict())

The report contains counts, resource types, output hashes, timings, policy provenance, and rejection records. Rejection records contain only a reason, line number, structural path, and SHA-256 hash of the rejected resource. They never contain a resource body, identifier, local absolute path, or URL.

Completed files are recorded in an atomic JSON checkpoint. If a process stops after a file commit, a later run verifies both the input and output hashes, skips that file, and resumes the remaining files. A partial .part file is never promoted to the final output. Output serialization is deterministic, so an interrupted and resumed synthetic export has byte-identical completed files and no duplicate resources.

When schema_policy is set, the checkpoint compatibility hash also covers the complete field policy, date-shift settings, and opaque fingerprints of the subject/date keys. A changed policy or key is therefore reprocessed instead of resuming incompatible output; raw key material is never checkpointed.

Field-level schema policies

Schema policies map canonical FHIR or OMOP paths to suppress, generalize, date-shift, route-to-deidentify, or keep. Bundled Safe Harbor-oriented and research limited-dataset variants are available for both schemas. They are technical controls, not a legal determination that a data set satisfies HIPAA or another regulatory framework.

Before processing, use lint_schema_policy(data, policy) to find observed fields without explicit rules and validate_schema_policy(policy, schema_fields) to compare a policy with a known schema. Uncovered identifier-shaped or identifier-declared fields are suppressed; other fields follow the policy's default_action and produce reviewable warnings.

from openmed.structured import apply_schema_policy, lint_schema_policy

findings = lint_schema_policy(bundle, "fhir_hipaa_safe_harbor")
if any(finding.severity == "error" for finding in findings):
    raise ValueError("FHIR schema has uncovered identifier fields")

deidentified = apply_schema_policy(
    bundle,
    "fhir_hipaa_safe_harbor",
    date_shift_secret=date_shift_secret,
)

FHIR Bundle wrappers and their resources are inspected separately. Unsafe Bundle identifiers, signatures, request/response metadata, links, and external fullUrl values are removed. References to removed external URLs are rewritten to safe relative references when their target entry has a valid resource type and id. Opaque urn:uuid and urn:oid linkage remains intact. Resource types without rules in the selected schema policy are rejected instead of copied.

For OMOP, pass a mapping of linked table names to rows or use apply_omop_file for a CSV/Parquet file. Linked rows use person_id or an explicit subject_key for consistent date offsets. Blank optional dates stay blank, direct person source identifiers are suppressed, and clinical *_source_value fields are routed through the configured text de-identifier.

from openmed.structured import apply_omop_file, apply_schema_policy

tables = apply_schema_policy(
    {"person": person_rows, "visit_occurrence": visit_rows},
    "omop_hipaa_safe_harbor",
    date_shift_secret=date_shift_secret,
)

apply_omop_file(
    "visit_occurrence.csv",
    "deidentified/visit_occurrence.parquet",
    "omop_hipaa_safe_harbor",
    date_shift_secret=date_shift_secret,
)

Fail-closed behavior

The default allow-list covers the supported FHIR R4 resource subset. Binary resources, unknown resource types, malformed nested resource paths, malformed or unsafe XHTML narratives, scripts, external narrative links, and event attributes are rejected. Rejected resources are not copied to output. The gateway does not use a best-effort pass-through fallback for unsafe input.

REST jobs

The service exposes asynchronous local/SMART-compatible job routes:

  • POST /fhir/bulk/exports or POST /fhir/bulk/imports starts a job and returns 202 Accepted with a Content-Location polling URL.
  • GET /fhir/bulk/exports/{job_id} polls status; the import path is an alias.
  • GET /fhir/bulk/exports/{job_id}/manifest returns the PHI-free output manifest.
  • GET /fhir/bulk/exports/{job_id}/report returns the PHI-free job report.
  • DELETE /fhir/bulk/exports/{job_id} requests cancellation.

For an offline job, provide input_dir and output_dir. To use SMART backend-services authentication, omit input_dir and provide the configured FHIR base URL, token URL, client ID, and an operator-supplied private key. The key, client assertion, access token, raw resources, and source URLs are held only in the active request/job path; they are not returned, checkpointed, logged, or placed in manifests, exceptions, or reports.

SMART status polling honors Retry-After. Manifest file downloads are bounded by max_inflight_downloads and max_buffered_resources, and each file is written to a temporary sibling before an atomic rename. Keep credentials in the caller's secret-management system; do not put them in source-controlled fixtures or committed configuration.