Multimodal and ambient threat model¶
This STRIDE-style model covers the v3.3 media and ambient boundaries. It maps existing, limited controls to enforcing modules and offline tests, and links unfinished controls to tracked gaps. It adds no enforcement, recording permission, clinical validation, provider qualification or release approval. The release scope remains #3194.
Assets, adversaries and trust boundaries¶
Protect source media, hidden document content, model artifacts and configuration, microphone consent state, audio buffers, voice biometrics, transcripts, draft notes, source coordinates and clocks, review decisions, exports and audit evidence. An input author can supply hostile media or embedded instructions; an artifact supplier can substitute weights or preprocessing configuration; a participant can spoof a speaker; an output reader can link supposedly anonymous modalities; an operator or integration can accidentally bypass consent, review or deletion.
| Boundary | Untrusted input and required separation |
|---|---|
| TB1: media to parser/decoder | Headers, archives, pixels, document layers and waveform metadata are untrusted. Bounded preflight does not isolate native code. |
| TB2: artifacts to provider | Weights, tokenizer, preprocessing and prompts need pinned provenance. Source instructions never grant tool authority. |
| TB3: device to session | Microphone permissions do not establish participant consent. Capture, pause, interruption, participant changes and withdrawal need session policy. |
| TB4: session to storage/evidence | Audio, embeddings, transcripts and drafts are sensitive payloads. Logs and audit evidence admit only controlled codes, counts, offsets, opaque references and digests. |
| TB5: modalities to fused evidence | Page coordinates, frame selection, timing, speaker aliases and revisions must remain bound to their source; a digest alone cannot establish identity or truth. |
| TB6: draft to review/export | Generated text is untrusted, non-diagnostic and draft-only. Explicit reviewer confirmation must bind the exact consequential output before any downstream action. |
Python and OpenMedKit deployments share these boundaries. The matrix covers individual enforcing paths. Each integration must assess its adapters, device permissions, retention and export paths. Local-first means no mandatory network after approved model download, no default telemetry, and no PHI cloud fallback. Apple Foundation Models processing must remain on-device.
Threat/control matrix¶
STRIDE categories are Spoofing, Tampering, Repudiation, Information disclosure, Denial of service and Elevation of privilege. Stable MA identifiers support review. A Control names both a repository module and its enforcing test; Gap names an issue. Limited controls do not imply end-to-end mitigation. Residual risks are deployment conditions, not blanket risk acceptance.
| ID | STRIDE | Threat and boundary | Control or tracked gap; limits |
|---|---|---|---|
| MA-01 | Spoofing | A declared media type or digest disguises different bytes at TB1. | Control: openmed/multimodal/preflight.py; tests/unit/multimodal/test_preflight.py tests media mismatch and digest mismatch abstention. An accepted manifest does not authenticate its author. |
| MA-02 | Denial of service | Oversized media or missing geometry reaches a decoder at TB1. | Control — Image, PDF and DICOM redaction entry points apply bounded size and geometry admission before invoking their decoders. openmed/multimodal/redaction_admission.py, tests/unit/multimodal/test_redaction_admission.py. Native codec execution and source formats outside these entry points require separate controls. |
| MA-03 | Elevation of privilege / Denial of service | Native PDF, TIFF, DICOM, WSI, audio or video decoding crashes or escapes the parent process at TB1. | Control — Explicit Linux probes run in disposable workers with enforced resource and syscall limits. openmed/multimodal/decoder_guard.py, tests/integration/multimodal/test_hostile_media.py. The probe API covers selected formats. Deployments need OS containment and controlled host access (R1). |
| MA-04 | Denial of service | Nested or expanding document archives exhaust resources at TB1. | Control — Shared ZIP admission bounds directory parsing and expansion before supported Office, EPUB and ODT extraction. openmed/multimodal/archive.py, tests/unit/multimodal/test_archive_admission.py. These limits cover callers that use the shared admission policy. |
| MA-05 | Information disclosure | Source metadata survives an image or document export at TB4/TB6. | Control — The scrubber removes supported image EXIF/XMP and PDF and DOCX properties. openmed/multimodal/metadata_scrub.py, tests/unit/multimodal/test_metadata_scrub.py. Control — The supported TIFF path drops label and macro directories and strips identifying description metadata. openmed/multimodal/slide_deidentification.py, tests/unit/multimodal/test_slide_deidentification.py. Proprietary slide containers need separate qualification. |
| MA-06 | Information disclosure | Burned-in identifiers, barcodes or missed image regions survive OCR/redaction at TB1/TB6. | Control — The auditor checks covered regions and evidence digests against explicit expected identifiers. openmed/multimodal/privacy/pixel_coverage.py, tests/unit/multimodal/privacy/test_pixel_coverage.py. Control — Configured local barcode detection masks found code regions and verifies supported raster outputs before write-back. openmed/multimodal/code_regions.py, tests/integration/test_code_region_redaction.py. Detector recall and human review leave linkage and missed-identifier risk (R2). |
| MA-07 | Information disclosure | A PDF looks covered while selectable text remains underneath at TB6. | Control: openmed/multimodal/verify_pdf.py; tests/unit/multimodal/test_verify_pdf.py tests residual text, unchanged regions, empty spans and value-free reports. Verification covers supplied regions; it is not hidden-content discovery. |
| MA-08 | Information disclosure | PDF annotations, forms, attachments/revisions or hidden OOXML parts bypass visible-text redaction at TB1/TB6. | Control — Structural PDF preflight inventories annotations, forms, attachments and revisions and refuses unsupported or malformed input. openmed/multimodal/pdf_inventory.py, tests/integration/test_pdf_inventory_preflight.py. Control — Supported OOXML exporters refuse outputs with unresolved hidden parts. openmed/multimodal/ooxml_residual.py, tests/integration/test_ooxml_residual_outputs.py. Unknown document structures still require operator review. |
| MA-09 | Elevation of privilege / Tampering | OCR, visual text or spoken instructions redirect a model or tool workflow at TB2/TB6. | Control — Visual document screening quarantines suspicious regions before building the provider prompt. openmed/multimodal/vision/safety/instruction_injection.py, tests/unit/multimodal/vision/safety/test_instruction_injection.py. Screen only the supported regions. The host owns authorization for every tool dispatch. |
| MA-10 | Tampering / Elevation of privilege | Substituted VLM/ASR weights, tokenizer or preprocessing alter inference at TB2. | Control — Versioned preprocessing operations bind output geometry and tensor evidence. openmed/multimodal/preprocessing.py, tests/unit/multimodal/test_preprocessing.py. Control — The local speech adapter requires a qualified local profile and checks pinned artifacts. openmed/multimodal/streaming/providers.py, tests/unit/multimodal/test_offline_asr_provider.py. Approved local artifacts and the installed runtime still require trust and deployment-specific evidence (R3). |
| MA-11 | Information disclosure | A provider places source content or free-form errors in diagnostics at TB4. | Control: openmed/multimodal/provider_result.py; tests/unit/multimodal/test_provider_result.py rejects unknown fields and sensitive payloads without echo. This protects envelope serialization only, not provider-internal logging or returned text. |
| MA-12 | Tampering | A stale cache result survives a model, policy or preprocessing change at TB4. | Control: openmed/multimodal/cache_key.py; tests/unit/multimodal/test_cache_key.py tests version/option invalidation and rejects arbitrary metadata. Keys do not enforce retention, authenticate cache writers or encrypt content. |
| MA-13 | Spoofing / Repudiation | Device permission is mistaken for consent; capture resumes after withdrawal, participant change or background interruption at TB3. | Control — The session gates capture on affirmative consent and fences interruption, withdrawal and cohort changes. openmed/multimodal/streaming/session.py, tests/unit/multimodal/streaming/test_session_policy.py. Control — The native capture adapter binds device permission to the session state. swift/OpenMedKit/Sources/OpenMedKit/AmbientAudioCapture.swift, swift/OpenMedKit/Tests/OpenMedKitTests/AmbientAudioCaptureTests.swift. Operators own consent validity and recording-law obligations (R4). |
| MA-14 | Information disclosure / Repudiation | Buffers, caches, temporary media or crash artifacts retain audio/transcripts/drafts after cancellation, failure or deletion at TB4. | Control — The audio owner enforces retention deadlines, destroys owned buffers and records bounded cleanup evidence. openmed/multimodal/streaming/privacy/audio_retention.py, tests/unit/multimodal/streaming/privacy/test_audio_retention.py. Swap, backups and copies owned by callers or providers remain outside the deletion receipt (R1). |
| MA-15 | Spoofing / Information disclosure | Persistent voice embeddings enable identification or cross-session speaker linkage at TB3/TB5. | Control — Voice handles enforce same-session comparison and tied retention, with no public serialization. openmed/multimodal/voice_embeddings.py, tests/unit/multimodal/test_voice_embeddings.py. Control — Diarization preserves uncertain observations and separates similarity from identity. openmed/multimodal/streaming/speech/diarization.py, tests/unit/multimodal/streaming/speech/test_diarization.py. Speaker similarity alone cannot authenticate participants. |
| MA-16 | Tampering / Information disclosure | Partial, revised, low-confidence or wrong-language transcripts release identifiers or change doses/negation at TB4/TB6. | Control — Unstable or unscreened speech stays in the private withholding buffer. openmed/multimodal/streaming/privacy/withholding_buffer.py, tests/unit/multimodal/streaming/privacy/test_withholding_buffer.py. Control — The revision ledger keeps stable source bindings. openmed/multimodal/streaming/transcript/revision_ledger.py, tests/unit/multimodal/streaming/transcript/test_revision_ledger.py. Control — Language routing uses exact offsets and abstains when review is needed. openmed/multimodal/transcript_language.py, tests/unit/multimodal/test_transcript_language.py. Control — Critical-token review holds uncertain doses, numbers and negation at their exact evidence references. openmed/multimodal/transcript_holds.py, tests/integration/test_transcript_holds.py. |
| MA-17 | Tampering | Non-speech hallucinations, repetition loops or overlap stitching create unsupported statements at TB5/TB6. | Control — The local ASR provider screens hypotheses using non-speech and repetition evidence before releasing finalized text. openmed/multimodal/streaming/providers.py, tests/integration/multimodal/test_streaming_asr_provider.py. Control — Overlap reconciliation preserves lineage and holds ambiguous seams. openmed/multimodal/streaming/transcript/window_stitching.py, tests/unit/multimodal/streaming/transcript/test_window_stitching.py. Synthetic tests establish these boundaries. Clinical transcript accuracy requires separate evidence. |
| MA-18 | Spoofing / Tampering / Repudiation | Clock drift, swapped sources, wrong image coordinates or speaker roles produce false cross-modal citations at TB5. | Control — Clock translation binds explicit source clocks and refuses ambiguous alignment. openmed/multimodal/provenance/clock_alignment.py, tests/unit/multimodal/provenance/test_clock_alignment.py. Control — Fused evidence keeps source coordinates and identities distinct. openmed/multimodal/evidence/fused_graph.py, tests/unit/multimodal/evidence/test_fused_graph.py. Control — Uncertain speaker roles remain subject to review. openmed/multimodal/streaming/ambient/speaker_roles.py, tests/unit/multimodal/streaming/ambient/test_speaker_roles.py. A digest supports byte comparison. Acquisition authenticity remains a host responsibility. |
| MA-19 | Tampering / Elevation of privilege | Stale review, corrections or unsupported claims become a finalized note or clinical export at TB6. | Control — Review decisions bind exact current evidence and stale inputs are held. swift/OpenMedKit/Sources/OpenMedKit/MultimodalReview.swift, swift/OpenMedKit/Tests/OpenMedKitTests/MultimodalReviewTests.swift. Control — Synthetic draft evaluation checks unsupported claims, attribution and negation separately. openmed/eval/ambient_drafts.py, tests/unit/eval/test_ambient_drafts.py. Control — Corrections invalidate dependent approvals against the revision-bound graph. openmed/multimodal/ambient_corrections.py, tests/integration/multimodal/test_ambient_corrections.py. Control — Passive FHIR document export binds a reviewed draft to its exact evidence and status. openmed/clinical/exporters/fhir/ambient.py, tests/integration/test_ambient_fhir.py. Consequential use requires explicit reviewer confirmation of the exact output. |
| MA-20 | Information disclosure | VLM output reconstructs identifiers or raw retained audio remains identifiable despite safe text at TB6. | Control — The vision adapter screens generated identifiers and withholds source token IDs. openmed/multimodal/generated_text_guard.py, tests/integration/multimodal/test_generated_text_guard.py. Control — The muting path uses a separate PCM derivative and withholds audio when alignment is uncertain. openmed/multimodal/streaming/privacy/audio_muting.py, tests/unit/multimodal/streaming/privacy/test_audio_muting.py. Faces, voices and cross-modal linkage retain re-identification risk (R2). |
Unsupported deployments and residual risks¶
- R1: host compromise and incomplete erasure. Untrusted decoders in a shared PHI process without effective OS isolation, resource limits and controlled temporary storage are unsupported. This model and #3206 do not replace OS sandboxing or malware scanning. Host administrators, hostile in-process code, device theft, swap, crash dumps and backups remain outside library guarantees.
- R2: inference and re-identification. Unreviewed media/transcript exports, claimed universal identifier recall, biometric anonymity or clinical accuracy are unsupported. Inspect burned-in/hidden content and linked evidence before release. Negative fixtures and recall measurements bound evidence, not risk to zero. Digests can themselves enable linkage and need access/retention controls.
- R3: artifact/provider trust. Unpinned, unqualified or remotely executing VLM/ASR artifacts and cloud fallback for PHI are unsupported. Use permissively licensed or caller-supplied approved assets; do not bundle restricted weights or datasets. Offline execution does not make a model trustworthy. No provider or hardware qualification, benchmark result or release readiness is asserted here.
- R4: capture and review authority. Invisible/background capture without validated consent and interruption handling, indefinite audio/embedding retention and cross-session biometric identification are unsupported. The host must enforce consent and reviewers' authority. Confirmation is an explicit decision bound to current evidence and output, never inferred from a UI visit. Automated diagnosis, note finalization and EHR writes are unsupported.
Operators must document disposition of these risks for their deployment; this page does not accept them on behalf of participants. Update the mapping when an enforcing path and its tests land.
Evidence maintenance¶
The reference gate checks every inline repository module/test path and source link on this page, and requires each threat row to have a module/test pair or an issue link. Deleting a referenced module or test fails the offline gate. It does not run or semantically qualify the linked control tests; changes to behavior require human mapping review and the relevant test suites.
The enforcing docs gates are tests/unit/test_multimodal_threat_model.py, tests/integration/test_multimodal_threat_model.py and tests/unit/test_docs_publication.py.
.venv/bin/python -m pytest tests/unit/test_multimodal_threat_model.py tests/integration/test_multimodal_threat_model.py tests/unit/test_docs_publication.py -q
make docs-build
The page includes no source payload examples, sample PHI, credentials or private paths. Tests use synthetic reference trees and deletion mutations, not media or model output. Follow the disclosure policy for security reports. Related analyses cover the redactor, multilingual ingestion and agent boundary.