Service incident runbooks¶
These runbooks cover four incidents for a self-hosted OpenMed REST service and its Journey deployment profile:
| Incident | Runbook |
|---|---|
| Leaked API key, JWKS signing material or job webhook secret | Credential compromise |
| Compromised mTLS client certificate | Client certificate compromise |
| Full disk under the model cache, async job store or Journey artifacts | Disk exhaustion |
| Damaged async job store or Journey store | Job and Journey store corruption |
Each runbook gives detection signals, containment, rotation or recovery, verification, and what must stay out of tickets and logs. They describe the settings, commands and reason codes that exist in the current release. Steps that need recovery tooling that has not shipped yet are marked pending. Use the breach notification runbook if you suspect PHI was exposed, and the general key management guide for audit and surrogate keys.
All examples use synthetic values. Replace <placeholders> with values from your own secret manager or deployment. Do not paste real values into a ticket.
How settings take effect¶
The REST service reads authentication, mTLS, and job-store settings once, when openmed.service.app.create_app builds the application. To change an API key, JWKS, CA bundle, principal mapping or job-store path:
- Update the value in your secret store, then in the process environment. With Helm, add them as Helm
extraEnventries that reference a Kubernetes Secret. - Restart the service.
OPENMED_SERVICE_SHUTDOWN_DRAIN_SECONDSbounds how long in-flight requests can finish during shutdown. - Run the verification steps in the runbook.
Restarts drop work that exists only in memory. Queued and running async jobs keep their submitted text in memory only, so a restart loses them. Their stored metadata stays at its last state. With more than one replica, restart them one at a time so the service stays available.
Rules for tickets, logs and chat¶
These rules apply to every step in every runbook:
- Never paste API keys, bearer tokens, JWKS documents, HMAC or webhook secrets, private keys, certificates in PEM form, DSNs or database passwords. Refer to a credential by its configured
id, its JWKSkid, or a certificate SHA-256 fingerprint. Only do this when the label contains no personal name. - Never attach the job store file, Journey database dumps, artifact blobs, model inputs, request or response bodies, or webhook payloads. These can contain document IDs, span offsets, label counts and text hashes linked to patients.
- Record reason codes, HTTP status codes, counts, timestamps, sizes, and SHA-256 digests of quarantined files. Do not record contents.
- Do not record private paths, host names of webhook receivers, or client IP addresses unless your incident process requires it and stores it with restricted access.
- Do not turn on debug logging, shell tracing, or environment dumps to investigate. These can print secrets. Never print a secret in a shell. Use files with mode
0600or a secret manager to pass values. - Keep quarantined files on encrypted storage that only the incident owners can read. Destroy them according to your retention policy.
Reason codes¶
The service and Journey tools return these controlled codes. Their output never includes the input value. Use the code to choose a runbook.
| Code | Where it appears | Runbook |
|---|---|---|
invalid_credentials | REST 401 error envelope for an unknown API key or a rejected JWT | Credential compromise |
authentication_required | REST 401 when no credential was sent | Credential compromise |
auth_rate_limited | REST 429 after repeated failed authentication | Credential compromise |
forbidden | REST 403 when a valid principal lacks a route scope | Client certificate compromise |
mtls_certificate_required | REST 401 when no verified client certificate arrived | Client certificate compromise |
mtls_certificate_invalid | REST 401 when the chain does not verify against the CA bundle | Client certificate compromise |
not_ready | REST 503 from GET /readyz or during shutdown | Disk exhaustion |
internal_error | REST 500, for example when the job store cannot be written | Disk exhaustion |
artifact_write_failed | Journey probe, worker readiness, or artifact store write | Disk exhaustion |
artifact_store_unavailable | Journey probe when the artifact root cannot be opened | Disk exhaustion |
artifact_read_failed | Journey artifact read or probe read-back failure | Job and Journey store corruption |
artifact_integrity_failed | Journey artifact read whose bytes no longer match their digest | Job and Journey store corruption |
artifact_verify_failed | Journey artifact write that failed its read-back digest check | Disk exhaustion |
transaction_failed | Journey metadata transaction that rolled back | Disk exhaustion |
commit_status_unknown | Journey metadata commit whose outcome cannot be proven | Job and Journey store corruption |
postgres_connect_failed | Journey store connection failure | Job and Journey store corruption |
store_open_failed | Journey store failed to open or migrate | Job and Journey store corruption |
schema_unsupported | Journey schema is newer than this release, or a migration checksum drifted | Job and Journey store corruption |
sqlite_integrity_failed | Local SQLite Journey store failed its structure check | Job and Journey store corruption |
foreign_key_integrity_failed | Journey store has broken or unvalidated relationships | Job and Journey store corruption |
stored_payload_invalid | Journey store row payload cannot be parsed | Job and Journey store corruption |
payload_hash_mismatch | Journey store row no longer matches its stored digest | Job and Journey store corruption |
The async job store does not return a reason code when it refuses to start. The service stops at startup with one of the fixed messages listed in Job and Journey store corruption.
Pending capabilities¶
The current release does not include these capabilities. The runbooks give the manual steps that work today and do not depend on them.
| Pending capability | Tracking |
|---|---|
| Reload API-key and JWT verifier sets without a restart, with an overlap window | Pending in #3699 |
| Reload TLS and mTLS credentials without dropping in-flight work | Pending in #3580 |
| Durable async jobs with leases, checkpoints, quarantine and replay | Pending in #3574 |
| Transactional model download, verification, activation and rollback | Pending in #3575 |
| Encrypted, integrity-checked backup and restore of service state | Pending in #3577 |
| Bounded no-PHI diagnostic and recovery bundle | Pending in #3600 |
| Local operator CLI for inspect, plan, verify and apply recovery actions | Pending in #3703 |
| Release recovery drill that exercises these runbooks | Pending in #3603 |
Until these ship, there is no read-only store inspection command, no bulk artifact sweep, no job replay, and no credential hot reload. Do not describe any of them as available in an incident record.