Encrypted Service Backup and Restore¶
openmed recovery creates and restores encrypted, integrity-checked recovery points for the durable local state an OpenMed service needs to resume safely. Restore always writes to a new, inactive directory. The live state changes only at an explicit activation step, which re-verifies the restored files and then atomically repoints one symbolic link.
The same operations are available from Python in openmed.service.recovery.
What a recovery point contains¶
| Category | Source | Notes |
|---|---|---|
journey_metadata | local Journey journey.sqlite3 | captured with the SQLite online backup API, in memory |
journey_artifacts | artifacts/blobs/sha256/** | content-addressed blobs, already sealed by the Journey key |
journey_encryption | artifacts/encryption.json | key identifiers and verifiers only, never key material |
job_metadata | async job store JSON | metadata only; raw job payloads never leave memory |
model_registry | registry slot, pointer, and lineage state | schema version checked on restore |
audit_chain | audit-chain JSON files | each chain must verify before backup and after restore |
These categories are classified but never copied:
| Category | Disposition |
|---|---|
encryption_keys | excluded secret: resolved through caller-owned key providers |
service_credentials | excluded secret: API keys, JWKS, TLS keys, DSNs |
model_artifacts | rebuildable: re-acquire through verified downloads or an air-gapped kit |
result_cache | rebuildable |
sqlite_journal_files | transient: -wal/-shm content is folded in by the online backup |
staging_files | transient |
job_payloads | transient: in memory only |
PostgreSQL-backed Journey deployments keep using pg_dump and storage snapshots, as described in the Compose and Helm guides. This tool covers the local SQLite and filesystem state.
State layout and the live link¶
Restored state always uses one canonical layout under a state root:
<state-root>/
journey/journey.sqlite3
journey/artifacts/encryption.json
journey/artifacts/blobs/sha256/<2>/<64>
jobs/service-jobs.json
registry/registry_state.json
audit/chain-0000.json
Run the service against a symbolic link, for example /var/lib/openmed/state, that points at the current generation directory. Set OPENMED_SERVICE_JOBS_STORE_PATH=/var/lib/openmed/state/jobs/service-jobs.json and open any local Journey store at /var/lib/openmed/state/journey. Activation replaces that link with one atomic rename. A live path that is a real directory is refused (live_target_unmanaged); move the directory into a generation and create the link once while the service is stopped. Activation is not supported on Windows (atomic_swap_unsupported).
Keys¶
Backups are sealed with AES-256-GCM envelopes under the caller's service-backup key. Keys never enter a backup. The CLI reads keys from caller-owned files given as KEY_ID=FILE; each file holds exactly 32 raw bytes or 64 hexadecimal characters. The first --key is the active key. Add older keys after it, so recovery points sealed before a rotation still open.
The inventory prints categories, object counts, sizes, and schema versions only.
Back up¶
Stop admission and every writer first, then assert it with --writers-stopped. The Python API also accepts a quiesce context-manager factory that a drain coordinator can provide instead.
openmed recovery backup /backups/rp-0 \
--state-root /var/lib/openmed/state \
--key backup-2026=/run/secrets/openmed-backup.key \
--writers-stopped --retention-days 35
openmed recovery backup /backups/rp-1 --parent /backups/rp-0 \
--state-root /var/lib/openmed/state \
--key backup-2026=/run/secrets/openmed-backup.key --writers-stopped
A backup fingerprints every source before and after capture. If anything changed in between, the backup is refused (snapshot_boundary_moved). The point is written to a hidden sibling directory, verified by decrypting every stored object, and renamed into place only when complete.
An incremental point stores only objects whose digest changed since its parent and references the rest. The manifest records the parent manifest digest, the chain sequence, and the snapshot bounds (Journey revision, job count, audit entry counts and heads, and schema versions).
Verify, restore, and activate¶
openmed recovery verify /backups/rp-1 --ancestor /backups/rp-0 \
--key backup-2026=/run/secrets/openmed-backup.key --max-age-hours 48
openmed recovery restore /backups/rp-1 /var/lib/openmed/generations/g7 \
--ancestor /backups/rp-0 --live /var/lib/openmed/state \
--key backup-2026=/run/secrets/openmed-backup.key
openmed recovery activate /var/lib/openmed/generations/g7 /var/lib/openmed/state \
--key backup-2026=/run/secrets/openmed-backup.key \
--journey-key journey-2026=/run/secrets/openmed-journey.key
Restore refuses, and removes its partial target, unless every check passes:
- the manifest and every chunk authenticate under the backup key, so corruption, truncation, reordering, and objects moved between points fail;
- every object digest and size matches the manifest;
- the chain is complete, its parent digests and sequence numbers match, and its sequence bounds never regress;
- the point has not passed its retention or the
--max-age-hourslimit; - component schema versions are supported and the Journey migrations match known checksums;
- every Journey artifact row has its blob, and the encryption metadata agrees with the database;
- the restored content reproduces the manifest's snapshot boundary.
Only then is a sealed staging record written. activate authenticates that record, re-hashes every staged file, opens the restored Journey store with the Journey key (an integrity check plus reading every artifact), runs any further checks, and calls any recovery-epoch hooks. Only after all of that does it repoint the link. Without --journey-key, activation of a state that has a Journey store is refused. A restored Journey store older than this release's schema (a pre-upgrade recovery point restored to roll back to the previous release) is never opened, because opening would migrate it forward; it is verified read-only in memory instead (SQLite integrity, foreign keys, and a known migration history). Keep traffic stopped until activation completes and the service's own readiness checks pass.
From Python, pass the production self-check through checks= and components that must fence pre-restore authority (for example approval nonce stores) through epoch_hooks=. Each hook's open_recovery_epoch(context) receives the recovery-point id, manifest and content digests, and snapshot bounds before the swap; raising refuses activation.
The previous generation is kept. To roll back, restore again into a new directory: a generation that has served traffic no longer matches its staging record. openmed recovery discard <dir> --live <link> removes an inactive restored directory and never the live target.
Upgrades, preflight, and migration rehearsal¶
Durable schemas only migrate forward, and rolling an image back does not undo a migration. openmed migrations registry lists every durable schema (Journey SQLite and PostgreSQL, the artifact encryption marker, job metadata, model registry state, audit chains, recovery manifests, encrypted job payload files, the model activation journal and pointer state, and audit continuity manifests) with its current version, the versions it upgrades from, what the previous release's binary does with it, and a rollback class:
| Class | Rollback |
|---|---|
backward_compatible | the previous binary keeps working on the new schema |
restore_required | restore a recovery point taken before the upgrade |
release_blocking | no rollback path; the release gate refuses it |
Before an upgrade, run a dry-run plan against the stopped state. It writes nothing, refuses newer, drifted, or unknown versions before any migration runs, and blocks a restore_required step until a verified recovery point of the current state is supplied:
openmed recovery backup /backups/pre-upgrade --state-root /var/lib/openmed/state \
--key backup-2026=/run/secrets/openmed-backup.key --writers-stopped
openmed migrations preflight --state-root /var/lib/openmed/state \
--recovery-point /backups/pre-upgrade \
--key backup-2026=/run/secrets/openmed-backup.key
The plan also reports free space, encryption-key availability, cross-component consistency, an unfinished recovery activation, and whether writers must stop. Continuity manifests beside the audit chains are observed automatically. The encrypted job payload directory and the model activation root live outside the state root, so the plan reports them as not_inspected unless the Python API (plan_service_migrations) is given their locations. To roll back, restore and activate the pre-upgrade point as above and start the previous release.
openmed migrations rehearse --signing-key gate-2026=/run/secrets/gate.key exercises every registered schema offline against synthetic fixtures: clean and repeated upgrades from every supported version, a migration killed mid-transaction, drift, a schema written by a newer binary, the previous release's binary, and restore-based rollback followed by a retry. It prints a value-free compatibility matrix and gate verdict, signed with HMAC-SHA256 when a key is given. PostgreSQL runs against an offline DB-API double there; the Python API accepts a connection factory for a disposable server.
Receipts and reason codes¶
Every command prints a versioned receipt (openmed.service.recovery-receipt.v1) holding the opaque recovery-point id, component categories, digests, sizes, schema versions, snapshot bounds, counts, timestamps, and one controlled reason code. Receipts, manifests, and errors never contain paths, file names, key identifiers, key material, or state content. Refusals exit with status 1; usage errors exit with status 2.
| Reason | Meaning |
|---|---|
quiesce_required | no drain hook and no --writers-stopped assertion |
snapshot_boundary_moved | a source changed during capture |
manifest_authentication_failed | wrong key or tampered manifest |
object_authentication_failed, object_corrupt | tampered, truncated, or reordered object |
chain_incomplete, chain_invalid | missing or substituted increment |
schema_incompatible | a component schema this release cannot read |
cross_reference_missing_artifact, snapshot_boundary_mismatch | restored components disagree |
recovery_point_expired, recovery_point_stale | retention or maximum age exceeded |
restore_incomplete, staged_state_modified | the staged directory cannot be activated |
activation_check_failed | a pre-activation check failed; details names it |
Limits¶
- The Journey database is snapshotted into memory, so backup memory grows with the database size.
- The CLI cannot drain a running service; stop writers before
backup. - PostgreSQL Journey stores, remote artifact stores, and cloud snapshots are out of scope.