Disk exhaustion runbook¶
Use this runbook when the volume behind the model cache, the async job store, or the Journey artifact store is full or nearly full. Read the shared rules first.
Where data is stored in the reference deployments:
| Data | Setting | Compose profile | Helm chart |
|---|---|---|---|
| Model cache | HF_HOME, HF_HUB_CACHE, OPENMED_CACHE_DIR | Compose volume openmed-cache at /cache | Helm persistence.size at Helm persistence.mountPath |
| Async job metadata | OPENMED_SERVICE_JOBS_STORE_PATH (default ~/.cache/openmed/service-jobs.json) | Not set; uses the default | Not set; uses the default |
| Journey artifacts | OPENMED_JOURNEY_ARTIFACT_ROOT | Compose volume openmed-cache at /cache/artifacts | Helm journey.artifacts.size at Helm journey.artifacts.mountPath |
| Journey metadata | OPENMED_JOURNEY_POSTGRES_DSN | Compose volume openmed-postgres | External or operator-managed PostgreSQL |
In the Compose profile, models and Journey artifacts share the Compose volume openmed-cache. If the model cache fills it, artifact writes also fail. Under Helm they use separate claims.
Never free space by deleting Journey artifact blobs, the job store, PostgreSQL data files, or SQLite -wal files. Artifacts are content-addressed evidence that metadata rows reference. Deleting them turns a capacity incident into a store corruption incident.
Model cache volume full¶
Detection signals¶
- Model downloads fail, or the first request for a model that is not loaded yet returns
500withinternal_error. - With
OPENMED_SERVICE_PRELOAD_MODELSset, startup fails during preload, orGET /readyzstays at503withnot_ready. - In the Compose profile, the Journey probe also reports
artifact_write_failed, because the volume is shared. - Volume usage alerts from your platform.
Containment¶
- Stop new downloads. The Compose profile already sets
OPENMED_OFFLINE,HF_HUB_OFFLINEandTRANSFORMERS_OFFLINE. Set them in other deployments. - Do not start more replicas on the same volume until space is free.
Recovery¶
-
List cached models and their sizes. The output contains repository IDs, sizes and last-access times only:
-
Remove models that are not in
OPENMED_SERVICE_PRELOAD_MODELSand not served by any client: -
If the cache is still too small, expand the volume. Under Helm, raise Helm
persistence.sizeif the storage class supports expansion, or move to a larger claim with Helmpersistence.existingClaim. - A download interrupted by a full disk can leave a partial snapshot. Clear that model and fetch it again. Use
openmed models pull <repo-id>on a connected host. For an air-gapped site, useopenmed airgap bundleon a connected host andopenmed airgap verifybefore you copy the kit in. Transactional download with rollback is pending in #3575.
Verification¶
Each model with integrity metadata must report PASS. Then confirm that GET /readyz returns 200 and that a synthetic request to a preloaded model succeeds.
Keep out of tickets and logs¶
- Request text used to reproduce the failure. Use synthetic text only.
- Hugging Face tokens (
HF_TOKEN) used to fetch models again. See the Hugging Face token policy. - Private host paths. Record the volume name, its size, and the used bytes.
Async job store volume full¶
The job store writes a complete new file next to the old one and then atomically replaces it. If the disk is full, the write fails. The temporary file is removed, and the previous file stays intact.
Detection signals¶
POST /jobsreturns500withinternal_errorbecause the new record cannot be saved.- Jobs stop advancing.
GET /jobs/{job_id}keeps returning an olderstatusor progress because the worker cannot save updates. openmed_service_request_totalshows500responses for/jobs.
Containment¶
- Stop or throttle
POST /jobsat your proxy, or with the service rate limitsOPENMED_SERVICE_RATE_LIMIT_RPSandOPENMED_SERVICE_RATE_LIMIT_BURST. - Lower
OPENMED_SERVICE_JOBS_TTL_SECONDSbefore the next restart so that jobs finishing after the restart expire sooner. Records that already finished keep their stored expiry time. Expired records are removed when other records are created or read.
Recovery¶
- Free space on that volume by removing data that does not belong to OpenMed stores, or expand the volume.
- To move the store, stop the service and copy the file to a larger volume with mode
0600. Then setOPENMED_SERVICE_JOBS_STORE_PATHto the new path and restart. - Jobs that were queued or running when the service restarted cannot resume. Their text lived only in memory. Clients must submit them again. Durable job replay is pending in #3574.
Verification¶
- Submit a synthetic job with
POST /jobsand pollGET /jobs/{job_id}until it reachesdone. - Run the job store check in the store corruption runbook. It must report
job store: loadable.
Keep out of tickets and logs¶
- The job store file, or excerpts from it. It contains document IDs, span offsets, labels and text hashes for every retained job.
- Job IDs together with document IDs. Record counts of affected jobs instead.
Journey artifact volume full¶
Detection signals¶
- The Journey worker readiness endpoint and the Journey probe command report the artifact component as unavailable with
artifact_write_failed, orartifact_verify_failedfor a truncated write. The worker writes and reads back a small synthetic probe artifact on each refresh. - Ingestion calls return a
failurestore result withartifact_write_failed. A metadata write on a full PostgreSQL volume returnstransaction_failedorcommit_status_unknown. - In the Compose profile, the
openmedhealth check runs the probe. It turns the container unhealthy.
Containment¶
- Pause the jobs that write new artifacts. Leave the stored data in place.
- Do not retry a write that returned
commit_status_unknownblindly. The commit may have succeeded. Read the record back first.
Recovery¶
- Expand the volume. Under Helm, raise Helm
journey.artifacts.sizeif the storage class supports expansion, or attach a larger claim with Helmjourney.artifacts.existingClaimafter copying the data with its permissions preserved. - In the Compose profile, free space on the shared volume by clearing unused models. See Model cache volume full.
- For PostgreSQL, add capacity using your database tooling. OpenMed does not manage database storage. Backup and restore tooling for service state is pending in #3577.
Verification¶
docker compose -f deploy/openmed-compose.yaml exec openmed \
python3 -m openmed.deploy.journey probe \
--worker-url http://journey-worker:8091/readyz \
--model-url http://127.0.0.1:8080/readyz
The JSON output lists controlled states and codes only. The artifact component must show artifact_store_ready, and the store component must show store_ready. The probe reads the DSN from OPENMED_JOURNEY_POSTGRES_DSN and never prints it. The Compose service journey-golden (profile smoke) runs python -m openmed.deploy.journey golden. It writes one synthetic record and expects golden_journey_verified. Run it only where a synthetic record in the production store is acceptable.
Keep out of tickets and logs¶
- Artifact blobs, file names under the artifact root, and content digests that you could match to a patient document. Record counts and used bytes.
OPENMED_JOURNEY_POSTGRES_DSNandOPENMED_POSTGRES_PASSWORD. Do not pass the DSN on a command line.