Credential compromise runbook¶
Use this runbook when a REST API key, JWT signing material, or an async job webhook secret may have been exposed. For example, it may have been pasted in a ticket or chat, committed to a repository, found in a support archive, or used by an unknown caller. Read the shared rules first. For a client certificate, use the client certificate runbook.
Credential changes take effect only after a restart. Verifier hot reload is pending in #3699.
Leaked API key¶
API keys are configured in OPENMED_SERVICE_AUTH_API_KEYS as a JSON list of entries with id, principal, key_hash and scopes. The service stores only the SHA-256 hash. A caller sends the raw key in the X-API-Key header or as Authorization: ApiKey <key>.
Detection signals¶
- A secret scanner or a person reports the raw key outside your secret store. See secret scanning.
openmed_service_request_totalshows more401,403or429responses on a route. This metric exists only whenOPENMED_SERVICE_METRICS_ENABLEDis on. The429responses use theauth_rate_limitedcode, which the failed authentication limiter (OPENMED_SERVICE_AUTH_FAILURE_RATE_LIMIT_RPS,OPENMED_SERVICE_AUTH_FAILURE_RATE_LIMIT_BURST) returns.- Access logs show unusual volume, routes or times. Each access-log record has the method, route template, status code, duration, model name and request ID. API-key and JWT requests do not record the key
idin the access log, so you cannot attribute traffic to one key from these logs alone.
Containment¶
- Identify the affected entry by its
id. Do this by hashing the reported key locally (see Rotation) and comparing the hash with the configuredkey_hashvalues. Do not send the key to anyone. - If the exposure is confirmed or likely, remove that entry from
OPENMED_SERVICE_AUTH_API_KEYSand restart the service. Requests with the removed key then fail withinvalid_credentials. - If you cannot restart quickly, block the affected routes or callers at your reverse proxy until the restart is done. OpenMed cannot revoke a key while it is running.
- Make sure
OPENMED_SERVICE_AUTH_ENABLEDistrue. KeepOPENMED_SERVICE_AUTH_DENY_BY_DEFAULTat its default oftrueso that a route without a declared scope still requires a credential.
Rotation¶
Use an overlap only when the old key was not exposed. Examples are scheduled rotation and an operator leaving. For a confirmed leak, remove the old entry in the same restart as you add the new one.
-
Generate the new key in your secret manager. Hash it without echoing it. The command below reads the key from a hidden prompt and prints only the
sha256:hash: -
Add a new entry with a new
idand the printedkey_hash. Do not reuse an oldid. Keep the scopes as narrow as the caller needs. - Restart the service and give the new key to the caller through the secret manager.
- For an overlap, remove the old entry in a second restart after the caller has switched.
Recovery¶
- Review what the compromised key could reach: its
scopesand the routes in REST authentication. For example, a key withjobs:readcould read job metadata withGET /jobs/{job_id}if the caller also knew a job ID. - If a route that returns PHI may have been used, start the breach notification runbook.
Verification¶
-
Confirm that the old key is rejected. Put the header line
X-API-Key: <old key>in a file with mode0600, written by your secret manager. Then send it to a scoped route. Using a header file keeps the key out of the command line and shell history:curl -sS -o /dev/null -w '%{http_code}\n' -H @"$OLD_KEY_HEADER_FILE" \ http://127.0.0.1:8080/models/loadedThe expected result is
401. Repeat with the new key's header file. The expected result is200, or403if that key does not havemodels:read. Delete both header files afterwards. -
Check that the
401and429rates return to their normal level.
Keep out of tickets and logs¶
- The raw key, the header file, and any shell history that contains the key. Record only the entry
idand the time it was removed. - The configured
OPENMED_SERVICE_AUTH_API_KEYSvalue. A hash cannot recover the key, but the full list shows which principals and scopes exist.
Leaked JWKS or JWT signing key¶
OPENMED_SERVICE_AUTH_JWKS_FILE and OPENMED_SERVICE_AUTH_JWKS hold the keys that verify bearer tokens. The service supports HS256 keys (kty oct), whose k member is a shared secret, and RS256 public keys (kty RSA). Keys from the file and the inline value are combined. The impact depends on what was exposed:
- An
octkey, or a JWKS document that contains one, is a shared secret. Anyone who has it can mint accepted tokens. - An
RSApublic key is not secret. The incident is the issuer's private key, which does not belong in OpenMed configuration.
Detection signals¶
- A JWKS document that contains an
octkey appears outside your secret store, or your identity provider reports a signing-key compromise. - Tokens arrive with an unexpected
kid, issuer or audience. The service rejects them withinvalid_credentials. Watch for401spikes inopenmed_service_request_total. - Valid-looking traffic arrives from callers you do not recognize. Token claims are not written to access logs, so use your identity provider's issuance records.
Containment¶
- Remove the compromised key from both
OPENMED_SERVICE_AUTH_JWKS_FILEandOPENMED_SERVICE_AUTH_JWKS, then restart. All tokens signed with that key then fail withinvalid_credentials. They fail even if they have not expired yet. - Check that
OPENMED_SERVICE_AUTH_JWT_ISSUERandOPENMED_SERVICE_AUTH_JWT_AUDIENCEare set. This stops tokens minted for other services from being accepted. - Keep
OPENMED_SERVICE_AUTH_JWT_MAX_LIFETIME_SECONDSshort. The production profile defaults it to one hour. It limits how long any token signed before the incident stays usable.
Rotation¶
- Create the new signing key at the issuer with a new, unique
kid. - Add the new public key, or the new
octkey, to the JWKS with thatkid. When more than one key matches the token algorithm, the service requires akidin the token header and accepts only an exact match. Tokens without akidare rejected during an overlap. Akidused twice rejects every token that carries it. - For a confirmed compromise, remove the old key in the same restart. For a scheduled rotation, keep both keys until tokens signed with the old key have expired, then remove the old key in a second restart.
Recovery¶
- Ask the issuer to stop issuing tokens with the old key and to revoke any client credentials that may have been exposed with it.
- Review which scopes those tokens could carry (
scope,scporscopesclaims). Start the breach notification runbook if a PHI-returning route may have been reached.
Verification¶
- Present a token signed with the old key. The expected result is
401withinvalid_credentials. Use a synthetic test client that the issuer can revoke afterwards. - Present a token from the new key with the expected issuer and audience. The expected result is
200, or403withforbiddenif a scope is missing.
Keep out of tickets and logs¶
- JWKS documents, the
kmember of any key, private keys, and tokens. Record only thekidvalues that were removed and added. - Decoded token claims. They can name people or systems. Record counts of rejected requests instead.
Leaked job webhook secret¶
A webhook secret is sent with each POST /jobs request in webhook.secret. The service signs the terminal webhook for that job with it. There is no service-wide webhook setting. The secret is kept in memory with the queued job and is not written to the job store. The store keeps only url_hash, the retry settings, and the webhook_delivery result. Your receiver verifies the signature, for example with openmed.service.signing.verify_request_signature.
Detection signals¶
- The receiver accepts a webhook with a valid signature for a job that you did not submit.
- The receiver's replay protection (
NonceCache) reports reused nonces. Or it sees deliveries whose timestamps fall outside its window. - The secret appears in a client config file, a log, or a ticket.
Containment¶
- Change the receiver so it accepts only the new secret. The service does not need a restart for this.
- Stop sending the old secret in new
POST /jobsrequests.
Rotation¶
Jobs that were queued before the change still sign with the old secret. To keep them deliverable during a scheduled rotation, the receiver can accept either secret for a short, fixed window. Try the new secret first:
from openmed.service.signing import (
InvalidSignatureError,
verify_request_signature,
)
def verify_with_overlap(method, path, body, headers, *, secrets, nonce_cache):
"""Accept a delivery signed by any secret in ``secrets``.
Failed attempts do not consume the nonce, so trying a second secret
keeps replay protection intact. Timestamp and replay errors still raise.
"""
for secret in secrets:
try:
return verify_request_signature(
method,
path,
body,
headers,
secret=secret,
nonce_cache=nonce_cache,
)
except InvalidSignatureError:
continue
raise InvalidSignatureError("Invalid request signature")
For a confirmed leak, do not use an overlap. Pass only the new secret.
Recovery¶
- When a delivery fails, the job's
webhook_deliveryrecords it. Callers can still pollGET /jobs/{job_id}for the terminal status, so a rejected webhook does not lose the result. - Treat webhooks that were accepted with the old secret during the exposure window as untrusted. Re-read their status with
GET /jobs/{job_id}.
Verification¶
- Submit a synthetic job with the new secret. Confirm the receiver accepts the delivery. Confirm
webhook_deliveryshows success inGET /jobs/{job_id}. - Confirm the receiver rejects a synthetic delivery signed with the old secret.
Keep out of tickets and logs¶
- Both secrets, the webhook URL, and receiver host names. The store keeps only
url_hashfor this reason. - The text of
webhook_deliveryerrors. It comes from the HTTP client and can name the receiver. - Webhook bodies. They include job IDs, document IDs, span offsets and label counts.