Secret rotation drill

Rotating a secret feels scary the first time and routine the tenth time. The trick is treating rotation as a three-step ritual instead of an emergency edit. Issue a second key at the provider, move the cluster onto it, then retire the old key once traffic proves out. Both keys stay valid at the provider during the window, so the service stays up the whole time.

terminal
bash
# 1. Issue a NEW key in the provider dashboard. Do not revoke the old
#    key yet: both stay valid at the provider during the overlap window.

# 2. Update the Secret in place with the new key, no YAML on disk
kubectl create secret generic enterprise-rag-secrets \
  --from-literal=OPENROUTER_API_KEY=sk-or-NEW-key \
  --dry-run=client -o yaml | kubectl apply -f -

# 3. Rolling restart so pods pick up the new key
kubectl rollout restart deployment/enterprise-rag
kubectl rollout status deployment/enterprise-rag

# 4. After validating traffic on the new key, revoke the OLD key
#    in the provider dashboard. No cluster change needed.

The overlap lives at the provider, not in the pod env. Pods that have not restarted yet keep sending the old key, and it still authorises because you have not revoked it. Once every replica is on the new key and error rates stay flat, revoking the old key is a dashboard click. The whole rotation completes without dropping a request.

One safety rail. Never delete the old key before validating the new one. Validation means watching error rate on the LLM provider call for at least one rolling restart cycle. Skipping that step is how a cleanup commit takes down production at 4pm on a Friday.

Quiz: Quiz

Loading practice…