Deploy runbook: notes API service

This is the runbook we follow to ship a new release of the notes API. Each step has a short caption explaining what the command does and what you should expect to see, followed by the exact snippet to run. Snippets are deliberately short so you can copy one block at a time and verify the result before moving on.

The language label above each block tells you which interpreter or file it belongs to - there is no token colouring, so the caption and the label are how you orient yourself.

Pre-flight checklist

Before touching production, walk this list top to bottom. Anything still unchecked is a blocker.

• [X] Release branch merged to main and CI is green
• [X] Database migration reviewed by a second engineer
• [ ] Staging smoke test passed in the last hour
• [ ] On-call engineer acknowledged in the deploy channel
• [ ] Rollback image tag noted in the incident doc

Step 1 - Confirm the build artifact

Pull the latest image digest and confirm it matches the tag CI published. The command prints the digest; compare it against the value in the release ticket before continuing.

docker pull registry.example.com/notes-api:1.8.0
docker inspect --format='{{index .RepoDigests 0}}' \
  registry.example.com/notes-api:1.8.0

Step 2 - Review the deploy manifest

The deployment is described by a small Kubernetes manifest. Check that replicas and the image tag are what you expect; a stale tag here is the most common cause of "the fix didn't ship".

apiVersion: apps/v1
kind: Deployment
metadata:
  name: notes-api
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: notes-api
          image: registry.example.com/notes-api:1.8.0
          envFrom:
            - secretRef:
                name: notes-api-secrets

Step 3 - Apply the configuration change

This release flips the read-through cache on. The config is stored as JSON in the config map; the snippet below is the exact payload we apply. Expect the API to log cache: enabled on next boot.

{
  "cache": {
    "enabled": true,
    "ttlSeconds": 300,
    "maxEntries": 10000
  },
  "featureFlags": {
    "inlineAttachments": false
  }
}

Step 4 - Roll the deployment

Trigger the rolling update and watch the rollout. The command blocks until all pods are healthy, then prints deployment "notes-api" successfully rolled out. If it stalls, jump to the rollback step.

kubectl apply -f deploy/notes-api.yaml
kubectl rollout status deployment/notes-api --timeout=120s

Step 5 - Verify the health endpoint

Once pods report ready, hit the health check from a script so the output is parseable. This small fetch helper returns the parsed body and throws on a non-200, which is what our post-deploy CI gate calls.

export async function checkHealth(baseUrl: string): Promise<unknown> {
  const res = await fetch(`${baseUrl}/healthz`);
  if (!res.ok) {
    throw new Error(`health check failed: ${res.status}`);
  }
  return res.json();
}

Step 6 - Run the post-deploy probe

For a deeper check we run a short Python probe that exercises a real read path and measures latency. A healthy run prints a single line with the status code and elapsed milliseconds.

import time
import urllib.request

start = time.perf_counter()
with urllib.request.urlopen("https://notes-api.example.com/v1/notes?limit=1") as r:
    status = r.status
elapsed_ms = round((time.perf_counter() - start) * 1000)
print(f"status={status} elapsed_ms={elapsed_ms}")

Step 7 - Spot-check the database

Confirm the migration landed and no rows were orphaned. This query counts notes created in the last hour grouped by status; a sudden zero in published usually means the writer can't reach the queue.

SELECT status, COUNT(*) AS n
FROM notes
WHERE created_at > now() - interval '1 hour'
GROUP BY status
ORDER BY n DESC;

Rollback - if any check fails

If a step fails, revert to the previous image tag. The change is a one-line edit to the manifest; apply it and re-run the rollout from Step 4. The diff below shows exactly what changes.

       containers:
         - name: notes-api
-          image: registry.example.com/notes-api:1.8.0
+          image: registry.example.com/notes-api:1.7.4

After the deploy

Once everything is green, post the release digest in the deploy channel and tick the remaining pre-flight boxes for the record. Close the incident doc only after the post-deploy probe has run clean twice, ten minutes apart.

Demonstrates NoteThink's fenced code rendering - language-labelled blocks across shell, YAML, JSON, TypeScript, Python, SQL, and diffs travel inline with your notes.