Infra & deploys

CI/CD, rollout.sh, Terraform, clouds, observability, runbooks.

Source repo: quill-router (TrustedRouter control plane) at main @ 778120494, read on 2026-10-19. Every claim cites path:line relative to the repo root. Nothing here was checked against live production. Where docs and code disagree, both are noted.

Trust boundary in one sentence. This repo deploys the control plane (billing, keys, routing metadata, website/console). The attested prompt path (Confidential Space / Nitro / SEV-SNP enclaves) is deployed from the separate quill-cloud-proxy repo (.github/workflows/deploy.yml:1-7). If a control-plane deploy goes wrong, the site goes down or keys stop working. Prompt confidentiality is not affected.

1. Deployment topology

flowchart TB
  user["Clients / SDKs / browsers"]

  subgraph CF["Cloudflare + Google Cloud DNS (both authoritative)"]
    dns["DNS for trustedrouter.com; DNS-only LB for gateway regions (no orange-cloud on prompt path)"]
  end

  subgraph GCP["GCP project quill-cloud-proxy (primary cloud)"]
    armor["Cloud Armor policies: legacy_edge, public_edge, internal_edge, lightning_funding"]
    glb["Global HTTPS LB: trusted-router-control-map, backend trusted-router-control-backend"]
    subgraph CR["Cloud Run service trusted-router (+ trusted-router-public, trusted-router-internal)"]
      cr1["us-central1 PRIMARY, min 8"]
      cr2["us-east4, min 8"]
      cr3["europe-west4, min 8"]
      cr4["southamerica-east1, min 2, gateway billing failover"]
    end
    cs["Confidential Space attested gateways: api.trustedrouter.com; us-central1, us-east4, europe-west4, us-west1"]
    spanner[("Spanner trusted-router-nam6, ENTERPRISE_PLUS, autoscale 1000-10000 PU, 45 pct CPU target")]
    ch[("ClickHouse tr-clickhouse-1..3, us-central1-a/b/c, 3 Keeper voters")]
    fp["Fastpath Go nodes x2 us-central1 + Pub/Sub settle log (dark install)"]
    jobs["Cloud Run Jobs: synthetic monitor, trust reconciler/tier jobs"]
    gcs[("GCS: deploy mutex journal, tfstate, fastpath releases")]
  end

  subgraph AWS["AWS account 330422590279 (EU, standalone)"]
    ga["Global Accelerator tr-eu-control-plane to NLB"]
    ecs["ECS Fargate tr-cp-euw1 / tr-cp-euw3"]
    nitro["Nitro enclaves quill-enclave-asg: eu-west-1 x2, eu-west-3 x2; api-aws.trustedrouter.com"]
    dsql[("Aurora DSQL")]
    awsch[("ClickHouse eu-west-3 + eu-north-1")]
    apprun["App Runner tr-eu: synthetic probes via EventBridge 1-min rule"]
  end

  subgraph AZ["Azure (standalone)"]
    aca["Container App control plane, uaenorth"]
    snp["SEV-SNP enclaves uaenorth + australiaeast; api-azure.trustedrouter.com"]
    pg[("Postgres Flexible Server")]
    azch[("ClickHouse")]
  end

  user --> dns
  dns --> glb
  dns --> cs
  dns --> ga
  dns --> aca
  dns --> snp
  armor --- glb
  glb --> cr1
  glb --> cr2
  glb --> cr3
  glb --> cr4
  cs -->|"/internal/gateway authorize + settle"| glb
  cr1 --> spanner
  cr2 --> spanner
  cr3 --> spanner
  cr4 --> spanner
  cr1 -->|"outbox to ingester"| ch
  fp --> spanner
  ga --> ecs
  ecs --> dsql
  ecs --> awsch
  nitro --> ecs
  aca --> pg
  aca --> azch
  snp --> aca

flowchart TB
  user["Clients / SDKs / browsers"]

  subgraph CF["Cloudflare + Google Cloud DNS (both authoritative)"]
    dns["DNS for trustedrouter.com; DNS-only LB for gateway regions (no orange-cloud on prompt path)"]
  end

  subgraph GCP["GCP project quill-cloud-proxy (primary cloud)"]
    armor["Cloud Armor policies: legacy_edge, public_edge, internal_edge, lightning_funding"]
    glb["Global HTTPS LB: trusted-router-control-map, backend trusted-router-control-backend"]
    subgraph CR["Cloud Run service trusted-router (+ trusted-router-public, trusted-router-internal)"]
      cr1["us-central1 PRIMARY, min 8"]
      cr2["us-east4, min 8"]
      cr3["europe-west4, min 8"]
      cr4["southamerica-east1, min 2, gateway billing failover"]
    end
    cs["Confidential Space attested gateways: api.trustedrouter.com; us-central1, us-east4, europe-west4, us-west1"]
    spanner[("Spanner trusted-router-nam6, ENTERPRISE_PLUS, autoscale 1000-10000 PU, 45 pct CPU target")]
    ch[("ClickHouse tr-clickhouse-1..3, us-central1-a/b/c, 3 Keeper voters")]
    fp["Fastpath Go nodes x2 us-central1 + Pub/Sub settle log (dark install)"]
    jobs["Cloud Run Jobs: synthetic monitor, trust reconciler/tier jobs"]
    gcs[("GCS: deploy mutex journal, tfstate, fastpath releases")]
  end

  subgraph AWS["AWS account 330422590279 (EU, standalone)"]
    ga["Global Accelerator tr-eu-control-plane to NLB"]
    ecs["ECS Fargate tr-cp-euw1 / tr-cp-euw3"]
    nitro["Nitro enclaves quill-enclave-asg: eu-west-1 x2, eu-west-3 x2; api-aws.trustedrouter.com"]
    dsql[("Aurora DSQL")]
    awsch[("ClickHouse eu-west-3 + eu-north-1")]
    apprun["App Runner tr-eu: synthetic probes via EventBridge 1-min rule"]
  end

  subgraph AZ["Azure (standalone)"]
    aca["Container App control plane, uaenorth"]
    snp["SEV-SNP enclaves uaenorth + australiaeast; api-azure.trustedrouter.com"]
    pg[("Postgres Flexible Server")]
    azch[("ClickHouse")]
  end

  user --> dns
  dns --> glb
  dns --> cs
  dns --> ga
  dns --> aca
  dns --> snp
  armor --- glb
  glb --> cr1
  glb --> cr2
  glb --> cr3
  glb --> cr4
  cs -->|"/internal/gateway authorize + settle"| glb
  cr1 --> spanner
  cr2 --> spanner
  cr3 --> spanner
  cr4 --> spanner
  cr1 -->|"outbox to ingester"| ch
  fp --> spanner
  ga --> ecs
  ecs --> dsql
  ecs --> awsch
  nitro --> ecs
  aca --> pg
  aca --> azch
  snp --> aca

Facts behind the diagram:

2. Push-to-main to production

flowchart TD
  push["git push / merge to main"]
  push --> ci["ci.yml: lint, mypy, spanner-emulator, frontend, 12+8 test shards, coverage 70 pct, proof-oracle, browser, fastpath Go, TLA+ proofs"]
  push --> filt{"deploy.yml path filter: src, scripts/deploy, scripts/entrypoint.sh, frontend/src, Dockerfile, .gcloudignore, pyproject, uv.lock, deploy.yml"}
  filt -->|"no match: docs-only"| nodeploy["No deploy"]
  filt -->|"match"| gate["gate-on-ci: poll ci.yml for SAME SHA until success (hotfix skips)"]
  push --> build["build-image: Cloud Build to Artifact Registry, tag = SHA"]
  ci -->|"success on main"| fpr["fastpath-release.yml: build Go binaries, upload to GCS"]
  gate --> admit["admit-cloud: acquire GCS deploy mutex (two-cloud journal)"]
  build --> admit
  admit --> mig["migrate-schema: idempotent Spanner DDL + analytics snapshot sync"]
  admit --> sec["sync-runtime-secrets"]
  mig --> cur["confirm-current-main: skip if a newer main exists"]
  sec --> cur
  cur --> warm["deploy: rollout.sh --no-traffic in all 4 regions in parallel (primary alone reconciles LB)"]
  warm --> hold{"TR_DEPLOY_HOLD_REGIONS?"}
  hold -->|"held"| skipr["traffic untouched in held region"]
  hold -->|"not held"| ramp["staged_traffic.sh us-central1: 10 to 50 to 100 pct with watchdog"]
  ramp --> canary["3-min canary watch us-central1"]
  canary -->|"trip"| rb["auto rollback us-central1 + abort"]
  canary --> bill["billing-path 5xx gate (assert_no_billing_5xx.sh)"]
  bill -->|"5xx"| rb
  bill --> sec2["rollout-secondaries: ramp europe-west4, us-east4, southamerica-east1 serially under the same mutex fence"]
  sec2 --> jobs["deploy synthetic monitor job, opt-in trust jobs, Data Manager check"]
  jobs --> smoke["smoke test prod incl. /trust image_digest + source_commit"]
  smoke --> pub["public-surface-companion: deploy + smoke T1 public service"]
  smoke --> vcc["verify-cloud-complete: verify_gcp_complete.sh"]
  pub --> fin["finalize-cloud: release mutex"]
  vcc --> fin
  fin --> post["workflow_run fan-out: prod-smoke, trust-drift, reconcile-cloud-releases (AWS/Azure catch-up)"]

flowchart TD
  push["git push / merge to main"]
  push --> ci["ci.yml: lint, mypy, spanner-emulator, frontend, 12+8 test shards, coverage 70 pct, proof-oracle, browser, fastpath Go, TLA+ proofs"]
  push --> filt{"deploy.yml path filter: src, scripts/deploy, scripts/entrypoint.sh, frontend/src, Dockerfile, .gcloudignore, pyproject, uv.lock, deploy.yml"}
  filt -->|"no match: docs-only"| nodeploy["No deploy"]
  filt -->|"match"| gate["gate-on-ci: poll ci.yml for SAME SHA until success (hotfix skips)"]
  push --> build["build-image: Cloud Build to Artifact Registry, tag = SHA"]
  ci -->|"success on main"| fpr["fastpath-release.yml: build Go binaries, upload to GCS"]
  gate --> admit["admit-cloud: acquire GCS deploy mutex (two-cloud journal)"]
  build --> admit
  admit --> mig["migrate-schema: idempotent Spanner DDL + analytics snapshot sync"]
  admit --> sec["sync-runtime-secrets"]
  mig --> cur["confirm-current-main: skip if a newer main exists"]
  sec --> cur
  cur --> warm["deploy: rollout.sh --no-traffic in all 4 regions in parallel (primary alone reconciles LB)"]
  warm --> hold{"TR_DEPLOY_HOLD_REGIONS?"}
  hold -->|"held"| skipr["traffic untouched in held region"]
  hold -->|"not held"| ramp["staged_traffic.sh us-central1: 10 to 50 to 100 pct with watchdog"]
  ramp --> canary["3-min canary watch us-central1"]
  canary -->|"trip"| rb["auto rollback us-central1 + abort"]
  canary --> bill["billing-path 5xx gate (assert_no_billing_5xx.sh)"]
  bill -->|"5xx"| rb
  bill --> sec2["rollout-secondaries: ramp europe-west4, us-east4, southamerica-east1 serially under the same mutex fence"]
  sec2 --> jobs["deploy synthetic monitor job, opt-in trust jobs, Data Manager check"]
  jobs --> smoke["smoke test prod incl. /trust image_digest + source_commit"]
  smoke --> pub["public-surface-companion: deploy + smoke T1 public service"]
  smoke --> vcc["verify-cloud-complete: verify_gcp_complete.sh"]
  pub --> fin["finalize-cloud: release mutex"]
  vcc --> fin
  fin --> post["workflow_run fan-out: prod-smoke, trust-drift, reconcile-cloud-releases (AWS/Azure catch-up)"]

Step-by-step citations (all .github/workflows/deploy.yml unless noted):

1. Trigger and path filter: :26-37. Manual workflow_dispatch with hotfix: true skips the CI gate and the watchdog baseline (:38-43, :73, :516-524).
2. Concurrency: group deploy-trusted-router, cancel-in-progress: false. One deploy runs at a time, with one pending slot that a newer push can replace. The GCS mutex is what actually serializes traffic changes (:45-49).
3. gate-on-ci polls ci.yml's conclusion for the same SHA (:57-107). ci.yml and deploy.yml start in parallel on push, so without this gate a red commit could roll out everywhere (:17-21).
4. build-image: Cloud Build pushes ${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPO}/${SERVICE}:${SHA} and uses a registry buildcache tag for layer caching (:109-208). Auth uses WIF to tr-deploy@quill-cloud-proxy, with no JSON keys (:9-12, :133).
5. admit-cloud takes the production mutex with deploy_mutex.sh acquire (:209-240). The wait is long because a quill-cloud-proxy gateway rollout can hold the lease for about 2h20m (:213-216).
6. migrate-schema runs idempotent migrate_*.sh steps (money primitives, retention, receipts, outboxes, async-settle admission, fastpath tables) and syncs analytics snapshots (:242-368).
7. sync-runtime-secrets validates SES credentials and pushes the ClickHouse reader credential (:370-407).
8. confirm-current-main stops a stale push from deploying over a newer main (:409-448).
9. deploy: - Re-asserts the mutex (:500-501) and captures pre-deploy revisions for rollback (:526). - Warms all 4 regions with no traffic in parallel (:544-600). Only the primary process may mutate the global LB (TR_DEPLOY_RECONCILE_LB, :594-597). - Resolves the primary baseline and honours holds (:657-712). - Staged 10/50/100 ramp in us-central1 (:713-723), then a 3-min canary (:734-742) with automatic rollback (:744-764). - Billing-path 5xx gate (:766-788). - Never-ramped regions have their staged-probe tags swept on failure, because a tagged revision keeps its minimum instances, which costs money (:882-901).
10. rollout-secondaries:
    • Imports the same mutex fence (:918-937) and ramps the secondaries serially (:958-970). The order is europe-west4, us-east4, southamerica-east1 (scripts/deploy/ramp_secondaries.sh:237).
    • Then deploys the synthetic monitor Cloud Run Job (:972), the opt-in trust jobs (:994) and the Google Data Manager guard (:1005).
    • Smoke-tests prod (:1026-1183), including asserting that source_commit, image_reference and image_digest appear in the trust records (:1160-1170).
11. public-surface-companion deploys the isolated T1 public service and runs outside the mutex (:1262-1338).
12. verify-cloud-complete runs bash scripts/deploy/verify_gcp_complete.sh with if: always() (:1340-1399). The long comment at :1340-1380 records why an echo-only stub is unacceptable.
13. finalize-cloud releases the mutex and records success or failure (:1401-1428).

The runbook's definition of done: "deployed means the primary is live" (all four regions warm, plus us-central1 ramp and canary, about 9 min). Full convergence is the later rollout-secondaries + verify-cloud-complete (docs/runbook.md:395-410). AGENTS.md:56-58 adds that a queued or partially completed rollout must never be called deployed.

3. GitHub Actions workflows

4. ENV_VARS inventory (config-as-code)

ENV_VARS is defined at scripts/deploy/rollout.sh:401-676. It is joined with | (:677) and applied with --set-env-vars plus TR_SERVING_REGION per region (:851). Setting a Cloud Run env var by hand does nothing, because the next rollout overwrites it (AGENTS.md:64-66). Secrets are separate: they are listed in SECRET_ENVS and pulled from Secret Manager (:101-112). Names only below. No values are listed.

Secrets (SECRET_ENVS, names only) at scripts/deploy/rollout.sh:101-112: TR_SENTRY_DSN, TR_STRIPE_SECRET_KEY, TR_STRIPE_WEBHOOK_SECRET, TR_INTERNAL_GATEWAY_TOKEN, plus the async-settle Ed25519 key mounted as a file at /secrets/async-settle/ticket.pem. The deploy fails if it is missing. Optional secrets are appended conditionally (:113-135).

Sizing knobs come from _lib.sh rather than ENV_VARS: TR_CONTROL_PLANE_REGIONS, TR_WARM_REGIONS, TR_CLOUD_RUN_MIN_INSTANCES_BY_REGION, TR_CLOUD_RUN_CONCURRENCY, TR_CLOUD_RUN_MAX_INSTANCES, TR_CLOUD_RUN_MEMORY, TR_CLOUD_RUN_INGRESS (scripts/deploy/_lib.sh:30-93).

Deploy-control vars (read by scripts, not by the app): TR_DEPLOY_TARGET_REGIONS, TR_DEPLOY_NO_TRAFFIC, TR_DEPLOY_RECONCILE_LB (deploy.yml:555-561); TR_DEPLOY_HOLD_REGIONS (repo variable, deploy.yml:482, :916); TR_DEPLOY_MUTEX_* (scripts/deploy/deploy_mutex.sh); TR_CLOUD_BAKE_HOURS, TR_CLOUD_DEPLOY_MODE, TR_CLOUD_BAKE_OVERRIDE (docs/runbook.md:425-470); TR_STAGED_WATCHDOG_SKIP_BASELINE, TR_BILLING_5XX_GATE_SKIP (hotfix only, deploy.yml:722, :770).

5. Rollout control-plane mechanics

• Fail-closed entry. rollout.sh refuses to run unless TR_ALLOW_DEPLOYED_COMBINED_SURFACE=true is set, which is a temporary bridge until the six-service cutover (#712) (scripts/deploy/rollout.sh:7-16). It also refuses unless the exact 40-char source revision matches the checkout and the release (:21-31).
• Staged traffic. staged_traffic.sh ramps 10, 50 and 100% with a synthetic watch between steps. If a watch trips, traffic goes back to 100% on the old revision (scripts/deploy/staged_traffic.sh:2-12). The watchdog is scripts/deploy/watchdog.py (per-region post-deploy). The default SLO class is router_core (staged_traffic.sh:18).
• Billing gate. assert_no_billing_5xx.sh fails the rollout if the candidate revision logged any billing-path 5xx, because Cloud Run readiness cannot see exhausted retries (scripts/deploy/assert_no_billing_5xx.sh:2-3).
• Deploy mutex / two-cloud coordinator.
• deploy_mutex.sh wraps cloud_rollout.py, a generation-CAS journal at gs://tr-deploy-mutex-quill-cloud-proxy/locks/trusted-router-production.json covering gcp/aws/azure (scripts/deploy/cloud_rollout.py:1-41).
• It is fleet-wide and lets one cloud roll out at a time. The enclave gateway rollouts in quill-cloud-proxy share it (docs/runbook.md:349-357).
• Inspect it with deploy_mutex.sh status. Manual removal is break-glass only, with --if-generation-match (docs/runbook.md:368-385).
• Conflict to resolve: the runbook says "locks expire after 90 minutes" (docs/runbook.md:386-387), but cloud_rollout.py:5 says "Expired/failed reservations are NEVER reclaimed just because time elapsed."
• Regional hold. TR_DEPLOY_HOLD_REGIONS (comma list or all) is a GitHub repo variable. Held regions still get a no-traffic warm, but their traffic is not touched. It is read at job start, so changing it does not affect a running job (scripts/deploy/_deploy_hold.sh; docs/runbook.md:398-421).
• Bake ladder / lifeboat (applies to AWS and Azure):
• When one cloud gets fresh code, another must keep serving a commit at least 24h old, the "lifeboat" (docs/runbook.md:425-447).
• The candidate may reach a second cloud only after 24h on first-parent main. Both paths require status.json overall_status=up.
• GCP auto-deploys and can never be the lifeboat.
• Canary mode: TR_CLOUD_DEPLOY_MODE=canary. The header of scripts/deploy/cloud_bake_gate.sh notes that the default rolling mode now uses the atomic two-cloud guard, and the ladder applies in promote/canary mode.
• Override with a recorded reason using the bake_override_reason input (deploy-aws-control-plane.yml:20-24).
• Completeness gate. "Deployed" and "complete" are different states. verify_cloud_complete.sh {aws|azure|gcp} is the executable definition of done (scripts/deploy/verify_cloud_complete.sh:1-30). It exists because AWS-EU served for 15 days with no analytics drain.
• Break-glass. scripts/deploy/break_glass_deploy.sh [--apply] ships from a laptop when GitHub Actions is down. It runs rollout.sh and a smoke check. It has no CI gate, no migrations, no staged canary or auto-rollback, and no secret sync or jobs, all listed in its header (:1-37, :111-136). See docs/runbook-github-outage.md.
• Image and attestation digest.
• The control-plane image is tagged with the commit SHA (deploy.yml:123-127).
• rollout.sh reads the enclave's gcp-release.json (local TRUST_FILE or URL), extracts source_commit, image_reference and image_digest (scripts/deploy/rollout.sh:85-98), and injects them as TR_TRUST_GCP_* (:482-486).
• AWS and Azure measurements are no longer pinned here. Each plane publishes its own record, and the control plane mirrors it by URL (:487-495).
• Note: the TRUST_FILE default is an operator's laptop path (scripts/deploy/_lib.sh:126). In CI the URL path is used.
• trust-drift.yml checks published values against live attestations daily and after each deploy.
• Dockerfile. uv:python3.12-bookworm-slim. Dependency layers sit before the source copy so the build cache works. Bytecode is precompiled because cold starts measured p50 34.6s / p95 217s without it (Dockerfile:19-29). Entrypoint scripts/entrypoint.sh runs python -m trusted_router.serve, whose listener stays open briefly after SIGTERM to avoid zero-latency 500s (Dockerfile:31-40).

6. Terraform (infra/)

There is one root, with state in gs://tr-infra-tfstate-quill-cloud-proxy/infra (infra/backend.tf). It owns only static plumbing. Release procedures stay in scripts/deploy/ (infra/README.md:1-36).

The key hazard: every infra apply re-asserts the whole root. On 2026-09-21 a one-line WIF change also reverted a hand edit and re-enabled a DSQL read path costing about $500/day for 27 minutes (infra/README.md:38-63).

7. fastpath/ and release/

• fastpath/ is the Go service for the fast-admission / batched-settlement design. Each internal/* package is a "shadow" of a TLA+ spec in proofs/, and tests check it against the TLC state graph. It runs over Spanner and Pub/Sub. Its schema is applied by scripts/deploy/migrate_fastpath.sh (fastpath/README.md:1-50).
• Production state is "P0 dark install": 2 nodes, empty allow-list, nothing admits until a workspace is switched on (infra/fastpath.tf:1-12).
• Release pipeline: CI green, then fastpath-release.yml, then GCS. Terraform pins a commit (infra/fastpath.tf:19), and a node picks it up when reset or replaced.
• Runbooks: docs/runbooks/fastpath-*.md.
• release/ holds first-window-plan.json and its schema. It is a reviewed plan for the async-settle shadow-v2 "first window": router and enclave SHAs/image digests, workspace, attachment bucket, per-region caps, surge and publishers. The checked-in copy is a template with REVIEW placeholders and zero SHAs. Its consumers are docs/design/async-settle-shadow-v2-activation.md:44, :99 and docs/design/async-settle-shadow-v2.md:3697.

8. Observability

• Sentry: src/trusted_router/sentry_config.py.
• send_default_pii=False (:240). before_send, before_send_log and before_breadcrumb all run _scrub (:301-331).
• Key-name denylist SENSITIVE_KEYS (:21), substring and prefix scrubbing (:43-95), and bytes redacted wholesale (:586).
• A floodgate rate-limits event floods (:453-500). A noise filter drops untrusted 405s (:334-370).
• The DSN is a secret (TR_SENTRY_DSN).
• GCP alert policies (YAML, applied by scripts):
• scripts/deploy/spanner-alerts/: api-failures, backup-workflow, capacity, contention, high-priority-cpu (45%, high-priority-cpu.yaml:11-19), latency.
• scripts/deploy/gateway-alerts/: billing-5xx ("any internal gateway billing 5xx"), billing-pressure, client-invisible-outage, client-telemetry-stale, control-plane-memory.
• scripts/deploy/clickhouse-alerts/: disk-capacity, node-availability.
• Fastpath alerts are in Terraform (infra/fastpath_alerts.tf).
• Paging policy: router-core 5m/1h burn rate ≥14.4× pages immediately; 6h ≥6× pages in waking hours; 24h ≥3× creates a review item (docs/runbook.md:105-110).
• Synthetic monitoring:
• GCP: Cloud Run Jobs deployed by scripts/deploy/synthetic.sh (uptime probes kept separate from the throughput benchmark, :1-5), redeployed every rollout (deploy.yml:972).
• AWS: EventBridge 1-min rule to App Runner tr-eu /internal/synthetic/run (HANDOFF.md:110-117; infra/aws_synthetic_monitoring.tf).
• Status: trustedrouter.com/status.json (per-cloud status, used by bake gates and freshness checks). Public page status.trustedrouter.com is linked from src/trusted_router/dashboard.py:4620 and allowed in CSP at middleware.py:458. The repo does not say who hosts the status page. Trust records are published at trust.trustedrouter.com.
• Logs: Axiom (TR_AXIOM_*), plus Cloud Logging with sampled exclusions (infra/logging_sampling.tf).
• Scheduled health checks: prod-smoke every 30 min, trust-drift, analytics/archive/telemetry freshness, cloud-staleness, typed-audit, provider-catalog-health (section 3).

9. Runbook and incident index

docs/runbook.md (symptom → action; 1638 lines)

Other runbooks

docs/incidents/

10. Deploy safety rules

1. Run all three CI gates locally: uv run ruff check . && uv run mypy && uv run pytest -q. Never run ruff format .. Coverage must stay ≥70% (AGENTS.md:27-40). Run the full suite, because tests pollute each other through the STORE proxy (:42-45).
2. Rebase before trusting a failure. The price bot advances main hourly (AGENTS.md:47-49).
3. Pushing to main deploys if a path in the filter changed (AGENTS.md:60-62; deploy.yml:26-37). Docs-only changes do not deploy.
4. Change config only in ENV_VARS. Hand edits to Cloud Run are overwritten (AGENTS.md:64-66).
5. Never call a queued or partial rollout "deployed". Verify behaviour and prod health after deploy, and report the exact blocker if a gate stops it (AGENTS.md:51-58).
6. Keep the gates intact: CI-green gate, mutex, staged ramp, canary, billing 5xx gate, completeness check. hotfix=true is for when prod is already sick (deploy.yml:38-43).
7. One cloud at a time. Respect the GCS mutex. Delete it only after status shows a dead owner, and use --if-generation-match (docs/runbook.md:349-387).
8. Use TR_DEPLOY_HOLD_REGIONS to freeze a region during an incident. It is read at job start (docs/runbook.md:398-421).
9. Bake ladder: fresh code must not serve on every cloud at once. Keep a ≥24h lifeboat, and record any override reason (docs/runbook.md:425-447).
10. Never run large unbatched DML on prod Spanner during a rolling deploy (AGENTS.md:68).
11. Read-only is not load-safe. No LIKE, regex or JSON scans of tr_entities.body. Use python -m scripts.inspect_payment_owner, SPANNER_SYS stats and ClickHouse. Never loosen the 45% CPU alarm (AGENTS.md:70-79).
12. Money code (reserve/settle/refund) needs differential or conformance tests. Run proof_oracle when touching it (AGENTS.md:38-40, :81-87).
13. Terraform has a single owner. An infra merge re-applies the whole root. Mirror any emergency hand edit in a PR within the hour, and check the apply log for unexpected resources (infra/README.md:38-63).
14. Cloud Armor rule changes need an owner's credentials before merge, because tr-deploy cannot weaken the edge (infra/cloud_armor.tf:9-14).
15. Do not enable Cloudflare proxying on the prompt path. It voids attestation (docs/runbook.md:118-119; HANDOFF.md:100).
16. Break-glass skips everything except rollout and smoke. Write down the prior revision and run migrations by hand (scripts/deploy/break_glass_deploy.sh:13-30).
17. A PCR0 re-pin on AWS touches three surfaces: both Fargate task definitions and App Runner tr-eu (HANDOFF.md:119-123).
18. "Finished" is not "complete". Bring-up scripts must end in verify_cloud_complete.sh (scripts/deploy/verify_cloud_complete.sh:8-25).