Production promotion readiness — audit 2026-09-17

What production needs before staging merges to master and the wipe/rebuild runs. Every line was verified against the live AWS account, the Terraform state, or the code — not inferred from a doc. Where a doc or memory disagreed with the measurement, the measurement is recorded and the stale source is named.

Method: four independent passes, so a gap has to hide in all four to survive — (1) a complete terraform plan, (2) a staging-vs-production Terraform state diff, (3) a config-key diff of the API between production's commit and staging, (4) the open items already written down in production-launch-readiness.md.

0. A bare terraform plan under-reports by half

terraform plan                              →  8 to add, 1 to change, 0 to destroy   ❌ WRONG
./infra/scripts/production-apply.sh plan    → 12 to add, 9 to change, 2 to destroy   ✅

The Cloudflare provider has no token in a bare shell, which aborts part of the graph walk — but Terraform still prints a plan summary, so the truncated result looks complete. Always use the wrapper, which sources CLOUDFLARE_TF_API_TOKEN from Secrets Manager.

This also means [[prod-terraform-has-unapplied-drift]] is stale: it records "8 SES resources + an RDS param change", which is precisely the truncated plan. The real drift is larger.

1. Terraform apply — required, and it must run BEFORE the deploy

Change	Detail
+8	SES bounce/complaint: 2 SNS topics, 2 policies, 2 HTTPS subscriptions, 2 SESv2 event destinations
+2	Crons appointment-reminder-sweep (hourly :45) and notification-deliverability-sweep (hourly :15)
+2 / −2	api and api-cron task definitions replaced
~8	Existing schedules updated in place
~1	rds.force_ssl pending-reboot → immediate; value stays 1, TLS enforcement unaffected

The two destroys are only the task-definition replacements. No data-bearing resource is destroyed. The task defs replace because NOTIFY_SES_FEEDBACK_TOPIC_ARNS references SNS ARNs that do not exist yet, so the container definition is computed — benign, and no new secrets are required by the apply itself.

Why the order is load-bearing

.github/actions/deploy-ecs-service runs describe-task-definition --task-definition restartix-production-api — the family name, which returns the latest ACTIVE revision — then clones it and swaps the image.

• Terraform first → registers the revision carrying the SES var → the deploy clones it → the service runs it. Correct.
• Deploy first → clones the current revision → the SES feedback variable never reaches the container, and Terraform's revision sits unused until the next deploy.

Merge before deploy, for a second reason

origin/master's workflow still pushes to the shared repos (restartix-api:latest); production's task definitions and Terraform state reference restartix-production-api:latest. A deploy dispatched from mastertoday would push to the wrong repository. The rename is on staging, so merging first fixes it. Do not "test deploy" from master beforehand.

2. Missing from production crons.tf — three

Staging defines 14 cron tasks; production defines 12, and the three below are absent. All four binaries are present in the image (services/api/cmd/).

• appointment-noshow-sweep — excluded on purpose, with a comment at crons.tf:204 saying "it reads appointments (migration 000046) and this database is at 000039 … Add it in the change that promotes F5's schema." This wipe is that change — post-wipe production is at 000052 and appointments exists. The stated precondition is now met.
• video-room-sweep
• video-usage-reconcile — the launch-readiness doc additionally asks for a success-pattern alarm on this one.

3. Video consultations (F5.5) have no production infrastructure

infra/envs/production/compute.tf contains no VIDEO_ROOM_SECRET, no DAILY_API_KEY, no DAILY_WEBHOOK_SECRET, no DAILY_GEO. Staging wires all four. There is no restartix/production/daily-bootstrap secret, and no video_room_secret resource in production state (staging has both).

This does not block boot. VideoRoomSecret is guarded by != "" so the lookup route is simply not mounted, and providers.Bootstrap skips a provider whose key is blank — exactly what staging did. The consequence is that video consultations are dead on arrival until the secrets are wired.

Already written down at production-launch-readiness.md line 87, which also names the one-shot cmd/video-webhook-register run required per environment after the API boots with the secret populated.

4. New API config since production's commit

Sixteen new keys. Only DATABASE_URL and REDIS_URL are required:"true" platform-wide and both exist, so the API boots. The rest default or degrade:

Key	State on production	Effect if unset
VIDEO_ROOM_SECRET	absent	video lookup route not mounted
DAILY_WEBHOOK_SECRET	absent	video provider not seeded
DAILY_GEO	absent	defaults to eu-central-1
BOOKING_CLIENT_SECRET	Terraform-generated restartix/{env}/booking-client-secret, wired 2026-09-24 — live only once applied	slot holds and public bookings on /programare answer 502 booking_unavailable
NOTIFY_SES_FEEDBACK_TOPIC_ARNS	supplied by the apply	—
RATELIMIT_* ×9, HOLD_STREAM_TIMEOUT	defaulted	—

No new buckets are required, and no new frontend build args.

5. One-shot steps after the deploy

• cmd/video-webhook-register — only once the API has booted with the Daily secret populated, so the provider row exists to verify against.
• Roll partitions immediately after migrate. production-reset-db.sh already runs both partition rolls at -ahead=3. This matters most for video_session_events: a partitioned table applied mid-month has no partition until the next cron, and those rows arrive by webhook — so the failure is silent, with consultations running normally and leaving no record.

6. Post-wipe re-seed — nothing below is recreated by a migration

• Superadmin. DROP SCHEMA takes platform_memberships; sign in to Console once, then grant under SET ROLE restartix.
• Entitlements. A new org defaults every regulated flag to FALSE; use Console → Plan → Entitlements or 000003-grandfather-org-entitlements.sql.
• 48 exercise_renders + 48 content_files. 000031_exercises_seed seeds exercises and exercise_tags and neither of these. Media bytes survive in S3/Bunny; only the pointers die, and re-baking is blocked because restartix-production-media is at desired 0 and CPU target-tracking cannot scale from zero. Re-insert from the backup, remapping exercise_id by slug — both table shapes are unchanged between chains.
• Redis is not flushed. cache.Platform("resolve","slug",…) is not UUID-derived, so re-creating the org under its previous slug serves a dead id for up to the 5-minute TTL.

7. Verified clean — no action

• btree_gist — was missing from production-reset-db.sh and would have died mid-000045, leaving schema_migrations dirty. Fixed 2026-09-17.
• ECR repositories — already managed resources in production state (7 repos
  • 7 lifecycle policies); the data-source→resource conversion is already applied, so the apply will not try to re-create them.
• Observability — production carries 12 alarms to staging's 3, plus the whole module.backups stack. Production is ahead here, not behind.
• Backups — snapshot prewipe-20260917-1106, both logical dumps, and a passing restore drill. See production-prewipe-backup.md.

8. Known-open — decide explicitly rather than discover later

These are pre-existing and not caused by the wipe, but the wipe is when the platform is re-presented to patients.

• No WAF or rate-limiting ruleset in Terraform, for any zone. Whatever L7 protection exists is dashboard-configured: unversioned and invisible to the repo. App-layer rate limiting is IP-keyed and is not DDoS defence.
• iOS video playback returns CoreMedia 403 — diagnosed 2026-10-02, the 403 was wrong. iOS loads media only while a user tap is fresh; priming ran after the Start server action + redirect, so the phone fetched the HLS playlists and no segments and priming timed out. Not a 403: Bunny token auth is off, no-referrer requests are allowed, and the Bunny log showed every iPhone request answered 200. Fixed in the Portal conductor (native-HLS grace enables Start; load() inside the tap; priming timeout now emits video_error with src_kind=priming). Reaches patients on a portal production deploy.
• No patient receives clinical email. Zero notify. calls in the whole appointments domain — no confirmation, reminder, or cancellation notice.
• Sentry is wired in apps/portal only.
• restartix-production-media is at desired 0 — no bakes possible.