Production apply / rebuild runbook

The authoritative, ordered procedure to stand up the production AWS substrate from the IaC in infra/envs/production — whether that is the first apply, a rebuild from nothing, or a disaster recovery. Production was opened early for the June-10 single-session demo (june-demo.md → Stream A) and has stayed live and serving real patients since 2026-06-05.

This document is maintained continuously until launch. Anything that changes the substrate — a new secret, a new cron, a new service, a changed ordering constraint — lands here in the same change that introduces it. The value of a rebuild runbook is entirely in its completeness; a step that exists only in someone's memory is a step that is already lost. When you discover a constraint by having something fail, record the failure alongside the fix, because the failure is what makes the ordering believable to the next reader.

Verify before you trust. Several claims below were wrong when checked against the live environment — service names, desired counts, and an ordering that only worked by accident. Where a line says something was verified and when, that is why.

Change log

Newest first. Add an entry whenever the procedure itself changes.

Date	Change
2026-08-26 (2)	000051 joins the range, and it rewrites RLS POLICIES. Six SECURITY DEFINER set-helpers replace the three patient-facing policies on exercises + session_exercises; the clinic exercise library goes 996ms → 5.9ms. It is additive and reversible (the down restores all six verbatim, round-trip verified), but it is the only entry in this range that changes who-can-see-what mechanics, so it wants the equivalence matrix run on staging against real rows before production. Nothing applied anywhere but local.
2026-08-26	Step 0 added: jit=off for restartix_app, and it is independent of this promotion. Postgres was JIT-compiling RLS-expanded plans — 1.1s and 13.8s of code generation for queries doing 8ms and 16ms of work, recompiled on every execution because generated code is never cached. Production is 2 vCPU, so this bounded total throughput to well under one program-page load per second. Applied to LOCAL only so far. The trap is that it needs a pgbouncer restart to take effect; without one, warm backends keep jit=on for up to server_lifetime.
2026-08-17 (2)	The range FOLDED back to 000040–000050. 000051–000055 were applied to no environment, so the consents-model correction and F3.5.3 form sessions were folded into 000043/000044 where their tables are defined, and 000054's starter-template seed was deleted outright. Production is unaffected — it has never applied any of 000040+, so it now runs a shorter range that builds the same schema. STAGING MUST BE REBUILT: it is at 000049 and two of the migrations it already applied have changed. The fold also caught two regressions 000055 had introduced by re-creating policies from stale copies — forms_select and form_template_versions_select had each silently lost a deleted_at IS NULL guard.
2026-08-17	One of the migrations DELETES consent rows. The org_service_disclaimer retirement deletes the purpose, its published versions and every consents row accepting it — deliberate while those rows are demo content, and irreversible once they are not. Added the ⛔ block in step 5 with the count-first query. (It lived in 000053 when this was written; it is in 000044 after the fold.)
2026-08-15	The promotion now owns FIVE catch-up scripts, and one of them straddles the deploy. Added ⛔ blocks in step 5 for 000023+000034 (history follows the person — additive, with a backfill that is not instant against production's rows, and a silent failure mode) and for 000008 (the consent subject). The 000008 one is deliberately split: Part A creates the new overload before the deploy, Part B drops the old one after it, with the removal repeated at the end of step 6. Combining them, which the first draft did, would have errored every consent gate from the DDL until a reviewer approved — the 2026-08-11 treatment_plans_enabled shape. Nothing was applied to production; this entry is the record, not the run.
2026-08-14	Caregiver grant codes need NO schema change in either environment — recorded under step 1 rather than step 5. They live in Redis, which is why; the operationally relevant fact is that maxmemory-policy = noeviction already covers them, and would not if it were ever loosened.
2026-08-14	Both catch-ups applied to STAGING; production deferred deliberately. Recorded in the two ⛔ blocks in step 5 — nothing in production needs either until the 000040–000050 range promotes, so they belong to that run rather than to a standalone DDL session against live patient data.
2026-08-13	000012 was edited in place too — organization_invites.patient_profile_id, for the P7 profile claim. Unlike the 000006 one this fails loudly if forgotten (the invite INSERT names a missing column).
2026-08-13	000006 was edited in place and production needs the catch-up. Added the ⛔ block before the has-rows warning: patient_caregivers_select_org_staff (the P7 family-account RLS policy) exists only in the corrected migration, which production will never re-run. Forgetting it fails silently — the Family panel renders empty rather than erroring.
2026-08-12	Video's absence is now written down rather than merely true. Step 2 records that the secrets script's final Daily section fails against production (no daily-bootstrap container — verified live) and why the substrate omits video entirely; step 8 records that the provider webhook is not a production step yet, and what it costs to forget once it is.
2026-08-11	The ordering rule gained its second direction — schema additions may precede a deploy, removals must follow it. Added after dropping a column ahead of the reviewer-gated deploy 500'd every authenticated request. Step 5 also now points at the catch-up scripts a restore may need.
2026-08-11	Step 4 (seed ECR) added, and it is load-bearing. Production now owns its own ECR repos, which a from-scratch apply creates EMPTY — so migrations, crons and services all had nothing to pull. Later steps renumbered. Also added the no-running-tasks alarms, the sibling cron task definition, and fixed two errors that would have failed on paste (media-service → media, clinic's desired count).

Operator = ops custodian (Ovidiu). Claude authored the IaC + scripts + deploy-production.yml; everything below is run by the custodian (or by Claude under direction). terraform apply, the Clerk prod app, secret population, and DNS registration are all operator steps — they are not done during authoring.

Golden rule: never terraform apply without reviewing the plan first. Production tfstate is shared; a second human reviews the plan.

Account 538636660353, eu-central-1. Platform hostnames are bare restartix.pro subdomains; the demo vanity host is platforma.restartix.ro on the separate restartix.ro Cloudflare zone.

---

0. Prerequisites (before any apply)

1. Bootstrap applied. The infra/envs/bootstrap composition must be applied — it owns the restartix-tfstate bucket and the restartix-deploy-production OIDC role (trusts the GitHub production Environment). Verify:

   aws iam get-role --role-name restartix-deploy-production --query 'Role.Arn' --output text

   If absent: cd infra/envs/bootstrap && terraform apply (the production entry already exists in its deploy_iam block).

2. GitHub production Environment exists with required reviewers. Repo → Settings → Environments → production → add required reviewers. This is the approval gate for deploy-production.yml; the OIDC role trusts environment:production, so without the Environment the deploy can't assume the role.

3. Clerk production application stood up. Separate Clerk app (own pk_live_ / sk_live_). Do NOT reuse staging keys with production patients. Configure its allowed origins / redirect URLs for portal.restartix.pro, console.restartix.pro, and platforma.restartix.ro.

4. Cloudflare Terraform token ready. Create a restartix-production-terraform API token with Zone:DNS:Edit + Zone:SSL:Edit on both restartix.pro and the restartix.tv vanity zone (the TV companion's own zone — the apex restartix.tv serves the prod TV app). Held by the custodian; populated into SM in step 3 so production-apply.sh works on later runs. Have it in hand for the first apply. (Staging's TF token needs the same restartix.tv scopes — plus Zone:Zone Settings:Edit — since staging owns that zone's settings + ruleset.)

5. restartix.tv zone exists on Cloudflare + zone id set. Register restartix.tv and add it as a Cloudflare zone (point the registrar NS at Cloudflare). Put its Zone ID in cloudflare_tv_zone_id in bothinfra/envs/production/terraform.tfvars and infra/envs/staging/terraform.tfvars (placeholders are committed). The prod apex restartix.tv + staging staging.restartix.tv both live here.

6. Legacy handoff secret value ready — the exact LEGACY_HANDOFF_SECRET set on the legacy restartix-admin-api (Railway). The demo entry flow (Stream B) 503s until production's copy matches it byte-for-byte.

7. Tooling: terraform >= 1.10, aws CLI, session-manager-plugin, psql (brew install libpq), jq, docker.

8. No conflicting DNS. Confirm api/console/clinic/portal/telemetry.restartix.pro and the *.clinic / *.portal wildcards are free on the Cloudflare restartix.pro zone (staging uses the -staging suffix, so they should be). On the restartix.tv zone, the apex (@) and staging records must be free. A pre-existing record makes the apply error on that record.

---

1. Terraform apply

The Cloudflare provider needs CLOUDFLARE_API_TOKEN at plan time (the cloudflare_ip_ranges data source + DNS records). On the first apply the SM secret isn't populated yet, so export it manually:

cd infra/envs/production
terraform init                       # initialises the S3 backend (production/terraform.tfstate)
export CLOUDFLARE_API_TOKEN=<restartix-production-terraform token>
terraform plan   -out=prod.plan      # REVIEW with a second human
terraform apply  prod.plan
unset CLOUDFLARE_API_TOKEN

Subsequent runs use the wrapper (sources the token from SM — never in shell history):

./infra/scripts/production-apply.sh plan
./infra/scripts/production-apply.sh apply

This creates: VPC + two NAT Gateways (HA, one per AZ — the interface VPC endpoints were removed 2026-07-31) + SSM bastion + SGs (ALB locked to Cloudflare IPs), platform CMK, RDS Multi-AZ, Redis Multi-AZ, all restartix/production/* SM containers (empty, zero versions), the backup bucket + backup CMK, the restartix-production-<svc> ECR repos (also empty — see step 4), ACM cert (+ Cloudflare validation records), ALB, ECS cluster + 7 services (media at desired_count = 0; clinic at 1), the api + telemetry migrations runners, the sibling small cron task definition, the crons + scheduler DLQ, edge DNS, the TV bucket, SES config set, alarms, and the budget.

Nothing can start yet, and that is expected. Two things Terraform creates are empty by design: the secret containers (step 2) and the ECR repos (step 4). ECS secret injection is fail-closed, so a task naming a versionless secret dies at ResourceInitializationError; a task naming an absent image dies at CannotPullContainerError. Expect both until those steps are done, and do not debug them as faults.

Aurora/RDS first-create takes ~10–15 min (Multi-AZ). The ACM cert validates via the Cloudflare records — allow a few minutes.

Redis runs maxmemory-policy = noeviction, and more now depends on that than when the choice was made. The parameter group was set that way because Redis holds auth SESSIONS, not just cache: volatile-lru (the redis7 default) would evict TTL-bearing keys under memory pressure and silently log patients out. noeviction turns memory exhaustion into a loud write error instead.

Since 2026-08-14 it also protects caregiver grant codes — the short-lived, single-use codes a patient reads out to give somebody access to their record (caregiver_grant: keys, 15-minute TTL). Under an eviction policy a code could disappear before it expired, and the person redeeming it would be told it was invalid while the person who issued it saw no problem at all.

Nothing to apply for that feature in any environment: no migration, no catch-up script, no new secret. The whole reason it is in Redis rather than a table is that a row whose only future is deletion does not warrant a migration plus a catch-up on two environments. What it does need is Redis reachable from the api — which is already true — and this policy left alone. Loosening it later would break sessions first and grant codes second.

The cost of noeviction is that memory exhaustion refuses WRITES, and nothing currently watches for it. Verified 2026-08-14: this substrate has no ElastiCache alarm of any kind — not memory, not evictions, not connections (step 9 covers ALB, RDS, ECS and NAT only). Every load-bearing Redis path is a write — session creation, hold acquisition, rate-limit counters, locks, JTI reservations, cache-aside fills, grant codes — so a full instance is not a slowdown, it is an abrupt platform-wide write failure with no warning ahead of it. Tracked in platform-completion.md.

---

2. Populate Secrets Manager

./infra/scripts/populate-production-secrets.sh

Prompts for (production values): Clerk pk_live_/sk_live_; Cloudflare SaaS token + TF token + zone id; production Bunny Stream + Storage Zone creds; SES from-address; the legacy handoff secret (must equal the legacy Railway value); Sentry DSNs (optional). Auto-generates ENCRYPTION_KEYS + BACKUP_ENCRYPTION_KEY.

ENCRYPTION_KEYS is the crown jewel — it decrypts every pii_regulated / auth_secret column and every Cat A provider credential. Once written, do not rotate without the credential-rotation runbook.

⛔ If this is a REBUILD of an existing production, STOP here

Do not let this script generate keys. It creates ENCRYPTION_KEYS and BACKUP_ENCRYPTION_KEY with openssl rand whenever the container is empty, and on a rebuild every container is empty. Two consequences, both permanent:

• A restored database becomes unreadable. Every encrypted column — the CNP on patient_profiles, every auth_secret — is ciphertext under the oldENCRYPTION_KEYS. A new key does not fail loudly; it fails as garbage.
• The backups themselves become unreadable. Layer 2 artifacts are encrypted under BACKUP_ENCRYPTION_KEY, deliberately in its own envelope and under its own CMK. Regenerate it and the Object-Locked bucket still holds every dump and none of them can be opened. There is no second copy of that key.

So before running this on a rebuild: recover the original values first and write them in, rather than accepting generated ones. Sources, in order — Secrets Manager may still hold them (deleted secrets are recoverable within their recovery_window: aws secretsmanager restore-secret), then the custodian's offline copy. If neither exists, encrypted data is gone; say so plainly rather than proceeding and discovering it at restore time.

The script skips already-populated secrets, so a re-run against an intact Secrets Manager is safe — it is the empty-container case that destroys data. This is why step 2 comes before any restore, and why it is worth confirming aws secretsmanager get-secret-value --secret-id restartix/production/encryption returns the expected key before going further.

Terraform already populated restartix/production/redis, the JWT signing keys, the bearer tokens, and the bunny-webhook secret (random_id). The RDS master password is RDS-managed.

The script's last section fails today, after everything else has succeeded. It prompts for the Daily (video) bootstrap and then writes restartix/production/daily-bootstrap, which does not exist — production's local.bootstrap_secret_names carries only the SES, S3 and Clerk seeds (describe-secret returns ResourceNotFoundException, verified live 2026-08-12), so update-secret errors and set -e aborts. Nothing before it is lost; re-running with --only <name> picks up anything you still need.

That the container is absent is deliberate, not an oversight: production has no video substrate at all — no daily-bootstrap, no video-room-secret, no video-room-sweep / video-usage-reconcile crons — because every one of them would be inert until 000049 promotes, and an api task definition naming a versionless secret takes the crons down with it (video-infrastructure-plan §5). Video goes to production as one ordered change, from a staging config proven first. What is drift is only the script, which prompts for a secret its environment has no container for.

---

3. Bootstrap DB roles + telemetry DB

Open the SSM tunnel through the bastion (production has no NAT instance):

# terminal A — leave running
./infra/scripts/production-tunnel.sh                 # localhost:15432 → RDS:5432

# terminal B
ENV=production CONNECT_HOST=localhost CONNECT_PORT=15432 \
  ./infra/scripts/bootstrap-db-roles.sh

Creates restartix (owner) + restartix_app (RLS-restricted) + telemetry_app, the telemetry database, the platform extensions (pgcrypto, uuid-ossp, pg_trgm, unaccent, pg_stat_statements, vector), the pgbouncer userlist, and writes the canonical DSNs into restartix/production/database + /telemetry-database. The script's aurora_* terraform outputs are deliberately named to match its contract — production is RDS, the values are the RDS writer endpoint / db name / master secret.

---

4. Seed the ECR repos

On a rebuild, nothing below this line can run until an image exists. Since 2026-08-11 production owns restartix-production-<svc> (previously it shared staging's restartix-<svc> repos, which were already full — so this step was invisible and the runbook did not have it). terraform apply creates the repos empty, and every task definition it registers names :latest, which does not resolve. The migrations runner in step 5 is the api image; the crons are the api image; the services are their own images. All of them fail to launch with CannotPullContainerError: … not found.

Push without deploying, so the services do not start against an unmigrated database:

./infra/scripts/push-image.sh api --env production          # NO --deploy

That builds and pushes :latest + :<git-sha>. The api is a Go binary with no build-time environment baking, so push-image.sh's staging NEXT_PUBLIC_* defaults do not apply to it — that caveat is real for the Next.js images, which is why the reviewer-gated workflow builds those in step 6.

Recovering an existing production rather than building a new one? Do not rebuild — copy the images that are already running. Registry-side, within the same account, this mounts layers instead of transferring them (about a second per image) and the result is verifiable by digest:

docker buildx imagetools create \
  --tag <registry>/restartix-production-api:<sha> \
  --tag <registry>/restartix-production-api:latest \
  <source>/restartix-api:<sha>

Verify with aws ecr describe-images --image-ids imageTag=<sha> --query 'imageDetails[0].imageDigest' on both sides and compare. A rebuild produces a different image from the same source; a copy produces the same image. During a recovery that distinction is the whole point.

---

5. Migrations + partition roll

Requires step 4 — the runner pulls restartix-production-api:latest at RunTask time.

The promotion sequence, in one place

Production sits at 000039. The pending range is 000040–000051 — F1–F6, video consultations, family plans, and the consents-model correction — and it does not promote by running the migrations alone. Four catch-up scripts and one extension statement travel with it, in this order. Each is detailed in its own ⛔ block below; this table exists so nobody has to reconstruct the order from prose.

#	Do	Why here
0	jit-off-app-role.sql as master, then restart pgbouncer	independent of this promotion — apply it today. Costs nothing, waits for nothing, and is worth ~1.5s per statement on every RLS-planned read
1	CREATE EXTENSION "btree_gist" as master	000045 runs it as restartix, which cannot, and dies mid-migration leaving schema_migrations dirty
2	000006-caregiver-staff-visibility.sql	edited in place; forgetting it empties the Family panel, silently
3	000012-invite-claims-profile.sql	edited in place; forgetting it errors every patient invitation, loudly
4	000006-caregiver-legal-representative.sql	edited in place; the attestation that lets a caregiver act for an ADULT. Without it the clinic surface 500s on a missing column
5	000023-000034-history-follows-the-person.sql	edited in place; forgetting it empties every per-person read, silently. Contains a backfill over every existing run and grant
6	000008-consent-subject.sql (Part A)	additive; creates the new overload beside the old so the running code keeps working
7	run the migrations 000040→000051	the range itself — ⚠️ see the 000044 block below before running it against real acceptances
8	partition roll -ahead=3	partitioned tables have no DEFAULT partition
9	step 6 — deploy	reviewer-gated; code-only
10	000008b-drop-old-consent-subject.sql (Part B)	the removal, and only now: run it earlier and every consent gate errors until approval

⛔ Step 0 — jit-off-app-role.sql. Not part of the promotion; apply it whenever.

This one does not belong to 000040–000050 and should not wait for it. It changes no data, no schema and no code — only how Postgres executes plans for restartix_app.

Postgres decides to JIT-compile a plan from its ESTIMATED cost, and RLS destroys that estimate: a SELECT on session_exercises is planned through four OR'd policies that recursively inline the policies of sessions, programs, protocols, patients, catalog_entries and session_runs, so a query returning 19 rows becomes a ~750-line plan with ~3000 JIT'd functions. Measured locally against the demo clinic's 40-session programme under real patient RLS context:

statement	jit=on	jit=off
per-session exercises + catalog	1112 ms	8.4 ms
program-exercises aggregate	13810 ms	16.1 ms

The compiled code is never reused — the same prepared statement executed six times on one connection paid 1491/1482/1471/1466/1507/1478 ms. Plans are cached (pgx + pgbouncer max_prepared_statements); machine code is not, and Postgres offers no mechanism to make it so.

Production runs db.t4g.medium — 2 vCPU — and JIT compilation is CPU work that cannot be shared between connections. So this is a capacity fix rather than a latency polish.

psql "$PROD_URL_AS_MASTER" -f infra/scripts/jit-off-app-role.sql

Then restart pgbouncer, or nothing changes. ALTER ROLE … SET applies at backend startup, and pgbouncer keeps warm server connections (min_pool_size = 5, server_lifetime = 3600), so existing backends keep jit=on for up to an hour. Verified live on local: SHOW jit returned off on a direct :5432 connection and on through pgbouncer until the container was restarted. Force a new deployment of the pgbouncer service, then confirm:

SHOW jit;   -- through pgbouncer: expect off

Reverting is ALTER ROLE restartix_app RESET jit; plus the same restart. It is deliberately NOT set on the restartix owner role — AdminPool reads bypass RLS, so their cost estimates are honest, and cross-org Console analytics is the one place JIT could genuinely pay off.

bootstrap-db-roles.sh now carries the same statement, so a from-scratch environment gets it at step 3 and does not need this script.

---

Steps 2–6 are all additive and all idempotent, so they may be run together in one session ahead of the migrations. Step 10 is the only one that must wait, and it is the one most likely to be forgotten, because skipping it breaks nothing today — see its block at the end of step 6 for what it leaves behind.

Nothing above has been applied to production. As of 2026-08-15 every script here is applied to LOCAL only, except 000006/000012 which are also on staging. Staging has not rehearsed the full sequence either — and a rehearsal there is worth doing first, because a from-scratch staging rebuild validated every ADD CONSTRAINT against an empty schema, which proves the migrations are well-formed and nothing about how they meet production's rows.

This step assumes a NEW production with no data to preserve. If you are recovering an existing one, the data comes first and these migrations are the wrong tool: restore per backup-disaster-recovery.md, then run the migrations runner only to carry the restored schema forward to the image's version.

A restore may also need the catch-up scripts, and nothing else will tell you so. infra/scripts/*.sql hold DDL for migrations that were edited in place after production had already passed them — golang-migrate compares version numbers, not contents, so it will not re-run an edited migration and the restored database keeps the pre-edit shape at a version number that looks current. Compare the restored schema_migrations against the scripts present in that directory: any whose migration number the restore already passed is a candidate. Each script states its own applicability in its header, and each is written to be a no-op where it does not apply. Two exceptions to run nothing blindly: HOTFIX-restore-treatment-plans-column.sql is an emergency rollback that re-adds a dropped column, and 000004b-drop-treatment-plans-column.sql is a removal — see the ordering rule below before either. Restoring over a freshly migrated schema is not the same operation and loses whichever side you did second. Confirm the encryption keys are the original ones (step 2) before restoring — a restore under a new key produces a database that looks healthy and returns garbage from every encrypted column.

⛔ Before promoting 000045, create btree_gist as master

000045_scheduling.up.sql runs CREATE EXTENSION IF NOT EXISTS btree_gist (the appointments double-booking EXCLUDE constraint needs the GiST opclass for its equality column). The migrations runner connects as restartix, which has no CREATE EXTENSION privilege on RDS/Aurora — so the statement fails, the migration dies part-way, and schema_migrations is left dirty at 45. Production is at 000039 and has not met this yet; it will the moment the 000040–000050 range promotes.

Step 3's script now pre-creates it, but step 3 does not re-run on an existing production — the roles are long since bootstrapped. So on the promotion run this is a manual statement, through the tunnel, as master, before the runner starts:

CREATE EXTENSION IF NOT EXISTS "btree_gist";

Found on the 2026-08-12 staging rebuild, which was the first run to migrate past 000044 on managed Postgres. It cannot be caught in local development: local restartix is a superuser, so the migration succeeds there every time.

⛔ 000012 was edited too — run its catch-up as well

000012_invites.up.sql gained organization_invites.patient_profile_id and a CHECK confining it to patient invitations. It is what lets a dependent on a family account CLAIM their own login: accepting the invitation stamps patient_profiles.human_id on the person who already exists, instead of onboarding minting a second profile and leaving every appointment, form and consent behind on the first.

psql "$PROD_DIRECT_URL" -f infra/scripts/000012-invite-claims-profile.sql

Idempotent (ADD COLUMN IF NOT EXISTS, constraint dropped then re-added). Verify:

SELECT column_name FROM information_schema.columns
 WHERE table_name = 'organization_invites' AND column_name = 'patient_profile_id';

Forgetting it fails LOUDLY rather than silently, unlike the 000006 one below: the invite INSERT names a column that does not exist, so issuing any patient invitation errors.

STATE (2026-08-14): applied to STAGING, deliberately NOT to production. Staging was at 000049 with neither object present; both are there now and verified. Production was left alone on purpose — see the note under the 000006 block below, which applies to both scripts.

⛔ 000006 was edited after production passed it — run the catch-up

000006_patient_identity.up.sql gained patient_caregivers_select_org_staff, the RLS policy that lets clinic staff read a caregiver link when both people are already their patients (P7 family accounts). Production passed version 6 long ago, so golang-migrate will never re-run it — the table keeps its original single policy, which matches only the caregiver themselves or the patient.

The symptom is silent and easy to misread: the Family panel on a patient record renders nothing, and the API returns an empty list rather than an error, so it looks like the patient simply has no relatives on file.

Through the tunnel, as restartix (no master needed — it owns the table):

psql "$PROD_DIRECT_URL" -f infra/scripts/000006-caregiver-staff-visibility.sql

Idempotent (DROP POLICY IF EXISTS then CREATE), so a re-run is harmless. Verify:

SELECT policyname FROM pg_policies WHERE tablename = 'patient_caregivers';
-- expect BOTH patient_caregivers_select and
--             patient_caregivers_select_org_staff

It grants SELECT only. 000006's REVOKE on INSERT/UPDATE/DELETE stands, so a clinic can read a family link and still cannot invent one.

STATE (2026-08-14): applied to STAGING, deliberately NOT to production.

Both catch-ups were run against staging (at 000049, clean) and verified: patient_caregivers now carries both policies, organization_invites has patient_profile_id + its CHECK + org_invites_select_claim_subject.

Production was deferred on a deliberate call, not forgotten. Nothing running in production needs either script today: prod is at 000039 and the code that reads these objects is in the unpromoted 000040–000050 range. Applying DDL to a live database serving real patients buys nothing until that code ships, and the ADD COLUMN — though a catalog-only change on a nullable column — still takes a brief ACCESS EXCLUSIVE lock that queues behind any in-flight transaction on organization_invites.

So the intended moment is the promotion run, alongside the btree_gist statement above. Whoever does that promotion must run BOTH scripts; they are not carried by the migrations, and golang-migrate will not re-run an edited migration. The failure modes differ and both are recorded above: 000012 fails loudly (invitations error), 000006 fails silently (the Family panel renders empty).

⛔ 000044 DELETES consent rows — read before promoting

000044_consent_forms retires org_service_disclaimer by deleting it outright, and that means:

DELETE FROM consents                 WHERE purpose_code = 'org_service_disclaimer';
DELETE FROM consent_purpose_versions WHERE purpose_code = 'org_service_disclaimer';
DELETE FROM consent_purposes         WHERE code = 'org_service_disclaimer';
DELETE FROM organization_legal_documents WHERE document_type = 'service_disclaimer';
DELETE FROM legal_document_templates     WHERE document_type = 'service_disclaimer';

…followed by narrowing both document_type CHECKs to ('terms', 'privacy_notice').

Every row recording that a patient accepted that document is destroyed, along with every published version of its text. The consents ledger is otherwise append-only under six-year retention, and this is the one statement in the schema that contradicts that.

It was written deliberately, and the migration's own header argues it: at the time, production held a demo tenant and the rows were demo content, so a retired_at column plus its guard checks would have been machinery carrying a case that did not exist. That argument is sound only while the rows are disposable.

THIS IS A REMOVAL, so the ordering rule applies — but its blast radius is small enough that the rule bends. The code on master today still knows the third document type: legaldocument.DocumentType admits service_disclaimer and the clinic's legal-documents page iterates a fixed ["terms", "privacy_notice", "service_disclaimer"]. So between running the migrations and completing the deploy, that page still offers a type whose CHECK no longer admits it, and publishing one returns a 500.

That is one admin action on one page — not the 2026-08-11 shape, where a dropped column 500'd every authenticated request. And the alternative is worse: the new code cannot run against the old schema at all (it reads consent_purposes.enforcement, which lands in this same migration), so deploying first is not available. Run the range, then deploy promptly, and do not publish a service disclaimer in between.

The re-consent gate moves in the SAFE direction here, which is worth knowing before you worry about it. It switches from legal_basis <> 'consent' (platform_terms, platform_privacy_notice, org_terms, org_privacy_notice, org_service_disclaimer — five) to enforcement = 'blocks_access' (the same four, minus the deleted one). No patient is newly blocked; one demand is withdrawn.

So before running the migrations, check:

SELECT count(*) AS acceptances, min(granted_at) AS oldest, max(granted_at) AS newest
  FROM consents WHERE purpose_code = 'org_service_disclaimer';

If that returns anything belonging to a real patient at a real clinic, stop and take the decision explicitly. The alternative is small and is described in the migration header: add consent_purposes.retired_at, stop requiring the purpose, and leave the rows alone — a one-column migration and three lines of check. Doing that after the DELETE has run is not possible; the evidence is gone.

Nothing else in 000040–000050 destroys data.

The one-live-enrollment invariant is NOT in this range. It was written as a forward 000051 on 2026-08-22 and then folded into 000023 beside uq_one_active_prescription_per_patient, whose "Enrollments are unconstrained" comment it corrects. 000023 is applied everywhere, so neither staging nor production gets the index until it is rebuilt from 000001 — both sit at a version number that looks current with the pre-edit shape, exactly as the 2026-08-21 / 08-22 folds left them. Until then, only protocols.Service's pre-check stands between a patient and a duplicate enrollment, and it cannot see a concurrent double-tap.

A rebuilt environment creates the index against an empty protocols, so there is nothing to reconcile. The forward version carried a backfill that ended duplicate live enrollments; the fold drops it, because at 000023 the table does not exist yet. If either environment is ever brought forward by hand instead of rebuilt, that backfill has to come back — the index cannot be created while duplicates exist:

SELECT patient_id, source_program_id, count(*)
  FROM protocols
 WHERE kind = 'enrollment'
   AND status IN ('active', 'paused')
   AND source_program_id IS NOT NULL
 GROUP BY 1, 2 HAVING count(*) > 1;

STATE (2026-08-17): applied to LOCAL only — and see the note under the local-database warning below, because at the time of writing the local database had not run them either.

⛔ 000006 gained the legal-representative attestation

Four columns on patient_caregivers — is_legal_representative, the basis the clinic was shown, and who attested it when — plus a CHECK and a partial index.

It is what lets a caregiver accept the clinic's terms for an ADULT. Without it the rule is minors-only, and an 80-year-old with no email has nobody who can answer for them: they hold no login, and a relative on whose account they sit has demonstrated no standing.

psql "$PROD_DIRECT_URL" -f infra/scripts/000006-caregiver-legal-representative.sql

Idempotent, and the CHECK cannot fail on existing rows — every one of them has is_legal_representative FALSE. Additive, so it belongs before the deploy. Verify:

SELECT column_name FROM information_schema.columns
 WHERE table_name = 'patient_caregivers'
   AND column_name IN ('is_legal_representative', 'representative_basis',
                       'representative_attested_by_principal_id',
                       'representative_attested_at');
-- expect all four

Forgetting it fails LOUDLY: the clinic's family panel selects the columns and errors on a missing one.

STATE (2026-08-15): applied to LOCAL only.

⛔ 000023 + 000034 were edited — run the history catch-up

Both gained patient_profile_id beside their existing patient_id (session_runs, patient_content_grants), plus a derive-on-insert trigger and four replaced RLS SELECT policies. It is what lets a patient who withdrew and came back still reach their own exercise history and the content they bought — a patients row is one processing chapter, and both of those follow the PERSON.

Production passed 23 and 34 long ago, so golang-migrate will never re-run them. Through the tunnel, as restartix (no master needed):

psql "$PROD_DIRECT_URL" -f infra/scripts/000023-000034-history-follows-the-person.sql

Idempotent (ADD COLUMN IF NOT EXISTS, CREATE INDEX IF NOT EXISTS, DROP POLICY IF EXISTS then CREATE). Purely additive, so it belongs before the deploy. Verify:

SELECT count(*) FILTER (WHERE patient_profile_id IS NULL AND patient_id IS NOT NULL)
  FROM session_runs;                       -- expect 0
SELECT count(*) FILTER (WHERE patient_profile_id IS NULL)
  FROM patient_content_grants;             -- expect 0
SELECT tgname FROM pg_trigger WHERE tgname LIKE '%derive_patient_profile%';
-- expect BOTH session_runs_ and patient_content_grants_ triggers

It contains a backfill, and production is the first environment where that is not instant. The two UPDATEs touch every existing run and grant. Check the row counts before running and expect a write lock on session_runs for the duration.

Forgetting it fails silently and twice over: the columns stay NULL, so every per-person read returns nothing — a patient's activity log renders empty and their purchased content stops unlocking — and the new RLS policies are absent, so the patient's own reads still resolve through a live patients row. Neither errors.

STATE (2026-08-15): applied to LOCAL only. Staging and production both owe it, and it belongs to the same promotion run as everything else here.

⛔ 000008 was edited — and this one is TWO scripts, in two places

current_required_consent_versions gained a third argument naming the subject of the question. Without it the function asked "has ANYONE this login speaks for accepted?", so a parent's own org_terms satisfied the consent gate for a child who had consented to nothing.

The split is not bookkeeping — it is this runbook's own ordering rule.CREATE OR REPLACE with a new signature OVERLOADS rather than replaces, so the addition and the removal are separable, and separating them is what keeps the running code working across a reviewer-gated approval.

Part A — here, before the deploy. Creates the three-argument form beside the old one. Both exist afterwards; arity resolves them unambiguously, and the currently-deployed code keeps calling the two-argument one.

psql "$PROD_DIRECT_URL" -f infra/scripts/000008-consent-subject.sql

SELECT pg_get_function_identity_arguments(oid)
  FROM pg_proc WHERE proname = 'current_required_consent_versions';
-- TWO rows at this point is CORRECT: `uuid, uuid` and `uuid, uuid, uuid`.

Part B — after step 6, once the deploy has shipped the three-argument caller: 000008b-drop-old-consent-subject.sql. It is listed again at the end of step 6 so it is not lost here.

Run Part B early and it errors every consent gate on the platform until a reviewer approves — the same shape as the 2026-08-11 treatment_plans_enabled drop. Skip it entirely and nothing errors, but the union-based defect stays one un-updated caller away from being live: a rollback to the previous image brings it straight back.

STATE (2026-08-15): both parts applied to LOCAL only (local ran an earlier combined form, so it already has only the three-argument function). Staging and production owe both.

Applying migrations to a production that has rows is its own risk, separate from this runbook. A fresh database validates DDL against zero rows, so every ADD CONSTRAINT succeeds by construction; against real rows it may not. Check constraint-adding migrations against the live data before running them, and never deploy application code ahead of the migration it depends on — the deploy workflow runs no migrations, so code expecting a newer schema fails on reads that have nothing to do with the new feature.

And the rule has a second direction, which is the one that has actually broken production. Code-ahead-of-schema is the obvious hazard; schema-ahead-of-code is the quiet one:

Schema ADDITIONS may go before a deploy. Schema REMOVALS must go after it.

A column the new code no longer reads is harmless while the old code still runs. A column the old code does read, dropped before that code is replaced, breaks every request that touches it. On 2026-08-11 organization_entitlements.treatment_plans_enabled was dropped ahead of the deploy: the old binary named it in LoadRegulatedFlags, which runs inside RequireOrganizationScope — so the drop did not break one endpoint, it 500'd every authenticated request on the platform until the column was restored by infra/scripts/HOTFIX-restore-treatment-plans-column.sql.

The advice that produced it — "script first, deploy second" — is true only when the deploy follows immediately. Production's deploy is reviewer-gated by design, so the same ordering is an outage lasting until someone approves. Split any change that both alters data and removes schema into two scripts and run the removal after step 6.

Blast radius is worth checking rather than assuming: before dropping anything, grep for the column and find out whether its reader sits in middleware. A column read once per request is a very different removal from a column read on one admin page.

Run the migration runners as one-shot ECS tasks (private subnets, migrations SG):

SUBNETS=$(terraform -chdir=infra/envs/production output -json private_subnet_ids | jq -c .)
MIGR_SG=$(terraform -chdir=infra/envs/production state show 'module.network.aws_security_group.migrations_runner' | awk -F'"' '/^[[:space:]]+id /{print $2; exit}')
NET="awsvpcConfiguration={subnets=${SUBNETS},securityGroups=[${MIGR_SG}],assignPublicIp=DISABLED}"

aws ecs run-task --cluster restartix-production-cluster \
  --task-definition restartix-production-migrations-runner \
  --launch-type FARGATE --network-configuration "$NET" --region eu-central-1
aws ecs run-task --cluster restartix-production-cluster \
  --task-definition restartix-production-telemetry-migrations-runner \
  --launch-type FARGATE --network-configuration "$NET" --region eu-central-1

Wait for each to reach STOPPED with exitCode 0 (check /ecs/restartix-production-migrations-runner logs).

Then roll partitions ahead — migrations seed only a baseline month, and the day-25 cron may be weeks away; partitioned tables (audit_log, notifications, usage_records, session_, media_) have no DEFAULT partition, so current-month inserts fail without this:

APP_SG=$(terraform -chdir=infra/envs/production state show 'module.network.aws_security_group.fargate_app' | awk -F'"' '/^[[:space:]]+id /{print $2; exit}')
APP_NET="awsvpcConfiguration={subnets=${SUBNETS},securityGroups=[${APP_SG}],assignPublicIp=DISABLED}"

aws ecs run-task --cluster restartix-production-cluster --task-definition restartix-production-api \
  --launch-type FARGATE --network-configuration "$APP_NET" --region eu-central-1 \
  --overrides '{"containerOverrides":[{"name":"api","command":["/bin/api-partition-roll","-ahead=3"]}]}'
aws ecs run-task --cluster restartix-production-cluster --task-definition restartix-production-telemetry \
  --launch-type FARGATE --network-configuration "$APP_NET" --region eu-central-1 \
  --overrides '{"containerOverrides":[{"name":"telemetry","command":["/bin/telemetry-partition-roll","-ahead=3"]}]}'

(The api / telemetry task defs exist after step 1, even though their services run at the desired counts set in compute.tf.)

---

6. Build + deploy service images

Production deploys go through the reviewer-gated workflow — Actions → Deploy to production → Run workflow → "Use workflow from": master → service: all. It rebuilds each image with production NEXT_PUBLIC_* build-args, pushes to restartix-production-<svc>, registers an immutable :<sha> task-def revision, and smoke-tests /health for the live services. The production Environment's required reviewers must approve before the jobs run.

master is correct here, unlike the staging workflow: this role trusts environment:production rather than a branch ref, and a workflow_dispatch workflow only appears in the Actions tab once its file is on the default branch.

The deploy runs no migrations. It is code-only; step 5 is the only thing that moves the schema. That separation is what makes the order in this document matter — deploying code that expects a newer schema breaks reads that have nothing to do with the new feature.

⛔ AFTER this deploy: drop the old consent-gate overload

The other half of the 000008 split in step 5. Only now, with the three-argument caller live:

psql "$PROD_DIRECT_URL" -f infra/scripts/000008b-drop-old-consent-subject.sql

SELECT pg_get_function_identity_arguments(oid)
  FROM pg_proc WHERE proname = 'current_required_consent_versions';
-- ONE row now, reading `uuid, uuid, uuid`.

This is the step most likely to be skipped, because skipping it breaks nothing today. What it leaves behind is the union-based consent gate, still callable — so a rollback to the previous image silently restores a state where one family member's acceptance answers for another's.

• media stays at desired_count = 0 (no HTTP smoke test). The service is restartix-production-media — not …-media-service, which is what this runbook said until 2026-08-11 and which fails as an unknown service. Its autoscaler owns desiredCount, so raise the floor rather than scaling by hand, which production-media-toggle.sh does:

  ./infra/scripts/production-media-toggle.sh up      # floor → 1, waits healthy
  # …bake renders, confirm `ready` on Bunny…
  ./infra/scripts/production-media-toggle.sh down    # restores min = 0

• clinic runs at desired_count = 1 (compute.tf, verified live 2026-08-11). Earlier revisions of this runbook and a comment in the workflow both grouped it with media at 0; the Terraform is authoritative.

Emergency local hotfix path is push-image.sh — but it now refuses non-staging targets without --env production + a typed confirmation, and bakes staging NEXT_PUBLIC_* defaults unless overridden. Prefer the workflow.

---

7. Cloudflare / DNS / hostnames

1. Platform records (api/console/clinic/portal/telemetry/tv + wildcards) are created by the edge-cloudflare module in step 1, proxied. Verify they resolve and serve.

2. Custom domains (Cloudflare for SaaS). Custom domains are a production-only feature (staging's wiring was scaffolding to prove the path). The mechanism is the same for both parts below: Cloudflare terminates the custom hostname's TLS at the edge and forwards to the portal origin; the ALB's custom-domain catch-all rule (custom_domain_catch_all, priority 50000) routes the unrecognized Host to the portal target group — there is no per-host ALB rule (the hardcoded platforma host header was removed in 8725a71); the portal resolves the org from the Host / X-Forwarded-Host. Only domain_type=portal is accepted today — clinic-app custom domains are gated in AddDomain (bb9450a) until the edge dispatcher ships (see apps/docs/features/platform/custom-domains.md).

   a. Real clinic custom domains — restartix.pro zone. Enable Cloudflare for SaaS on restartix.pro; set the zone fallback origin to portal.restartix.pro (the prod portal ALB target); create the proxied customers.restartix.pro CNAME (the SaaS CNAME target — must match the api task def's CLOUDFLARE_SAAS_CNAME_TARGET). Clinics then self-register via the Console/Clinic custom-domain UI → cloudflaresaas.Register (wired into AddDomain, 4ff8be9) → they CNAME their domain at customers.restartix.pro; VerifyDomain polls Cloudflare until status=verified and resolveOrganizationByDomain serves it.

   b. Demo vanity host platforma.restartix.ro — restartix.ro zone (separate from restartix.pro). Enable Cloudflare for SaaS on restartix.ro; set its fallback origin to portal.restartix.pro. Register platforma.restartix.ro (DV cert), then add a verified organization_domains row for the restartix org (domain=platforma.restartix.ro, domain_type=portal) so resolveOrganizationByDomain resolves it.

3. Apex landing site (restartix.pro + www). The bare apex is served by a static splash from S3 (apps/landing/), same model as the TV companion — an apex bucket restartix.pro (Cloudflare flattens the proxied CNAME at the apex) plus a redirect-only www.restartix.pro bucket → 301 → apex. Both DNS records and buckets are created by step 1 (compute.tf). After apply, deploy the content with the landing job in deploy-production.yml (gh workflow run deploy-production.yml -f service=landing), then verify https://restartix.pro and https://www.restartix.pro. Without this the apex is an NXDOMAIN. (apps/landing/index.html is a placeholder splash — swap in real marketing content when ready.)

4. Flexible-SSL for S3-website hosts (cross-env, do once). The S3-website hosts have HTTP-only origins, and Cloudflare allows only ONE http_config_settings ruleset per zone — both rulesets are owned by staging's cloudflare.tf. Two zones are involved:

   • restartix.pro zone: the apex landing (restartix.pro) + www.restartix.pro. Rule lives in cloudflare_ruleset.restartix_pro_flexible_ssl.
   • restartix.tv zone: the TV companion apex (restartix.tv, prod) + staging.restartix.tv. Rule lives in cloudflare_ruleset.restartix_tv_flexible_ssl (+ cloudflare_zone_settings_override.restartix_tv for always_use_https).

   Apply staging so both rulesets land on their zones, otherwise those hosts return Cloudflare 525. The rules no-op until the production DNS resolves, so applying on staging ahead of production is safe. (The restartix_pro_flexible_ssl ruleset was renamed from tv_staging_flexible_ssl — terraform state mv it on staging before that apply; the exact command is in the resource's comment in infra/envs/staging/cloudflare.tf.)

5. ALB origin lock. The ALB SG accepts 443 only from Cloudflare IP ranges (data.cloudflare_ip_ranges). Confirm a direct hit to the ALB DNS name from a non-Cloudflare IP is refused, and all real traffic flows through Cloudflare.

---

8. Bunny + webhooks

Append the production webhook token to the production Bunny Stream library's webhook URL in the Bunny dashboard:

TOKEN=$(aws secretsmanager get-secret-value --secret-id restartix/production/bunny-stream-webhook-secret \
  --region eu-central-1 --query SecretString --output text)
# Set the library webhook URL to:
#   https://api.restartix.pro/webhooks/bunnystream?token=$TOKEN

Bunny is the only webhook production registers today. The video provider's (cmd/video-webhook-register) is not a step here yet and must not be run against production before the video substrate lands — there is no provider row to verify deliveries against, so it would arm an endpoint whose every delivery fails. When video promotes, this section gains the second registration and it is the easiest item in the whole runbook to forget: rooms open, consultations run, and the event stream, the minutes meter and the per-participant timeline are simply never populated. Procedure and credentials in video-infrastructure-plan §4.1; the staging form of the same step is in staging-rebuild-runbook.

---

9. Observability + cost

1. Confirm SNS email. Set alerts_email_subscribers in infra/envs/production/terraform.tfvars to an ops distribution address, apply, then click the confirmation link AWS emails (alarms/budgets don't deliver until confirmed).

2. Activate the Environment cost-allocation tag in Billing → Cost allocation tags (so the production budget's TagKeyValue filter matches; takes ~24h to populate). Until then the budget tracks $0 — drop the cost_filter temporarily for an account-wide budget if you need coverage sooner.

3. Confirm the six *-no-running-tasks alarms reach OK, not ALARM. Added 2026-08-11, one per always-on service (media is excluded — it idles at zero by design). They read RunningTaskCount from ECS/ContainerInsights, so Container Insights must be enabled on the cluster or the metric is never published; because the alarms treat missing data as breaching, that misconfiguration pages for every service at once. They sit in INSUFFICIENT_DATA for the first few minutes after creation and settle once datapoints land — verified taking about five minutes on 2026-08-11.

   aws cloudwatch describe-alarms --alarm-name-prefix restartix-production \
     --region eu-central-1 --query 'MetricAlarms[?StateValue!=`OK`].[AlarmName,StateValue]' --output text

   Empty output is the goal. Two families of alarm are expected not to be OK and are not incidents: the auto-created TargetTracking-…-AlarmLow-… pairs sit permanently in ALARM on an idle cluster, and media's pair reads INSUFFICIENT_DATA. Only the restartix-production-* named alarms matter.

4. Seed a datapoint for any cron heartbeat alarm immediately after the apply. A fresh heartbeat alarm sits in ALARM until its first success, which on a daily cadence could be a day away.

---

10. Smoke test

curl -fsS https://api.restartix.pro/healthz       | jq    # status/postgres/redis ok
curl -fsS https://portal.restartix.pro/api/health | jq
curl -fsS https://telemetry.restartix.pro/healthz | jq
curl -fsS https://console.restartix.pro/api/health| jq

Then exercise the demo end-to-end: legacy "Try it" → /enter (handoff verify + Clerk sign-in token) → /demo (consent + claim) → play the session → stats.

---

What this substrate intentionally does NOT do (tracked follow-ups)

• Layer 2 daily pg_dump backup is off. SHIPPED — enabled in production 2026-08-02 (enable_backup_cron = true), covering both the core and telemetry databases, each encrypted under its own envelope key into the Object-Locked bucket. On a rebuild, keep the ordering the flag's own comment records: deploy an api image carrying /bin/backup-runner + pg_dump first, then flip the flag — arming a 02:00 UTC schedule against an image that cannot satisfy it makes the backup-missing alarm page for a self-inflicted reason (treat_missing_data = "breaching"). The restore drill (restore-drill.sh, entirely in-VPC) is what proves the artifacts actually restore; a verified checksum is not a proven restore. See backup-disaster-recovery.md.
• Sentry SDKs not wired. The restartix/production/sentry container + DSN plumbing pattern are ready (see compute.tf), but the Go + Next.js SDKs + CI source-map upload are an app-owner follow-up. SENTRY_DSN is NOT in the task defs yet (wiring an unpopulated secret would fail task startup).
• HA NAT (one Gateway per AZ) is a documented Phase 2 lever — single-AZ NAT Gateway at launch.
• Real (v2) legal text / 412 re-consent, the full programs, DSAR/erasure, the k6 load gate, and cross-region backups are the real-launch gate, not the demo (june-demo.md → "Relationship to the real production launch").

Stress-test levers (Stream H)

• RDS bump: rds_instance_class tfvar → db.t4g.large / db.r6g.large for the stress weeks if the DB is the bottleneck; size back down after (Multi-AZ failover ~1 min).
• ECS autoscaling absorbs the load delta between the ~5000-patient stress peak and the ~100-concurrent tail (min/max per service in compute.tf).
• Keep the env live continuously — do NOT trade away HA (Multi-AZ RDS, Redis replica, 2× core/portal) to cut the ~$600/mo floor.

Teardown / safety

RDS (deletion_protection=true, final snapshot) and the ALB (enable_deletion_protection=true) refuse terraform destroy until you clear the flags in a deliberate apply. SM secrets have a 30-day recovery window. The backup bucket has Object Lock COMPLIANCE (7-year) — objects can't be deleted inside the window even by root. There is intentionally no production reset script.