Skip to content

Production Launch Readiness

The operational gate that flips the platform from "F11 hardening complete" to "real clinic, real patients, real revenue." This is distinct from F11: F11 is the technical/regulatory feature set (features.md → F11); this document is the operational checklist that sequences the cutover and confirms every external dependency, every runbook, and every sign-off is in place.

This is not a feature spec. It's a gate document — a living checklist whose real-launch target is ~September 2026 (full launch: real clinics + real patients + the ~20k legacy-patient migration). Realigned to September full-launch, 2026-06-19.

Production is already live (opened early at the June-10 demo)

The production substrate (infra/envs/production/) went live 2026-06-05 and stays live continuously — it was opened early to serve the June-10 single-session demo and has been serving real patients since. The September gate is therefore graduation, not provisioning: the env, RDS, ECS services, crons, alarms, and budgets already exist (verify each against the Infrastructure checklist below). What September adds on top of the running env is:

  1. Real terms + re-consent — publish real v2 platform_terms plus each clinic's org_terms, which version-bumps every demo alumnus and migrated patient into a 412 re-consent on next request (the consent ledger's RequireCurrentConsents gate).
  2. The ~20k legacy-patient migration — transform/import the ~20k legacy patients (11k+ treatment plans, 5k+ subscriptions) into the existing live prod DB, including catalog Bucket 3 (D2C first-party clinic + tier/premium seed). Tracked launch-gate item; plan closer to launch.
  3. Flip the deferred items live — media-service + clinic app to running desired counts, full programs feature + a seeded patient catalog, DSAR/erasure end-to-end, the k6 load gate, cross-region backups, and the steady-state HA posture (see the per-area checklists below for which are done vs. open).

The active September plan is june-launch.md; its scope folds into this gate.

The three gates, in order:

  1. Foundation gate (1E.3) — staging deployed, foundation acceptance test passes against AWS staging. No real patients. (foundation.md)
  2. Layer 2 / F-tier feature build-out — F1–F12 features built on top of the substrate. (features.md)
  3. Production launch readiness — this doc. The operational gate before real clinics see real production.

See decisions.md for the architectural decisions feeding this gate.


Acceptance criteria

The platform is ready for first paying clinic when all of the following are true:

  • Foundation 1E.3 closed (staging gate passed) — CLOSED 2026-05-15
  • All F-tier features required at launch are merged, tested, deployed to staging, and validated end-to-end
  • F11 hardening complete (technical + regulatory features shipped, production environment provisioned and load-tested)
  • All pre-launch wiring tasks below complete
  • Romanian regulatory counsel has signed off on F11.0.5 findings
  • Legacy-data migration runbook executed in dry-run against a copy of the legacy database
  • Incident response on-call rotation exists and has been tested
  • Sign-off list below is fully checked

If any one is missing, the platform is not launching today.


Pre-cutover gates

Grouped by area. Each must be green before the cutover runbook executes.

Live production incidents (hard launch blockers)

Because prod is already serving real patients, open production incidents are launch blockers in their own right — they have to be closed before the env graduates to the September full launch.

  • [ ] 🚨 iOS video-playback failure — exercise videos fail to play on iOS Safari: the Bunny hotlink-protected URL returns a CoreMedia 403 (only ~30% session-completion on iOS today). Root cause is hotlink protection rejecting the CoreMedia/AVPlayer request; the fix is token-authenticated signed Bunny URLs (the player requests a per-asset signed URL the CDN accepts) rather than relying on referer/origin hotlink rules. Both a current production incident and a hard launch blocker — a third of patients can't complete a session without it. Tracked launch-gate item.

Foundation + features

  • [x] Foundation 1E.3 closed and locked (CLOSED 2026-05-15)apps/docs/implementation-plan/foundation.md shows the 1A–1C + 1E.3 substrate checkboxes checked (1D admin surfaces remain partially in flight, tracked there)
  • [ ] F1–F8 features that are part of launch scope are merged, tested, and deployed to staging
  • [ ] F9 telerehab is in launch scope OR explicitly deferred — confirm decision
  • [ ] F10 Telemetry service is deployed if F9 ships at launch (telerehab depends on telemetry; see telemetry/index.md)
  • [ ] F11 technical hardening complete: GDPR DSAR endpoint works end-to-end, prod KMS rotation tested, security scan passes, performance benchmarks recorded against staging-shape production environment

Infrastructure

The env is already live (opened 2026-06-05) — this section is now validate the live demo-prod env + flip the remaining HA/scale posture, not "provision from scratch." Items the Terraform already implements are pre-checked; the open boxes are the validation tests + steady-state decisions that haven't run yet.

  • [x] Production AWS environment provisioned via Terraform (infra/envs/production/) and idempotent — done; live since 2026-06-05

  • [x] RDS Multi-AZ instance running (db.t4g.medium, multi_az = true in main.tf) — provisioned; failover still needs a validation test (see open item below)

  • [x] The base crons (api-partition-roll, telemetry-partition-roll, usage-quota-reset, usage-summary-rollup, check-providers at 1-min prod cadence) are defined in crons.tf via EventBridge Scheduler → ECS RunTask — confirm each has actually run successfully in prod at least once (CloudWatch success-pattern + DLQ depth = 0)

  • [x] CloudWatch alarms + alerts SNS topic configured per monitoring.md (observability.tf) — done (ALB, per-service CPU/mem, NAT alarms enabled)

  • [x] AWS Budgets alarms set at 50% / 80% / 100% (actual) + 100% (forecasted) of monthly budget (observability.tf) — done

  • [ ] Synthetic RDS-failover test — force a Multi-AZ failover (reboot-with-failover) and confirm pgbouncer reconnects + the app recovers within target; not yet exercised in prod

  • [ ] pgbouncer drain test — confirm one pgbouncer task can drain/redeploy without disrupting in-flight runtime traffic

  • [ ] Custom-hostname WAF decision — see the WAF-coverage open item under Operational readiness; the L7-WAF-for-custom-domains call still has to land

  • [ ] Steady-state app_replicas decision — the demo posture pins low replica counts; pick + apply the launch steady-state (and the matching media/clinic desired counts — see the demo→launch state delta under External services)

  • [x] Layer 2 backups live and PROVEN RESTORABLE (2026-08-02). cmd/backup-runner + pg_dump in Dockerfile.api shipped; enable_backup_cron = true; daily 02:00 UTC. Covers both databases — core and telemetry (telemetry is a separate database, so the core dump never contained it). A backup-missing CloudWatch alarm fires on absence of a success heartbeat, and the heartbeat only publishes after the uploaded object is read back and re-checksummed.

    The first drill failed, and that is the point. Production could not be restored from its own backup: principal_is_human() referenced principals unqualified, and pg_dump restores under an empty search_path, so the inline CHECK on platform_memberships killed the data load. Fixed in migration 000039. No checksum or alarm could have caught it — only an actual restore.

    Verified green on 2026-08-02 via ./infra/scripts/restore-drill.sh production: core restored in 11s with all 13 structural counts matching source exactly (180 tables, 731 indexes, 2285 constraints, 246 functions, 109 RLS-enabled tables, 292 RLS policies, 371 partitions) and 119,865 rows across 62 tables; telemetry restored in 3.3s with 50,817 rows and 70 partitions.

  • [ ] Recurring restore drills. Run monthly and after any change to the backup pipeline, schema tooling, or Postgres major version. Currently on-demand — decide whether to schedule it before launch.

  • [ ] ALB + Cloudflare end-to-end (HTTPS, WAF active, custom-hostname provisioning verified) — proven for the demo path; re-verify at launch scale

  • [x] Production origin is edge-only. The production ALB security group accepts inbound 443 solely from Cloudflare's published IP ranges (data.cloudflare_ip_rangesalb_ingress_cidrs in envs/production/main.tf, consumed by modules/network/main.tf). This is the control that makes every edge protection actually binding: without it an attacker reaches the origin directly, bypassing the WAF and forging CF-Connecting-IP to rotate rate-limit buckets at will — the bypass internal/shared/clientip/clientip.go flags in its own doc comment. Verified in Terraform 2026-08-09.

  • [ ] No Cloudflare WAF or rate-limiting ruleset exists in Terraform — for any zone. envs/{staging,production}/cloudflare.tf manage zone settings (TLS 1.2 floor, always-HTTPS), managed headers, flexible-SSL and a www redirect, and nothing else. Whatever L7 protection exists today is dashboard-configured: unversioned, unreviewed, and invisible to anyone reading the repo. Land edge rate-limiting rules + the managed WAF ruleset as Terraform at the zone level, so clinic custom hostnames inherit them and the config is reviewable. Note this is a precondition for option (a) of the custom-hostname WAF decision below, which presumes zone WAF rules that do not currently exist.

  • [ ] Staging's ALB is open to 0.0.0.0/0 (modules/network/variables.tf, documented as an orange-cloud-masked escape hatch). Consequence worth stating: staging's origin is directly reachable and its rate-limit keys are forgeable, so staging is not a faithful rehearsal of production's edge posture. Decide whether to close it or accept it explicitly.

  • [ ] App-layer rate limiting is not DDoS defence, and shouldn't be counted as one. By the time ratelimit.Middleware rejects a request it has already reached an ECS task, taken a connection, and cost a Redis round-trip. It also cannot answer a distributed flood at all: the public surfaces most likely to be targeted — org resolve (300/min in prod), public booking (5/min), holds (60/min) — are all IP-keyed, so rotating source IPs hands the attacker a fresh bucket per IP. Only the edge sees the aggregate. See P60.

  • [ ] Video consultations (F5.5) have NO infrastructure at allgrep -i video infra/ matches nothing, while the feature is built and browser-tested locally. Needed before 000049 promotes: three secrets (VIDEO_ROOM_SECRET generated per env; DAILY_API_KEY + DAILY_WEBHOOK_SECRET as a bootstrap pair) wired into the api task-def, an EventBridge entry for video-usage-reconcile with a success-pattern alarm, and a one-shot cmd/video-webhook-register run per environment. Partitions need no Terraform (video_session_events already rolls under api-partition-roll), but the promotion runbook must roll partitions immediately after migrate: a new partitioned table applied mid-month has no partition for that month until 02:00 the next day, and these rows arrive from a webhook, so the failure is silent — consultations run normally and leave no record. Full list in features.md → F5.5 — infra and ops.

  • [ ] Async render pipeline — scale-out (deferred from staging). Staging runs a single, fixed media task; the durable render queue + dispatcher work without any scaling infra. Two pieces are intentionally not wired until production load justifies them (see async-render-pipeline.md → Production rollout):

    • [ ] Media autoscaling on queue depth. API emits a CloudWatch metric (pending-render count / oldest-pending age); a Fargate target-tracking policy scales the media service out when the bake queue builds and back in when it drains. Until this ships, 429 backpressure + the dispatcher's retry/backoff keep a single media task safe — just slower under a burst.
    • [ ] Redis concurrency token-bucket + status cache. The moment media runs more than one instance, add a Redis token-bucket sized to total media bake capacity (acquire before dispatch, release after) so the uncoordinated per-instance dispatchers don't over-dispatch past total capacity. Same trigger is the natural point to add the Redis cache for the patient "preparing" poll. Pinned to the first media scale-out so "later" isn't "too late."

Testing & CI

The 5 layers of testing the platform should have at production launch. Where each layer sits, what it catches, and what's already in place.

LayerWhatWhere it runsStatus
UnitPure-function tests with mocks. make test (Go race-detector) + pnpm test (TS).Local + CI on every PR✅ Wired — ci.yml runs make/pnpm checks + tests on every PR (PR #33).
IntegrationReal Postgres via testcontainers. make test-integration. Includes setup_clinic_test.go.Local + CI on every PR✅ Wired — ci.yml runs make test-integration on every PR (PR #33).
SmokeDeployed service responds correctly to a basic request. aws ecs wait services-stable + curl /healthz + jq assertion.Post-deploy in GitHub Actions✅ Wired in both deploy-staging.yml and deploy-production.yml (the prod workflow asserts api + telemetry /healthz).
End-to-endReal user flows through real UI against deployed env (Playwright/Cypress).Nightly cron against stagingDeferred until F-tier features worth E2E-testing
LoadPerformance + capacity under realistic traffic (k6/Locust).On-demand against dedicated load-test envDeferred until pre-launch

Pre-launch wiring:

  • [x] CI on every PR.github/workflows/ci.yml runs pnpm check + make check + make test + make test-integration on every pull_request (PR #33). Catches regressions before merge. ~5-15 min per PR; well under the GitHub Actions free-tier quota even at heavy PR volume.
  • [x] Production smoke test — already wired in deploy-production.yml: each service job runs aws ecs wait services-stable and the api + telemetry jobs additionally curl -fsS .../healthz + jq-assert status == "ok" (and postgres/redis for api). Note deploy-production.yml rebuilds images per-env (each job tags :${{ github.sha }} and docker-build-pushes; Next.js bakes env-specific NEXT_PUBLIC_* at build time) — there is no SHA/release-tag promotion of staging artifacts. Open follow-up: extend the prod smoke beyond /healthz (e.g. an authenticated canary) per the cutover acceptance step.
  • [ ] External synthetic monitor on production — Cloudflare Health Checks (free) or Better Stack / UptimeRobot. Continuous external probe of https://api.restartix.pro/healthz from multiple regions; catches outages independent of the deploy pipeline.
  • [ ] Real-user monitoring — Sentry SDK in api + 3 Next.js apps captures unhandled errors with stack traces and user context. Already on the External services checklist below; cross-listed here because it's the production half of the testing picture.
  • [ ] Nightly E2E against staging — Playwright suite covering ~5 golden paths (sign in, create org, schedule appointment, view treatment plan, sign out). Scheduled GHA workflow that runs against https://*-staging.restartix.pro. Worth adding when ≥3 F-tier features are live.
  • [ ] Pre-launch load test — k6 script targeting expected launch-day traffic profile (20k users in first org migration; ~5k month-1 active; ~1k peak concurrent). Run against staging or a dedicated load-test env. Validates autoscaling thresholds + Aurora ACU sizing + pgbouncer pool capacity.

Test-fixture contact details — one email domain, one phone number:

Every fabricated email in a fixture, seed, factory or UI placeholder uses @restartix.com. Every fabricated phone number is 0755 617 273 (+40755617273 normalized).

Never invent one. An invented address or number is a real person's address or number roughly as often as it isn't — and the platform is being built toward transactional email and SMS. The day a test factory, a seeded demo org or a stray staging run points its notifier at production credentials, an invented contact detail becomes a message delivered to a stranger who never heard of the clinic. There is no recall, and under GDPR an unsolicited message to an uninvolved data subject is a reportable event, not an embarrassment.

Two narrow exceptions, both because the test is specifically about the shape of a contact detail:

  • RFC 2606 reserved domains (example.com, *.test, *.invalid, *.example) where the case exercises an unusual domain structure the house domain cannot express — a two-letter domain, a multi-label suffix, plus-addressing. These are permanently undeliverable by standard.
  • Ofcom's reserved drama range +44 20 7946 0xxx, used in phone_test.go for the one case that needs a genuinely foreign calling code. Permanently unallocated.

Anything reserved is safe; anything registrable is not. clinic.com, example.ro and gmail.com are registrable, and all three were in fixtures before this rule was written (swept 2026-08-06).

The house number is also the number in every doc comment that illustrates the phone normalizer, so grep -rE '0?7[0-9]{2}[ .-]?[0-9]{3}[ .-]?[0-9]{3}' over the repo returns exactly one number. That is what makes the convention checkable rather than aspirational.

What we deliberately do NOT do:

  • E2E or load tests against production with real user data — too risky.
  • Full integration test suite (setup_clinic_test.go) re-run against staging on every deploy — 1E.2 closed by scope reduction; the GHA smoke step covers the high-value cases. The full suite stays as local + PR-CI run.
  • Cross-cloud chaos engineering — overkill until you have multi-region production.

External services + sub-processors

  • [ ] Sentry — org created, projects per service (API + 3 Next.js apps + Telemetry if F10 launches), source-map upload step in CI working, release tracking by image SHA, alert routing to Slack tested
  • [ ] Cloudflare — Pro plan active, Cloudflare for SaaS configured, custom-hostnames API token in Secrets Manager, end-to-end test of clinic registering a custom domain
  • [ ] Bunny CDN — account active in EU region, DPA reviewed and signed, Bunny Stream library set up, admin upload workflow tested with reference exercise videos, signed playback URLs working from Patient Portal
  • [ ] Clerk — production-mode keys (not test keys), HIPAA-eligible plan if applicable, BAA signed, webhook endpoint working
  • [ ] Daily.co — production keys, HIPAA-eligible plan if F5 telerehab video is in scope, BAA signed
  • [ ] Anthropic — API keys provisioned for AI agent capabilities (foundation 1C.8); per-org budget controls active via 1C.7 metering; AI agent service shape decided and documented (foundation memory: AI agent runtime is still an open scoping question — close before agents go live)
  • [ ] AWS SES (infra) — production identity verified, DKIM + SPF + DMARC configured, sandbox exit confirmed, account-level suppression list active, SES configuration set with bounce + complaint event destinations pointing at an SNS topic
  • [ ] AWS SES (app-layer code, gap re-verified 2026-06-19) — the API still has no bounce/complaint webhook handler and no notification_suppression table. Build before launch:
    • SNS topic restartix-prod-ses-feedback subscribed to SES bounce + complaint events; HTTPS subscription points at a new public API endpoint with SNS signature verification
    • Migration adding a notification_suppression table (recipient, reason: hard_bounce / complaint / manual, suppressed_at, source event ID) — RLS restricts reads to platform-admin (suppression is a platform concern, not per-org)
    • Webhook handler at POST /v1/internal/ses-feedback that verifies the SNS signature, parses the SES event payload, inserts into notification_suppression
    • EmailChannel precheck in internal/core/notify/: before dispatch, query suppression by recipient address; if present, mark the notification dead-lettered with dead_letter_reason='suppressed' rather than calling SES
    • migrations/core/000010_notifications.up.sql deferred-list line about "Bounce / complaint webhooks + suppression list automation" gets ticked here
  • [ ] All sub-processors disclosed in the platform DPA template (1B.10) and the Romanian-localized version

September-gate items folded in from june-launch.md

The demo opened prod early with several features pinned off or deferred. September flips these live — each is a launch-gate row here, sourced from june-launch.md:

  • [ ] Media service + clinic app live — both run at reduced/zero desired counts for the demo; bring media and clinic to their launch desired counts (see the demo→launch state delta below)
  • [ ] Full programs feature + seeded patient catalog — the demo shipped a single curated session; September needs the full programs feature and a seeded patient-facing catalog
  • [ ] DSAR / erasure end-to-end — patient access-export + anonymization-erasure validated against real (migrated) data, not just the demo path
  • [ ] k6 load gate — see the Pre-launch load test item under Testing & CI; run against the ~20k migration + month-1 active profile before opening to real clinics
  • [ ] Cross-region backups — reconcile with the "What's deliberately not in this gate" note below, which defers cross-region S3 backup replication to within the first quarter post-launch. Decision needed: is cross-region a launch blocker for September, or does it stay a post-launch Q-1 item? Whichever way it lands, it presupposes the daily logical backup is actually on — i.e. enable_backup_cron = true (currently false, see Infrastructure) is the prerequisite.

Demo → launch state delta. Flipping out of the demo posture means, at minimum: (1) set app_replicas to the launch steady-state (the demo pins it low); (2) raise media desired_count from its demo value (and clinic desired_count from 0); (3) remove the /dev/perf route and the diag-* diagnostic harness that exist only for demo-window debugging. Capture the full delta in the cutover runbook so nothing demo-only leaks into the launched env.

  • [ ] F11.0.5 Romanian compliance pass — counsel engaged, full findings documented, privacy notice template (1B.10) and DPA revised, ANSPDCP enforcement scan complete
  • [ ] Sub-processor list published to a public-facing page on the platform website per GDPR Art. 28 transparency
  • [ ] DPA template ready for clinic onboarding — countersigned versions stored per clinic
  • [ ] MDR Class I posture confirmed by regulatory counsel (or upgraded to Class IIa with appropriate process changes — see CLAUDE.md → Medical Device Readiness)
  • [ ] AWS BAA accepted via AWS Artifact (free; HIPAA-eligibility on the AWS account)
  • [ ] Legacy product DPA termination plan — when the legacy product shuts down, what's the data-handover and termination notice to its current users

Data migration

  • [ ] Legacy migration runbook in deployment.md → Runbook: launch-day legacy-data migration executed end-to-end as a dry run against a copy of the legacy database
  • [ ] Row-count and integrity validation queries documented and known to pass against the dry-run output
  • [ ] Legacy passwords (if migrating user accounts) — confirm Clerk's password import path or force-reset-on-first-login flow
  • [ ] Patient consent re-acquisition flow ready — legacy consents may not satisfy the new consent ledger schema; per-purpose re-consent on first login if needed
  • [ ] Rollback plan validated: PITR restore from "pre-launch" RDS snapshot works in <1h

Operational readiness

  • [ ] On-call rotation documented and committed to (PagerDuty / Slack alerts / phone tree)
  • [ ] Incident response playbook in monitoring.md → Incident Response Procedures reviewed by all on-call engineers
  • [ ] Synthetic incident drill — chaos test from monitoring.md executed in staging; alerts fired correctly; on-call responded within target time
  • [ ] Status page — public status page configured (statuspage.io / similar), automated by CloudWatch alarms or manual updates
  • [ ] Support escalation path — first-line clinic support routes to a human; engineering escalation path defined; severity levels with response-time targets
  • [ ] support@restartix.pro mailbox provisioned — referenced in the break-glass email template (break_glass_opened.email.{en,ro}.tmpl). The template tells the recipient to contact support if anything looks wrong; without a working mailbox at that address, the email is misleading. Set up via Google Workspace / Microsoft 365 / Fastmail / similar with MX records pointing at the chosen provider. Same provisioning thread can land noreply@restartix.pro (send-only, no inbox needed) and any other addresses the platform uses.
  • [ ] Cloudflare WAF coverage decision for clinic custom domains — by default, Cloudflare WAF rules on the restartix.pro zone do NOT extend to clinic custom hostnames (e.g. physio-bucharest.ro) via Cloudflare for SaaS. DDoS protection is always-on; L7 WAF is not. Three options at launch: (a) subscribe to Cloudflare's "WAF for SaaS" add-on so zone WAF rules extend to all custom hostnames, (b) reintroduce AWS WAF on the ALB to cover L7 attacks at the origin (reverses the "Cloudflare-only WAF" decision in decisions.md), or (c) accept the gap and rely on application-layer defenses. Decision needs to land before the first clinic with a custom domain handles real patient traffic. Two corrections from the 2026-08-09 audit: option (a) presumes zone WAF rules that do not exist in Terraform (see the edge items under Infrastructure — nothing but TLS/redirect/header settings is managed as code), and option (c) is weaker than it sounds, because app-layer rate limiting is IP-keyed and a distributed flood gets a fresh bucket per source IP.
  • [ ] Customer success runbook — what does first-clinic onboarding look like? Manual handholding for the first ~5 clinics, then standardized
  • [ ] Documentation portal for clinics — admin-facing how-to guides for setup-a-clinic, manage-staff, configure-billing, etc.

Deferred: promote the 2026-08-11 entitlement + usage work

  • [ ] 24 commits sit unpushed on staging as of 2026-08-11, deferred by the platform owner. Production runs a binary behind its own schema. Nothing is broken — the one incompatibility was resolved by restoring organization_entitlements.treatment_plans_enabled — but the gap grows with every further change, and the ORDER of promotion is now load-bearing:

    1. infra/scripts/000003-grandfather-org-entitlements.sqlbefore or with the deploy. The new code gates video on video_consultations_enabled, which is FALSE on production; without this the clinic loses video consultations the moment it ships.
    2. 000004-entitlement-scope.sql, 000004-tiers-is-public.sqlbefore. Additions; the new code needs the columns.
    3. 000004-limit-doctrine.sql, 000004-entitlement-cleanup.sql — any time. Data only.
    4. Deploy.
    5. 000004b-drop-treatment-plans-column.sqlafter, and only after. This is the one that took production down on 2026-08-11 when run early.

    The rule the split encodes: schema additions can precede a deploy; removals must follow it. "Script first, deploy second" is safe only when the deploy follows immediately — with a human-gated deploy it is an outage with a wait attached.

  • [ ] Staging is at 000038 and materially drifted — behind production, and never touched by this work. It is not a faithful rehearsal for any of the above until reset.

Sales + commercial readiness

  • [ ] First-clinic contract signed and includes the platform's standard MSA + DPA

  • [ ] Pricing locked for the launch tier (shared-mode default per tenant-isolation.md)

  • [ ] Billing flow tested end-to-end — clinic onboarded, subscription created, first invoice generated (manual via FGO at launch is fine; F12 engine ships later)

  • [ ] Patient consent flow at launch tested — patient signs up, accepts platform + clinic consents, onboards into the clinic, can withdraw consents granularly

  • [ ] Decide whether tenancy_mode = 'dedicated' is worth building for the RestartiX clinic itself. Raised 2026-08-11, deliberately unanswered.

    The RestartiX clinic moved onto the dedicated tier in production on 2026-08-11 (commercial SKU: unlimited caps, all four commercial entitlements). Its tenancy_mode is still shared, and that is correct, not an oversight — the two are different axes that happen to share a word. The tier is what an org is billed; the tenancy mode is where its data physically lives. See glossary → Tenancy Mode.

    What dedicated tenancy actually means today is one structural change: a per-tenant auth-provider organisation (its own Clerk org). Own-S3 and own-CMK are explicitly NOT part of it — they ship later as entitlements, once an exit/portability tool and a documented crypto-shred runbook exist. See decisions.md → Why entitlements, not columns.

    Nothing can provision it. Every creation path lands shared; the column is a schema reservation and no Go code branches on it. Flipping it by hand would claim an isolation posture the infrastructure does not provide, which is a false compliance statement rather than a display bug.

    The question to answer before launch is whether being our own first dedicated tenant is worth the provisioner, or whether shared-mode isolation (RLS + prefix scoping + app-layer checks) is what we would honestly tell a customer is sufficient — because if it is sufficient for us, the argument for selling dedicated to anyone else gets harder, and if it is not, we are running our own clinic on a posture we would not sell. Either answer is informative; leaving it undecided is the only bad outcome.

    Note also: the docs say the dedicated tier "maps to" tenancy_mode = 'dedicated'. Nothing performs or enforces that mapping — the tier move on 2026-08-11 left the column untouched, correctly. That sentence describes an intention, not a behaviour.


Cutover runbook

Day-of sequence. Estimate: 2–6h depending on legacy-data migration size.

This is not a first-ever cold cutover. Production has been live and serving real patients since 2026-06-05. T-0 transforms/imports the ~20k legacy patients into the existing live prod DB (alongside the demo-alumni rows already there) — it does not stand up a new env. Run the import via the existing tooling: the SSM tunnel (infra/scripts/production-tunnel.sh) + infra/scripts/production-apply.sh for schema/data steps. infra/scripts/production-reset-db.sh exists but is destructive — it is not part of this runbook (the prod DB has real patient data; never reset it at cutover).

T-7 days

  • [ ] Communicate cutover schedule to first clinic (start time, expected duration, what they need to do)
  • [ ] Take a manual RDS snapshot of the live production DB as the pre-migration rollback point (the DB already holds real demo-era patient data — this is the restore target if the import goes wrong)
  • [ ] Confirm all sub-processor health (Cloudflare, Sentry, Bunny, Clerk, Daily.co, SES, Anthropic)
  • [ ] Final dry-run of the legacy migration against a copy

T-1 day

  • [ ] Final go/no-go review with everyone on the sign-off list below
  • [ ] Confirm on-call rotation knows it's "live tomorrow"
  • [ ] Pre-warm Cloudflare cache rules; verify edge cert + custom-hostname for the first clinic

T-0 (cutover day)

The target DB is the already-live production database, reached via the SSM tunnel (production-tunnel.sh) + production-apply.sh. Steps 1–7 import the legacy data into it; the migration freeze becomes forward-only at this gate — once real migrated patients are in the live DB, migrations are no longer freely reorderable/wipeable (the early-dev "migrations are editable" rule ends here; from T-0 on, every schema change is a new forward migration).

  1. Maintenance mode on legacy — return 503 with a branded maintenance page; communicate to legacy users
  2. Snapshot legacy database — full pg_dump from the legacy host, verified
  3. Pull legacy dump to a workstation with the operations IAM role
  4. Run data-transform pipeline (services/migration-tools/legacy-import per the runbook in deployment.md) — transform + import the ~20k patients (11k+ treatment plans, 5k+ subscriptions) into the live prod DB via the SSM tunnel
  5. Publish real v2 terms + version-bump re-consent — publish the real platform_terms and each clinic's org_terms; the version bump puts every demo alumnus and every freshly-migrated patient into a 412 re-consent gate on next request (the consent ledger's RequireCurrentConsents). Migrated patients accept on first login
  6. Validate row counts against expected targets — block on any mismatch (account for the pre-existing demo-era rows already in the DB)
  7. Spot-check a few real legacy users in the new system — sign in, hit the re-consent gate, verify profile, see expected appointments
  8. Take a "post-import" RDS snapshot as a known-good launch state
  9. Remove maintenance mode, route DNS to production
  10. Synthetic acceptance test — run a small canary script against production: sign in, list orgs, list specialists, create a test appointment, confirm it persists
  11. Notify first clinic — they can start onboarding their staff and patients
  12. Watch monitoring for 24h — on-call active, alarm channels open, dashboards visible

T+24h, T+1 week, T+1 month

  • [ ] T+24h: review error rate, p99 latency, no-failed-deploys, alarm-noise volume; if any of these is concerning, pause new clinic onboarding until resolved
  • [ ] T+1 week: post-launch retrospective with the team; what surprised us, what we'd do differently, what runbooks need updating
  • [ ] T+1 month: first paid invoice cycle complete (clinic charged successfully, AI cost roll-up correct, no dunning surprises)

Post-cutover monitoring (first 30 days)

What to watch, who watches, what triggers action.

SignalWhereThresholdAction
5xx error rateCloudWatch alarm restartix-production-api-5xx> 0.5% sustained 10mOn-call investigation
p99 latencyCloudWatch (API target group)> 2s sustained 10mOn-call investigation
RDS connection saturationCloudWatch> 80% of max_connectionsCapacity review
RDS replica lag (when read replicas exist)CloudWatch> 5s sustainedReplica health check
Clerk auth failuresClerk dashboard + SentrySpike vs. baselinePossible auth incident
Sentry new-error rateSentryNew issue class with high volumeTriage same-day
Bunny CDN delivery errorsBunny dashboard> 1% sustainedCDN health check
AWS spend trajectoryAWS Cost Explorer + Budgets> forecast for the monthCost review
Daily backup statusCloudWatchMissed backup or checksum mismatchCritical — investigate immediately
Audit-partition-roll cronCloudWatchFailureCritical — ensure next-month partitions exist

The full alarm catalogue lives in monitoring.md. This table is the post-launch focused subset.


Sign-off list

Before the cutover runbook executes, every signature here is required:

Sign-offOwnerConfirms
EngineeringTech leadF11 hardening complete; production environment validated; backup posture verified; incident response playbook reviewed
Regulatory / ComplianceRomanian counsel (F11.0.5)Privacy notice + DPA templates approved; MDR class confirmed; data residency confirmed; sub-processor list published
OperationsOn-call leadOn-call rotation in place; status page live; support escalation defined; chaos drill complete
Customer SuccessFirst-clinic onboarding leadFirst clinic ready; onboarding runbook tested; documentation portal usable
CommercialFounder / business ownerFirst-clinic contract signed; pricing locked; billing flow tested

What's deliberately not in this gate

To keep the gate honest about what blocks launch vs. what's nice-to-have:

  • F12 Billing engine — not a launch blocker. First clinic gets manually-cut FGO invoices until the engine ships (features.md → F12).
  • F13 Dedicated tenancy mode — deferred until first paying dedicated-mode contract.
  • Multi-region / data residency per-tenant — out of scope per CLAUDE.md.
  • Mobile apps — open decision in features.md; web-only is acceptable for launch.
  • Customer-managed KMS — Phase 1 uses AWS-managed; CMK migration triggers documented in aws-infrastructure.md → Customer-managed KMS migration path.
  • Datadog APM — CloudWatch + Sentry covers the launch; Datadog deferred until traffic + team scale justify it.
  • Cross-region S3 backup replication — Layer 3 of backup-disaster-recovery.md deferred to within first quarter post-launch.
  • Phase 2 read replicas — added when triggers in scaling-architecture.md → Lever 5 fire.