Video consultations — the infrastructure that does not exist yet

Status (2026-08-10): written for STAGING, unapplied. Production deliberately carries none of it. The §1 defects are fixed and the cron changes in §3.2 are applied to production. grep -i video infra/ matched nothing when this was first written (2026-08-09), while F5.5 was built and browser-tested locally against a real provider account.

Production is excluded on purpose rather than forgotten: every video resource would be inert there until 000049 promotes, and the daily-bootstrap reference would actively break its crons (§5). It goes up as one ordered change, from a staging config that will by then be proven.

This is the whole substrate F5.5 needs in staging and production, plus the defects that surfaced while working it out — two of which are older than the feature and one of which was live in production.

Everything below is verified against the repository, not inferred. Where a claim came from asking the provider's live API, it says so.

---

1. What was already broken

These were found while planning the video crons. Two are fixed in the same change as this document; the third is a decision.

1.1 A scheduled binary was never in the image — FIXED

appointment-noshow-sweep was built in services/api/deploy/Dockerfile.api and never copied into the runtime stage. Its schedule is declared every fifteen minutes in staging and production (infra/envs/{staging,production}/crons.tf), so any run would have failed at exec.

No run ever did, because the schedule was never applied. Verified against live AWS on 2026-08-10: aws scheduler list-schedules returns ten production schedules and appointment-noshow-sweep is not among them, in either environment. Two independent gaps — a binary missing from the image and a crons.tf entry that never reached AWS — and the second is the only reason the first never fired. An earlier revision of this section claimed every run since it was added had failed; that was read off the repository and never checked against the live scheduler.

That the repo's cron declarations run ahead of what is applied is not specific to this one: expire-hard-cap-protocols is live in production and absent from staging, from the same drift.

Either way the consequence stands, and the ordering now has a second precondition: appointments does not exist on production (schema 000039; the table arrives in 000046), so arming the sweep today gives a job that exits 1 every fifteen minutes with nothing watching. It is therefore absent from production's crons.tf entirely — not gated behind a flag, because appointments exists permanently once it exists and a flag would be flipped once and then be dead config forever. It is live in staging, which reaches 000049 via the wipe-and-remigrate. The day F5 promotes without it, this becomes a platform whose no-show rate is structurally zero — precisely the defect the command's own doc comment describes in the system it replaces, and which it was written to correct.

Nothing would have caught it. A scheduled task that cannot start produces no application log, and the scheduled-task alarms are metric filters over log groups — they alarm on the absence of a success line, but a container that dies at exec never reaches any line, and the alarm's own window is what it is measured against. The scheduler's dead-letter queue sees RunTask launch failures, not what the process did after launching.

Fixed by copying it (and the two new video binaries) into the runtime stage, and by a check-cron-binaries target now in make check that fails the build when a binary is built and dropped.

1.2 Both video crons logged success on runs that failed — FIXED

video-room-sweep and video-usage-reconcile each signal failure by exit code — a room found and not closed, a provider over-count our append-only log cannot correct. Both logged "...: ok" before testing that condition and calling os.Exit(1).

Since the alarms are metric filters keyed on the success pattern, a failing run would have reported itself healthy on the day it failed. Now the success line is logged only on success, and the failure branch logs a distinct pattern (video-room-sweep: incomplete, video-usage-reconcile: overcount) that an alarm can key on.

An honest log with nothing watching it is still silence, which is why §3 wires a heartbeat on the sweep. The reconcile cannot have one — see §3.1 — so §1.3 is its only possible cover.

1.3 Nothing alarms on a cron's exit code — OPEN, platform-wide

The fix above closes the gap for these two commands by making the log honest. The underlying gap is not video-specific: no cron in this platform is alarmed on its ECS task exit code. Every alarm is a log-pattern filter, and the DLQ only sees launch failures.

The general fix is an EventBridge rule on ECS Task State Change watching stoppedReason / exitCode != 0 for the api task families, added to infra/modules/scheduled-tasks. That would close it for every cron at once, including the class of failure in §1.1. Not decided.

---

2. Secrets

Two existing patterns apply, and which one a secret takes is decided by who generates the value.

2.1 VIDEO_ROOM_SECRET — ours, generated per environment

It derives the opaque room name from the appointment id and never leaves the process. It is not a provider credential. Empty disables video outright rather than degrading — a room name derived from an empty secret is derived from the appointment alone, which is the legacy defect (where the room name travels in emails) arriving silently.

Same shape as media_service_bearer_token / telemetry_internal_read_secret: a standalone random_id + aws_secretsmanager_secret + aws_secretsmanager_secret_version triple, referenced by bare ARN — no local.sm entry, no :KEY:: JSON suffix.

	staging	production
Status	written, unapplied	not present — lands with 000049
Secret name	restartix/staging/video-room-secret	restartix/production/video-room-secret
Recovery window	0	local.recovery_window (30)

Wired into module.api.secret_env_vars as VIDEO_ROOM_SECRET = aws_secretsmanager_secret.video_room_secret.arn.

The value must differ per environment. A shared one would make staging able to derive production's room names, which are the only thing standing between an appointment id and a joinable room.

2.2 DAILY_API_KEY + DAILY_WEBHOOK_SECRET — bootstrap pair, set by hand

Seed-only, exactly like CLERK_SECRET_KEY and the SES pair: read once to create the platform_service_providers row, after which the row is canonical and these are ignored. Rotation therefore happens through the Console, not a redeploy.

They need no new Terraform resource — one new name in the existing bootstrap_secret_names list is enough, because aws_secretsmanager_secret.runtime already iterates it:

• staging: "restartix/staging/daily-bootstrap"
• production: "restartix/production/daily-bootstrap"

Then a local.sm entry, and two secret_env_vars lines using the JSON-key suffix form: "${local.sm.daily_bootstrap}:DAILY_API_KEY::".

infra/scripts/populate-*-secrets.sh carries a hardcoded secret list. Both scripts now include daily-bootstrap — without that, the secret is created empty and skipped in silence, which surfaces later as video being disabled for a reason nothing explains. The block prompts for the API key (blank = skip, video disabled) and generates the webhook secret itself, so the value registered with the provider and the value seeding the row come from one place.

DAILY_WEBHOOK_SECRET is ours to choose, not the provider's to issue: we generate it and hand it to the provider at registration (§4). Per environment, because a webhook's HMAC is per webhook.

Populate BOTH keys before the API ever boots with either of them set. providers.Bootstrap inserts ON CONFLICT … DO NOTHING (bootstrap.go:59), so a boot with the API key present and the webhook secret empty writes a row carrying an empty secret and no redeploy ever corrects it — the row is canonical from that moment. VerifyWebhook then fail-closes on every delivery, which is the right refusal and an expensive one: the provider retries, circuit-breaks, and the webhook goes FAILED (§4.1) while the API looks entirely healthy. Recovery is a Console rotation, not a deploy. Same trap, same shape, for a mistyped secret: the value the register command hands the provider must be byte-identical to the one that seeded the row.

2.3 DAILY_GEO

Plain env var, not a secret: DAILY_GEO = "eu-central-1". It has a Go default, so it is wired anyway on purpose — "where does a consultation's media land" is the first question a clinic's DPO asks, and a default buried in Go source cannot answer it.

It is load-bearing exactly once. The value is marshalled into platform_service_providers.config at bootstrap alongside the credentials (cmd/api/main.go), so after the row exists the task-definition value is inert — changing the region is a Console edit, not a redeploy. Wire it anyway, in both environments: it is what makes the region a reviewable line in compute.tf rather than a fact only the database knows. BuildDailyProvider refuses an empty geo rather than defaulting, so the failure mode here is a provider that will not build, never a room in an undeclared region.

2.4 IAM

Nothing to change. infra/modules/ecs-service derives the execution role's secretsmanager:GetSecretValue policy from whatever ARNs appear in secret_env_vars, and kms_key_arns_for_decrypt on module.api already covers the platform CMK that envelopes these.

---

3. Crons

The crons run the API task definition with a command override. Every entry in crons.tf points at module.api.task_definition_arn and overrides only containerOverrides.command; the module strips the revision suffix so it always runs the latest ACTIVE revision. Consequence: adding the three variables above to module.api gives them to every cron automatically — existing and new — with no per-cron wiring.

Two entries per environment. Production entries additionally carry the subnet_ids = null / security_group_ids = null / dead_letter_arn = null triad every production task-map entry has.

Cron	Command	Cadence	Heartbeat	Non-zero exit means
video-room-sweep	/bin/video-room-sweep	every 15 min	yes, max_age_days = 1	a room was found and not closed
video-usage-reconcile	/bin/video-usage-reconcile	monthly, 1st at 03:05 UTC	impossible — see §3.1	the provider counted fewer minutes than we did

03:05 rather than 03:00: usage-summary-rollup already runs at 03:00 on the 1st off the same task definition. Concurrent RunTasks are harmless, but five minutes of separation costs nothing and keeps one cron's logs from interleaving with the other's during the one window each month either of them is interesting.

video-room-sweep is not housekeeping. Since a session ending no longer closes a room (see features.md → F5.5), the only other closer is a clinician marking the consultation done — and consultations are not always marked done. Without the sweep:

• the room stays active forever, so the appointment reads as a consultation still in progress months later;
• its minutes are never settled, because metering happens at the close;
• the room is never deleted at the provider. Expiry does not reclaim it — exp stops new participants joining and nothing else — so an undeleted room counts against the account's ceiling indefinitely.

It is also where the event stream is repaired from the provider's own record, so a persistently non-zero events_repaired is the earliest signal that webhook delivery has degraded.

Nobody is ever stranded waiting for it: eject_at_room_exp removes whoever is still connected at expiry, provider-side, whether or not the job runs.

Both entries point at module.api.cron_task_definition_arn, not task_definition_arn — the smaller sibling task definition (256 CPU / 512 MiB) added 2026-08-10, rather than the service's 1 vCPU / 2 GB. Same image, env, secrets, roles and log group; only the size differs. See §3.2.

Order matters, and it has bitten before (see the backup-runner comment in crons.tf): the task definition pulls :latest at RunTask time, so deploy the image carrying the new binaries first, then create the schedules. Reversed, a schedule is armed against an image with no such binary — §1.1 in miniature.

And 000049 must be applied to that database before either schedule is created, for the same reason appointment-noshow-sweep is gated: a sweep whose tables do not exist exits 1 on every run, and nothing is watching. Image, then migration, then schedule.

3.1 Alarming: one heartbeat, and one cron that cannot have one

The scheduled-tasks module already carries the mechanism — heartbeat = { log_group_name, success_pattern, max_age_days } builds a metric filter over the success line plus a treat_missing_data = "breaching" alarm. Only the two partition-roll crons use it today. video-room-sweep gets the third:

heartbeat = {
  log_group_name  = module.api.log_group_name
  success_pattern = "\"video-room-sweep: ok\""
  max_age_days    = 1
}

At 96 runs a day, a full day of silence is unambiguous. Note the alarm's period is fixed at 86400s and evaluation_periods = max_age_days, so 1 is the floor the module can express — a sweep failing for an afternoon is not caught, a sweep failing for a day is.

Day one, the alarm sits in ALARM until the first success (breaching on missing data). Seed one datapoint immediately after apply, exactly as the module comment instructs for the partition rolls:

aws cloudwatch put-metric-data --namespace RestartiX/CronHeartbeats \
  --metric-name restartix-<env>-video-room-sweep-ok --value 1

video-usage-reconcile cannot have a heartbeat, and that is a property of the mechanism rather than an oversight. The alarm counts days without a success; a monthly cron is silent for ~31 of them by design. This is the same wall that moved the partition rolls from day-25-monthly to daily in 2026-06 — a dead-man's switch needs a cadence shorter than its own window. Three ways out, none of them free:

• Leave it uncovered until §1.3 lands. The reconcile's whole job is to catch a discrepancy at leisure, a month before invoicing; a run that fails silently costs one month of assurance, not one month of money. This is the recommendation.
• Move it to daily. It is idempotent and a no-op outside its window, so this is safe — but it turns a once-a-month provider aggregation call into thirty, against an account we share with the legacy system (§6).
• Alarm on the failure pattern instead of the success one (video-usage-reconcile: overcount). Fires on presence rather than absence, so it needs no cadence — but it catches only the failure the process survived to log, never the run that never started. That is the §1.1 class, and it is the one the DLQ also misses.

---

3.2 What a cron costs, and why these run a smaller task

Measured on production, 2026-08-10, not estimated.

Every cron run is a fresh Fargate task. Eight sampled prod tasks ran 28–33s wall (pullStartedAt → stoppedAt; the image is 75 MB and pulls in 4–9s over the S3 gateway endpoint). All are under Fargate's 1-minute billing minimum, so every run bills exactly one minute — and until 2026-08-10 it billed that minute at the api service's size, because crons RunTask the service's task definition with a command override.

That size was 1 vCPU / 2 GB. Container Insights on the same family: 10.5 MiB used of 2048 MiB reserved, 3.4 CPU units of 1024. So a sweep rented an HTTP-sized task for a full minute to do seconds of work at 0.5% of its memory.

Hence cron_task_cpu / cron_task_memory on the ecs-service module — a sibling task definition at Fargate's 256/512 floor, identical in image, env vars, secrets, IAM roles and log group (the heartbeat metric filters key on the log group; diverging would break api-partition-roll's alarm silently). At eu-central-1 rates that moves a run from $0.000946 to $0.000237.

Cadence	Runs/mo	Was	Now
every 15 min	2,920	$2.76	$0.69
monthly	1	—	—

So the video substrate's crons cost ~$0.70/month, plus ~$0.40 in Secrets Manager + KMS calls (17 secrets injected per task start, at $0.05 and $0.03 per 10k). EventBridge Scheduler is ~$0.004 at this volume, CloudWatch Logs ~$0.01. Call it $1.10/month for the pair.

Two findings from the same measurement, both outside this document's scope but recorded because they came from it: check-providers ran at rate(1 minute) — 43,800 runs/month, ~$47 — to decrypt three rows, since that binary registers no Cat A factories and therefore probes nothing. It is now hourly. And the whole prod cron fleet fell from ~$63/month to ~$4.

4. Out-of-band steps

4.1 Register the webhook — once per environment

cmd/video-webhook-register is an operator tool, not a cron. It reads its credentials straight from the environment rather than through config.Load(), refuses non-https, and is idempotent.

It runs from a workstation, so the operator supplies the credentials by hand — the same pair that seeded the provider row, read back out of Secrets Manager rather than retyped:

export DAILY_API_KEY=$(aws secretsmanager get-secret-value \
  --secret-id restartix/staging/daily-bootstrap --query SecretString --output text | jq -r .DAILY_API_KEY)
export DAILY_WEBHOOK_SECRET=$(aws secretsmanager get-secret-value \
  --secret-id restartix/staging/daily-bootstrap --query SecretString --output text | jq -r .DAILY_WEBHOOK_SECRET)

go run ./cmd/video-webhook-register -url https://api-staging.restartix.pro/webhooks/daily

Production is the same with restartix/production/daily-bootstrap and https://api.restartix.pro/webhooks/daily.

Read the secret; never retype it. The value registered here is what the provider signs with, and the value in the provider row is what we verify with. They are stored in two different systems and compared by HMAC, so a divergence produces no error anywhere — only deliveries that fail verification until the provider gives up (§2.2). Reading both from the one secret that seeded the row is what makes them the same value by construction rather than by care.

Register only after the API has booted with the secret populated, so the provider row exists first. Registering against an environment whose row was never seeded arms a webhook whose deliveries can never verify.

Miss this and nothing looks broken. Rooms open, patients connect, consultations run — and the event stream, the minutes meter and the per-participant timeline are never populated. It is the single easiest item here to forget and the most expensive to notice late.

A registered webhook can also go FAILED on its own. The provider circuit-breaks after repeated delivery failures and then stops sending everything; the record reads state: FAILED, retryType: circuit-breaker. Observed in development on 2026-08-09 after a tunnel dropped mid-session. Re-running the command re-arms it — it now prints the state before and after, and exits non-zero if the update leaves it non-ACTIVE. There is no alarm for this in any environment.

4.2 Roll partitions immediately after migrating

Staging does not get an incremental migrate up. It sits at 000038 — behind production, and materially drifted (six columns, two of them NOT NULL, plus eight indexes) from migrations that were edited in place before anything had been promoted. 000049 cannot land on top of that honestly. The plan is a full wipe and re-migrate (infra/scripts/reset-staging-db.sh, which refuses to run unless ENV=staging), after which staging carries 000001–000049 from a clean base and is a faithful rehearsal for production again. Everything below then applies to production only, where the schema is verified zero-drift at 000039.

000049 seeds only its authoring month, per the minimal-seed pattern. Applied in any later month, the current month's partition does not exist until the 02:00 roll, and there is no DEFAULT partition by design — so every inbound call event fails its INSERT for up to a day.

For most tables that surfaces as a user seeing an error. Here the rows arrive from a provider webhook, so it surfaces as deliveries answered non-2xx, a provider that retries and then stops, and a permanently empty timeline for those consultations.

Migrations run from the migrations_runner task definition (migrate -path /migrations -database "$DATABASE_DIRECT_URL" up) as a manual RunTask; the deploy workflow has no migrate step. Appending && /bin/api-partition-roll -ahead=3 to that command makes this impossible to forget, for every future partitioned table rather than just this one. It is idempotent, so re-running costs nothing. Proposed, not applied.

---

5. The order everything goes up in

Two dependencies decide this order, and both were found the hard way rather than reasoned about.

Terraform applies before the app deploy. The deploy action runs describe-task-definition --task-definition <family>, which returns the latest ACTIVE revision — so it clones whatever Terraform last registered. Apply first and the video env vars reach the service on the next deploy; deploy first and the service runs without them until the deploy after that, because ignore_changes = [task_definition] stops Terraform moving the service itself.

The secret is populated before any task definition references it. Terraform creates daily-bootstrap as an empty container with zero versions, and ECS secret injection is fail-closed: a task launched from a revision naming a versionless secret dies at ResourceInitializationError. The service survives that (it stays pinned by ignore_changes), but the crons do not — they follow the latest revision by design, so they would begin failing to start within minutes. This is precisely why production gets none of this until it is promoted as one ordered change.

#	Step	Why here
1	terraform apply -target= the secret containers only	Creates them without touching a task definition or a schedule
1a	…and add any pending moved resources to that same target set	Terraform errors with Moved resource instances excluded by targeting otherwise — see below
2	populate-staging-secrets.sh --only daily-bootstrap	The ARN must resolve before step 3 registers a revision naming it. A blank API key is a valid answer — it disables video
3	terraform apply — everything else	Task defs, schedules, and whatever unrelated drift the env carries
4	Deploy the apps	CI clones step 3's revision and pins :<sha>; :latest now carries 000049 and both video binaries
5	reset-staging-db.sh (staging) / migrate (production)	Must follow step 4 — the migrations runner pulls :latest at RunTask time, so an earlier reset migrates with the old image
6	Seed the heartbeat, register the webhook (§4.1), verify a cron exits 0	The heartbeat alarm sits in ALARM until its first success

Step 1 will refuse to plan if the environment carries a pending moved. Terraform will not accept a target set that splits a move — it errors Moved resource instances excluded by targeting, the same trap network/README.md records for the HA-NAT cutover. Staging has exactly this today: the private route table went per-AZ in the module, so aws_route.private_default is moving to [0]. Add the moving resources to the target set:

./staging-apply.sh plan \
  -target=module.network \
  -target=aws_secretsmanager_secret.runtime \
  -target=aws_secretsmanager_secret.video_room_secret \
  -target=aws_secretsmanager_secret_version.video_room_secret \
  -target=random_id.video_room_secret

which plans as 6 add / 3 change / 0 destroy — the secrets plus the route restructure, still no task definitions and no schedules. Read the plan before assuming the target set is right; the moves an environment carries are whatever has drifted since its last apply, not a fixed list.

Between steps 3 and 5 the crons fire against a database that has not been migrated yet and fail, and step 3 restores desired_count, so services come up against the old schema. In staging both are harmless and both clear at step 5. To avoid them entirely, hold the schedules back and apply -target=module.crons as a final pass after step 5.

The short form: infra, then apps, then the database — the wipe depends on the deployed image, and the deploy depends on Terraform's task definition.

6. What needs no change

• ALB / listener rules. aws_lb_listener_rule matches on host_header only, not path, so /webhooks/daily is already routed to the API target group exactly as Bunny Stream's inbound webhook is on the same host.
• Security groups / WAF. Production's ALB accepts inbound 443 solely from Cloudflare's published ranges, and the provider's webhook traverses the same path the existing inbound webhook already does. No WAF exists on either ALB — pre-existing and not video-specific.
• Partition rolling. video_session_events is registered in partitionroll.rollers, so the existing daily api-partition-roll cron rolls it and partitions.EnsureMonthly applies the same per-leaf REVOKE ALL … FROM restartix_app. Recorded here so nobody adds a second roller.

---

7. The provider account

Today there is one account, and it is the legacy system's production account — 13,228 meetings, real patient names visible in its dashboard. A separate account per environment is strongly recommended before real clinics onboard.

Sharing is survivable, and it is worth being precise about why rather than hand-waving:

• Metering is safe. Reconcile resolves every room against our own database and counts anything it does not own as Unknown, so one environment can never meter another's minutes.
• Presence and span reads are safe. Both filter to one exact room name, and FetchRoomSpans refuses a name that is not ours before it asks.
• Webhooks do not cross — they EXCLUDE. This was wrong here until 2026-08-12, and the correction is the strongest argument on the page. A domain may hold exactly one webhook: a second create answers 400 only 1 webhook is allowed per domain. So two environments on one account do not both receive events with some noise; the one registered last receives everything and the other receives nothing, in the silent way §4.1 describes. Promoting video to production on this account would therefore take the slot from staging, and re-pointing it back would take it from production. cmd/video-webhook-register now refuses to move a slot that points elsewhere unless given -replace, so this is at least a decision rather than an accident.
• Room refs cross. A room ref is a secret with no egress target, and production's would transit the staging process during a usage read.
• The room ceiling is 100,000 (confirmed on the account dashboard, not the help-centre page, which documents a lower tier) and is shared. Not a practical constraint.

---

8. Open decisions

Decision	Why it is not made here
Separate provider account per environment	Account and billing decision, not code — but no longer optional if both environments need an event stream, because the one-webhook-per-domain limit (§7) makes that impossible on a shared account. It stays open only for as long as production video is unshipped. A second API key on the same account does NOT help: keys authenticate, they do not scope webhooks, rooms or metering.
Exit-code alarming for crons (§1.3)	A cross-cutting change to scheduled-tasks affecting every cron, not just these two. It is also the only thing that can ever cover video-usage-reconcile (§3.1), so this one is no longer purely optional — it is the difference between a monthly job that is watched and one that is not.
Appending the partition roll to migrations_runner (§4.2)	Changes how every migration is applied; worth a deliberate yes.
Cron cadences	15 min / monthly match the nearest existing convention and the binaries' own doc comments. Confirm before applying.
Alarming the FAILED webhook state (§4.1)	A registered webhook can circuit-break itself and nothing notices. The cheap version is a state read inside video-room-sweep, which already runs every 15 minutes and already talks to the provider — but it widens a sweep into a monitor. Not decided.

---

Related

• features.md → F5.5 — the feature, its eight clinic-driven reversals, and what is still open
• production-launch-readiness.md — carries this as one unchecked box under Infrastructure
• monitoring.md — the alarm conventions §1.3 is about