Async Render Pipeline

How exercise renders are produced in the background — durably, decoupled from the UI, and scaled on demand — without the media service ever needing API or database access.

Status: CORE SHIPPED (phases 1–3); patient-readiness + ops remain. The durable queue, the background dispatcher, the publish-trigger cutover, and the Bunny transcode-complete webhook are implemented and green — the pipeline runs end-to-end (publish → enqueue → bake → transcoding → webhook → ready). What remains is the patient-facing readiness surface (Phase 4) and the scaling/reconcile ops work (Phase 5). See Implementation status.

SQL/columns are illustrative

Shapes convey intent; the authoritative schema lives in services/api/migrations/core/. Reuse existing tables where noted rather than adding new ones.

---

Why this exists

A render (compose a per-recipe MP4, upload to Bunny Stream, emit the cue manifest + TV baseline rungs) takes 30s–2min of ffmpeg. Today the API produces them synchronously, fanned out as goroutines:

EnsureRendersForSession (sessions/service.go) spawns one goroutine per exercise with context.WithoutCancel. That detaches from the request — so the goroutines do outlive a closed UI — but the model is fragile:

• Not durable. An API restart (deploy / crash / OOM) loses every in-flight bake. No retry, no dead-letter, no resume.
• Unbounded fan-out. 100 exercises → 100 goroutines → 100 simultaneous media calls. No backpressure; this thunders the media fleet.
• "Ready" is premature. The render is marked ready right after the upload, but Bunny transcodes the HLS ladder asynchronously (1=created → 4=ready). The composer "does NOT block on transcode completion" (stream/client.go). So there is a window where the DB says ready but the HLS isn't playable.

The async pipeline fixes all three with a durable queue, a SKIP-LOCKED worker, and a Bunny webhook for true readiness.

---

Core principle: the media service stays stateless

The single most important constraint: the media service must not need API or DB access. It stays exactly what it is today — a stateless render worker: given a recipe + an asset location, bake, upload to Bunny, return the result. No queue, no database, no knowledge of programs or patients.

All async machinery lives in the API. The only thing that changes is who calls media: today a request goroutine, tomorrow a durable background worker. Media doesn't know or care. This preserves the clean split — media = pure compute, API = state + orchestration — and means the entire pipeline can be built without touching the media service's contract.

---

Architecture

 Clinic UI ──publish──▶ API ──enqueue──▶ exercise_renders (status='pending')   ← the queue (Postgres)
                                                       │
                       API render worker  ◀───────┘   SELECT … FOR UPDATE SKIP LOCKED
                       (one per API instance; the notify-dispatcher pattern)
                                   │
                                   ├── HTTP POST /v1/exercises/compose ──▶  Media service
                                   │      (stateless: stage assets → bake → verify → upload → return)
                                   ▼
            render lifecycle:  pending → rendering → transcoding → ready
                                                  │                  ▲
                                                  └─ failed          │
                                                                     │
            Bunny ── webhook (video status=4) ──▶ API inbound-webhook receiver ┘

Five pieces, all but the media call living Core-API-side:

1. Queue — the exercise_renders table itself. A status='pending' row is a queued job. Durable, transactional, and already content-addressed by recipe_hash (duplicate work dedupes for free). No new table, no new infra — this is exactly how notifications is the notify queue.
2. Worker — an API background dispatcher, a direct copy of the notify dispatcher (notify/dispatcher.go): FOR UPDATE SKIP LOCKED claim, poll interval, lease + retry + dead-letter. It claims a pending render, calls media over HTTP off the request path, and persists the result.
3. State machine (below) — adds a transcoding phase so readiness is honest.
4. Bunny webhook — flips transcoding → ready when Bunny finishes the HLS ladder, via the existing inbound-webhook framework (inboundwebhooks/). Bunny is its first real consumer.
5. Redis (cache layer) — status cache for patient polling, a concurrency limiter, and an optional wake signal.

---

Render state machine

The status enum already exists (pending | rendering | ready | failed, migration 000022). Add one state — transcoding — to separate "we uploaded the master" from "Bunny can serve it."

State	Meaning	Driven by	Next
pending	Enqueued, not yet claimed	enqueue (publish)	rendering
rendering	A worker claimed it and is baking + uploading	worker (lease held)	transcoding / failed
transcoding	Master uploaded; awaiting Bunny's HLS transcode	worker → Bunny	ready / failed
ready	Bunny transcode complete; HLS playable	Bunny webhook (or poll backstop)	terminal
failed	Bake or transcode failed after retries	worker / webhook	terminal (dead-letter)

Invariant carried forward + extended: status='ready' ⟺ manifest_url IS NOT NULL AND Bunny transcode complete (today it's only the manifest half).

Lease + retry. A claimed render carries a claimed_at / lease (a worker that dies lets the lease expire → another worker re-claims). A bounded retry count → dead-letter on exhaustion. Identical to notify's delivery state machine.

Idempotency / resume. Because renders are content-addressed (recipe_hash + master sha), a retry produces the same artifacts. The Storage-Zone outputs (manifest, baselines) are content-addressed paths → re-bake overwrites, never orphans. Only the Bunny Stream video gets a fresh GUID per attempt — see absorbed parked items.

---

Enqueue path (publish-triggered)

Trigger: program publish (the settled decision — sessions are final at publish, so no baking doses the specialist then edits).

On publish, for each exercise in each session:

1. Compute the render key (exercise_id, recipe_hash, language).
2. Cache hit (a ready render already exists — the common case, since renders are shared) → nothing to do.
3. Miss → insert/flip an exercise_renders row to pending. Return immediately; the specialist's UI never blocks.

Because content is shared, most enqueues are cache hits — a published program with 12 exercises usually produces 0–2 new pending rows.

---

Readiness model

Two-phase, then program-level all-or-nothing.

• A render is ready only when Bunny has transcoded it (the webhook), not merely when uploaded.
• A program (for the patient) is ready only when every one of its renders is ready — all-or-nothing for v1.

Why all-or-nothing and not partial: "last session ready, first not" is worse than "preparing" — the patient can't start. The ideal is prefix-ready (usable up to the first not-ready item, rendering racing ahead of the patient's progress), but that requires the player to hold mid-program when the next item isn't ready yet. That's a real complexity; deferred. v1 = all-or-nothing; prefix-ready is a later enhancement.

---

Patient & clinic experience

• Specialist assigns immediately after publish — no waiting on renders.
• Patient's account shows the program "preparing" (derived from render status, cached in Redis so polling doesn't hit Postgres).
• When all renders go ready → program flips "ready" → patient sees it (optional notify via the notify primitive).
• A render that dead-letters → notify the prescribing clinic ("Exercise X couldn't be prepared"). Patient stays "preparing" on that program (all-or-nothing) until it's fixed and re-rendered.

---

Scaling

The insight that frames everything

Renders dedupe at the exercise level, and exercises have an ownership tier:

• Platform exercises (shared catalog) → one exercise_id → renders shared across every clinic. Prescribed 1000× = one render.
• Clinic-custom exercises (org-owned, the orgs/{organization_id}/ tier) → a distinct exercise_id per org → renders scoped to that org, shared across that org's programs and patients.

So the total render universe = platform library (fixed, globally shared) + Σ per-org custom libraries — it grows with custom content, not with patients, prescriptions, or clinic count. After the first bake everything is a cache hit. The queue and metrics key on exercise_id, so both tiers flow through identical machinery.

Consequences:

• "10 clinics × 100 exercises" is overwhelmingly cache hits → near-instant. Real baking happens only for genuinely-new combos.
• "2 specialists × 100 new renders at once" (the real worst case) = ~200 jobs queued, draining at media throughput.

The bottleneck and the scaling lever

The cost is the ffmpeg bake — CPU-bound, in the media service. The Postgres queue and the API workers are trivially light. So:

• Scale the media service on queue depth. The API publishes a CloudWatch metric (pending-render count / oldest-pending age); ECS Fargate target-tracks it. Queue builds → media scales out (e.g. 2 → 20) → 200 renders drain in parallel in minutes → queue empties → scale back in. This is the when/how: scale the bakers on queue depth. See scaling.md.
• Backpressure: media enforces its own per-instance concurrency and returns 429 when full; the worker retries with backoff. Self-regulating; no global coordinator needed at one instance.
• Concurrency limiter (Redis), added at the first media scale-out. The moment media runs more than one instance, uncoordinated workers across the API fleet can over-dispatch past total capacity. At that point introduce a Redis token-bucket sized to total media capacity (acquire before dispatch, release after). Pinned to that trigger so "later" isn't "too late."
• Fairness: FIFO with a light priority bump for renders blocking an assigned patient's program, so one specialist's 100-job batch doesn't starve another's.

Worker topology

The API runs as several instances. Every instance runs the dispatcher, and they coordinate purely through FOR UPDATE SKIP LOCKED — when one instance locks a row, the others skip it and take the next. No leader election, no external coordinator, nothing to configure. notify already operates this way.

---

Redis roles (summary)

Use	When
Status cache for patient "preparing" polling	from v1 (cheap, high read:write)
Concurrency token-bucket sizing dispatch to media capacity	at first media scale-out (>1 media instance)
Wake signal (pub/sub) to react instantly instead of polling	optional optimization

---

Production rollout (deferred from staging)

The durable queue, the dispatcher, the Bunny-webhook readiness, and the dead-letter→owner notify all run on staging with no scaling infra — a single, fixed media task, coordinated by 429 backpressure + retry/backoff. Two pieces are intentionally deferred until production load justifies them; they are tracked in production-launch-readiness.md → Infrastructure so they surface when the production substrate is built:

• Media autoscaling on queue depth. API emits a CloudWatch metric (pending-render count / oldest-pending age); a Fargate target-tracking policy scales the media service out as the queue builds and back in as it drains. Not on staging: one media task handles staging's load, and 429 + backoff keep it safe under a burst (just slower). Wire it with the production infra.
• Redis concurrency token-bucket + status cache. Pinned to the first media scale-out: the moment media runs more than one instance, the uncoordinated per-Core-API-instance dispatchers can over-dispatch past total bake capacity, so add a Redis token-bucket sized to total capacity (acquire before dispatch, release after). The same trigger is the natural point to add the Redis cache for the patient "preparing" poll (cheap, high read:write).

Neither deferral is a correctness gap — both are throughput/coordination optimizations that only bind once the media tier scales past one instance.

---

What this absorbs

The pipeline is the natural home for several items parked during the bug-fix pass. The first two are a tracked, not-yet-built follow-up (decided 2026-05-30): they are an optimization on the working defer-delete (e2e4c57 already prevents orphans — the only cost today is a wasted re-upload on a retry), and the bake-reorder half is media-service work best done in media-owner context, so it is deferred out of the Core-API pipeline build:

• Orphaned Bunny video / wasteful cleanup (follow-up). Replace the interim "delete the video on any downstream failure" (commit e2e4c57) with: reorder the bake so the cheap Storage-Zone steps (baselines) run before the expensive Bunny upload, and make the render resumable — record the Bunny GUID against the master sha as soon as it's uploaded so a retry reuses the transcode instead of re-uploading. The state machine makes this clean. (media-service bake-pipeline change.)
• Reconciliation sweep (follow-up). A scheduled worker job that restores DB state from Bunny + Storage artifacts (including video_collection_id, which the current ReconcileRendersFromBunny drops because its data source is the Storage Zone, not Bunny Stream's per-video collection) and deletes true orphans (videos with no DB reference past a grace period). This is the guarantee the per-request cleanup can't give. (Core-API cron wiring of the existing reconcile.)
• CFR / short-audio input-gate warnings (commit d96a7b7 logs them) → surfaced to the Console through the asset-validation result.
• Cue-offset drift constraint: if the cue-offset math is touched here, keep drift ≤ current (it's ~tens of ms, cosmetic) — the correct improvement is deriving offsets from the final master. See the cue-offset-drift-parked note.

---

Decisions (settled)

#	Decision	Choice
1	Render trigger	Program publish (sessions final)
2	Program readiness	All-or-nothing v1; prefix-ready later
3	Concurrency control	429 backpressure now; Redis token-bucket at first media scale-out
4	Transcode-complete signal	Bunny webhook (push) into our inbound receiver; poll-before-rebake backstop (asks Bunny the real state) for missed webhooks
5	Worker topology	All API instances poll, coordinated by SKIP LOCKED (no leader election)

---

Implementation status

Phase	Scope	State
1	Durable queue (exercise_renders + transcoding state + lease columns + claim index), exercises.Dispatcher (SKIP-LOCKED claim, concurrent bake, retry/dead-letter, 429 backpressure), repo claim/transition methods	Shipped (cf87abb)
2	Publish-trigger cutover: exercises.Service.EnqueueRender (cache-aware, no media call), sessions enqueue helpers, programs.Publish enqueues, session-save goroutine fan-out retired, retry/re-bake → enqueue, dispatcher started in cmd/api	Shipped (a753402)
3	Bunny transcode-complete webhook (first Cat-D consumer, P52) at /webhooks/bunnystream: status 4 → ready, status 5 → failed; exercise.render_ready / render_failed internal events; stuck-transcoding backstop	Shipped (e096b36)
3-h	Transcode backstop hardening: poll-before-rebake. The stuck-transcoding backstop now polls Bunny (media.VideoStatus → media GET /v1/exercises/videos/{guid}/status → Bunny Stream) and recovers / fails / waits / re-bakes per the real state instead of blind-re-baking on a timer. Fixes the unbounded re-bake loop on a never-arriving webhook (e.g. local dev) + the needless re-upload when a transcode finished but its webhook was dropped. Re-bake (video-missing only) is bounded by MaxAttempts → dead-letter.	Shipped
B	Sessions adopt the program publish lifecycle (draft→published→archived): PublishSession = durable flip + enqueue (not a readiness gate); AttachSession requires a published source; program-publish flips its draft sessions; edit-after-publish guard. Closes the standalone-session render gap (the prerequisite for the patient catalog). Clinic frontend + i18n + OpenAPI.	Shipped (6f5fe93)
4	Honest play readiness: transcoding no longer exposes a manifest in the play payload (play-gate fix); CreateRun enforces ErrSessionPreparing; uniform exercise.render_failed from bake-exhaustion; dead-letter → exercise-owner notify (platform→superadmins, org→content.write managers). Per-session "preparing" badge already shipped + handles transcoding.	Shipped (17afa0e, b495565)
4 (deferred)	Program-level all-or-nothing readiness rollup (per-session preparing suffices; no consumer surface yet); Redis status cache for the preparing poll (→ production, first media scale-out).	Deferred
5	Ops/scale: media autoscale CloudWatch metric + Redis token-bucket (→ production, pinned in production-launch-readiness.md); reconcile reorder + resumable-reuse + scheduled reconciliation sweep (media-side optimization on the working defer-delete)	Deferred / tracked (596e27c pins the prod reminders)

Behaviour change shipped with Phase 2: renders no longer bake during clinic authoring — they start at program publish (decision #1). The clinic render-state poll still works; it reads exercise_renders and shows pending/absent until publish enqueues. A clinic-frontend copy tweak ("videos prepare on publish") is a possible follow-up.

The Phase-1 webhook security note (token-based ?token= shared secret, constant-time compared) and the transcode backstop are implemented; a stronger HMAC scheme stays a documented future item.

Transcode backstop = poll-before-rebake (hardened, see status row 3-h). A render stuck in transcoding past TranscodeTimeout is not blindly re-baked. The backstop asks Bunny the video's real state (via media.VideoStatus → the media service, which owns the Bunny Stream credential) and acts on the answer: finished → recover the existing video to ready (no re-upload — the lost-webhook win), error → terminal failed, still processing → wait (a slow ladder isn't a lost webhook), missing (404) → re-bake — and only this last path re-uploads, bounded by MaxAttempts so a perpetually-vanishing video dead-letters instead of looping. This replaces the original blind-re-bake-on-timer, whose two failure modes were (a) an unbounded re-bake loop in any environment where the inbound webhook can't reach the API (e.g. local dev — Bunny can't POST to localhost), and (b) a needless re-upload + orphaned Bunny video whenever a transcode actually finished but the webhook was merely dropped.

Implementation touchpoints

• Worker: new internal/core/render/ (or fold into exercises) dispatcher modeled on notify/dispatcher.go — claim query, lease, retry/dead-letter, calls media.Compose.
• Queue: reuse exercise_renders; add transcoding to the status CHECK + a claimed_at/lease + attempts column. Editable pre-prod (no new migration file needed).
• Enqueue: the program-publish path inserts/flips pending rows instead of spawning goroutines; retire the EnsureRendersForSession goroutine fan-out.
• Webhook: a Bunny-specific handler on inboundwebhooks/ mapping video.status=4 → render ready (keyed by video_id), with dedup.
• Readiness read: program/session "ready" derivation + Redis status cache for the patient surface.
• Notify: dead-letter → clinic notification (notify primitive).
• Autoscale: API emits the pending-render CloudWatch metric; Fargate target-tracking policy on the media service.
• Reconcile + reorder + resumable folded in per What this absorbs.

---

Open / future

• Prefix-ready program readiness (player holds when the next item isn't ready).
• Pose-tracking timing — if cue timestamps ever drive a measurement window, revisit cue-offset precision (today's drift is cosmetic).
• Bunny webhook security — signature verification on the inbound handler (Cat D framework concern).