Cue Manifest

The patient client's playback contract: one full-bake MP4 per (exercise × recipe × language × intro_variant × seed) plus a metadata-only cue manifest that drives overlays.

Architecture history

This document used to describe a different model — an audio bundle + N video primitives + per-cue swap orchestrated by Web Audio. After cross-device testing in May 2026 (iPhone SE stuck at 0/0, iPhone Chrome stuck at 1/2, LG webOS smart TV stuck at 1/2, Galaxy A13 frame stutter from drift correction), that model was abandoned. The composer now emits a single MP4 per render — exactly what composition.md describes — plus a small JSON manifest. The patient client plays the MP4 in one <video> element and reads video.currentTime against the cue list to drive overlays. See the git history of this file for the previous spec. (Earlier prose called this the "hybrid" manifest because of the v1→v2 transition; the qualifier is no longer load-bearing — every render today is full-bake-with-cues.)

What ships per render

Composer produces TWO artifacts per (exercise × recipe × language × intro_variant × seed):

1. Full-bake MP4 — same artifact composition.md describes: intro → set 1 → pause → set 2 → outro, audio + video muxed, uploaded to Bunny Stream.
2. Cue manifest (this doc) — a metadata-only JSON describing where each cue starts in the MP4. ~1 KB. Uploaded to Bunny Storage Zone at platform/exercises/{exercise}/cues/{stem}.{version}.json (F9.2 Phase 1 tier prefix; see programs-and-assignments → Storage). Versioned by sha256 of the MP4 so each render is immutable.

The patient client fetches the manifest, plays manifest.video_url (HLS) in a single <video> element, and runs a RAF loop that maps video.currentTime to the active cue. Overlays (rep counter "3 din 5", set indicator, side label, "Mă doare" pain context, between-exercise countdown) read from that mapping.

Asset model

Identical to composition.md — same assets/{slug}/ layout, same manifest.json asset contract, same variant naming. The cue-manifest emitter adds one feature on top of the full-bake: when a recipe sets rest_between_sets_s, bakeFraming pads the pause segment with freeze-frame video + silence to reach the configured rest length (FLOOR semantics — never shorter than the natural pause duration).

Cue manifest schema

The output manifest.json (the render manifest, not the asset one) is the contract the patient client consumes:

type CueKind =
  | "intro"
  | "intro_with_instructions"
  | "work"
  | "rest_between_sets"    // same exercise, between same-side sets
  | "rest_side_switch"     // same exercise, L↔R boundary
  | "outro";

type Cue =
  | { at_ms: number; kind: "intro" }
  | { at_ms: number; kind: "intro_with_instructions" }
  | {
      at_ms: number;
      kind: "work";
      side?: "left" | "right";  // omitted for unilateral exercises
      set: number;              // 1-based
      target_reps: number;
    }
  | { at_ms: number; kind: "rest_between_sets" }
  | { at_ms: number; kind: "rest_side_switch" }
  | { at_ms: number; kind: "outro" };

type CueManifest = {
  schema_version: 1;
  exercise: string;
  recipe_hash: string;
  language: string;
  asset_version: number;

  /** Content-addressed version — sha256(video.mp4) prefix.
   *  Same MP4 content → same version → same URL. */
  version: string;

  /** Bunny Stream GUID of the full-bake MP4. */
  video_id: string;
  /** HLS playlist URL. */
  video_url: string;
  /** Direct 720p MP4 URL used by the Portal session-conductor as the
   *  HLS fallback in three real failure paths (hls.js not supported,
   *  fatal HLS error, dynamic-import failure). NOT consumed by the
   *  smart-TV path — TV reconstructs its own URL from SessionPayload
   *  (`video_cdn_hostname` + `video_id`). */
  video_mp4_url: string;
  /** Total MP4 duration in ms (sum of segment durations). */
  duration_ms: number;

  /** Cue timeline. Each cue marks the START of its segment in the MP4;
   *  the next cue's at_ms is the segment's END. The last cue's segment
   *  ends at duration_ms. Cues are sorted by at_ms. */
  cues: Cue[];
};

Cue boundaries align exactly with MP4 segment transitions because the emitter derives them from probed sub-bake durations — not from a separate timeline that could drift relative to the rendered video.

Schema version history

The current schema is version: 1, reset alongside the Phase 1 refactor (rename hybridManifest → cueManifest, drop hybrid-era field naming). The pre-refactor schema (then labelled version: 2, with recipe/seed/intro_variant/mocked_slots fields and the platform/exercise-manifests/... Bunny path) is orphan — wiped from Bunny Storage as part of the same refactor. The much earlier "audio bundle + per-primitive video refs" model (originally version: 1) is documented above in the Architecture history note and is also orphan.

Render flow

When the TypeScript composer at experiments/exercise-composer/ receives a recipe with --emit hybrid (default):

1. Load asset bundle — loadExercise(assetsRoot, exercise, language) — same as full-bake.
2. Validate recipe — reps must be a multiple of reps_per_video_block in [reps_per_video_block, counts_per_audio_master]; sides exist; language declared.
3. Pick variants (seeded) — pickVariantPair enforces matched (video, vo) takes by variant index. Same picker as full-bake.
4. Bake framing + sets — composeRecipe produces intro / pause / outro / per-set sub-bakes, then concatenates them with -c:v libx264 -preset veryfast -crf 20 (re-encode on concat is mandatory — -c copy produces non-monotonic DTS that breaks Safari and Bunny's HLS transcoder; see feedback_composer_concat_reencode memory or git history for context). Returns the segment timeline alongside picks.
5. Hash the MP4 — sha256 prefix becomes the manifest version.
6. Walk the timeline → cues — each segment becomes one cue; at_ms is the cumulative offset; work cues carry set, side, target_reps.
7. Write manifest locally as manifest.local.json with video_id: "pending:mp4".

The uploader (upload-hybrid.ts) then: 8. Upload MP4 to Bunny Stream into a per-exercise collection. SHA-256 content cache skips re-uploads when the same MP4 was already pushed. 9. Rewrite manifest with the real video_id, video_url (HLS), and video_mp4_url (720p MP4). 10. Write versioned local copy at {stem}.{version}.json. 11. Upload manifest JSON to Bunny Storage Zone at platform/exercises/{exercise}/cues/{stem}.{version}.json. Prints the public URL.

The public manifest URL is what gets pasted into the patient client (or, in time, served by API as part of GET /v1/patients/{id}/treatment-plans/today).

Patient client integration

apps/portal/lib/session/full-bake-conductor.ts is the runtime conductor:

• One <video> element drives the whole session. Its src swaps between exercises; user-gesture stickiness stays on the same element.
• HLS attach: native HLS on iOS Safari (via canPlayType("application/vnd.apple.mpegurl")); hls.js everywhere else with a 480p cap on small viewports.
• RAF loop reads video.currentTime, binary-searches the cue list, emits snapshot updates when the active cue changes.
• Inter-exercise transition: on the active video's ended event, attach HLS for the next exercise's MP4, start a countdown using exercise.rest_after_exercise_s, then play.
• Manifest pre-fetch for exercise N+1 happens in the background during exercise N's playback. Cheap (~1 KB JSON); makes transitions instant.
• Pause/resume handles visibility, manual, and network-stall reasons. Wake Lock acquired during playback.

Smart TV path: when navigator.userAgent matches webOS|Tizen|SmartTV|SMART-TV|HbbTV|NetCast, the stage renders Bunny's hosted iframe player (player.mediadelivery.net/embed/{library}/{guid}) instead of our <video>. Trade-off: no overlays over the iframe. Accepted because TV is a v0 viewing surface — phone/tablet remains the primary patient device.

What's NOT in the manifest

• User-visible strings ("Pauză", "Set 1 din 3", "Partea stângă") — derived by the patient client at runtime from kind + cue fields + locale (next-intl). Manifest is pure data + identifiers; translation is downstream.
• Per-rep cues — explicit kind: "rep" markers per individual rep. Deferred until pose tracking ships. The rep count is interpolated from video.currentTime within a work cue (the rep video is uniform-tempo by construction); explicit per-rep markers add manifest size without runtime value today.
• Clinical metadata — exercise name, body region, contraindications, difficulty live in the platform DB, not the manifest. Same separation as composition.md.

Why this won

Cue-manifest v1 won on storage (~22 GB / catalog vs ~180 GB for full-bake) and authoring economics. But it lost on reliability across the device long tail. The hybrid keeps the cue timeline value (rep counter, side label, pain context) and trades the storage savings for:

• Single <video> element instead of N + Web Audio sync
• One clock (video.currentTime) — no drift correction needed
• One HLS attach at a time — fits TV decoder slot caps
• One user-gesture sticking to one element — iOS gesture inheritance never breaks
• Bunny does the HLS transcoding from the MP4; the long tail of TV/codec quirks lives in their player, not ours

Storage cost at ~$0.005/GB-month: ~$1/month per language for the full catalog. Negligible.