Patient Session Player

The patient-side surface that plays a session — a group of exercises with materials, optional cues, and post-session feedback. Source-agnostic: the same player consumes sessions from treatment plans, guided sessions, guided programs, and future source kinds without changes.

Layer 10 — Telerehabilitation

The patient session player is a Layer 10 product surface. It depends on the Layer 1 foundation (auth, RLS, audit) + the exercise composition pipeline (cue manifest). It is consumed by Layer 11 telemetry for engagement + pose ingest.

What this is

A telerehab patient opens the Portal, sees today's queued session, taps "Începe sesiunea", and the player guides them through:

1. Safety reminder — platform-level "Atenție" before every session.
2. Materials checklist — what the patient needs on hand.
3. Pose-tracking opt-in — camera consent + framing check.
4. Exercise loop — single <video> plays each exercise's full-bake MP4; overlays drive off cue timing (rep counter, set indicator, side label, pain pill).
5. Post-session feedback — VAS pain + RPE effort + notes.
6. Done — summary of what happened.

The player owns the playback + interaction surface. It does not own session authoring, treatment-plan structure, exercise catalog browsing, or session scheduling — those live in their respective feature areas.

Source-agnostic premise

A session = a group of exercises. Sessions come from many sources. The player is uniform across all of them:

Source kind	Description	Sequence?	Adherence?
treatment_plan_session	Per-patient, prescribed by a specialist.	Yes — Day X of Y	Yes (with scheduled_to_date for "on track / behind")
guided_session	Library item, standalone, available to all patients.	No	No — engagement only (times_played, last_played_at)
guided_program_session	Sequenced library program (group of guided sessions).	Yes — Day X of Y	Yes
future (challenges, courses, …)	TBD	TBD	TBD

The player consumes a SessionPayload. Source context lives in SessionContext (discriminated union) and is read only by the surfaces outside the kiosk run flow (today-landing, completion summary). This separation is load-bearing: adding a new source kind doesn't change the player.

Type contract

Wire types live at apps/portal/lib/session/types.ts. Backend will mirror these once the real fetch endpoint lands.

SessionPayload

What the player consumes. Strictly agnostic of source.

type SessionPayload = {
  session_id: string;                 // stable per-session-definition id
  name: string;                       // "Mobilizare & detensionare"
  subtitle?: string;                  // "Săptămâna 2 · Ziua 1"
  materials_needed: string[];         // e.g. ["Saltea", "Pernă"]
  estimated_duration_s: number;
  exercises: ExerciseInSession[];     // ordered
};

ExerciseInSession

type ExerciseInSession = {
  id: string;                         // per-run row id
  exercise_id: string;                // FK to exercises catalog
  sequence_order: number;

  name: string;
  short_description: string;
  body_region_label: string;          // from exercise_body_regions

  manifest_url: string;               // cue manifest URL (Bunny Storage Zone)

  mode: "video_only" | "hold" | "reps";  // only "reps" is implemented today
  sets: number;
  reps_per_set?: number;              // mode === "reps"
  hold_seconds?: number;              // mode === "hold" (not yet built)
  side: "none" | "left_then_right" | "right_then_left" | "single";
  rest_between_sets_s: number;
  rest_after_exercise_s: number;

  instructions: ExerciseInstruction[];
  contraindications: ExerciseContraindication[];
  equipment: string[];

  pose_tracking_compatible: boolean;
  prescribed_total_s: number;
};

SessionContext

Discriminated by source kind. Read by surfaces outside the player.

type SessionContext =
  | { kind: "treatment_plan_session"; plan_id; plan_name; sequence_number;
      adherence: { sessions_completed; sessions_scheduled_to_date; sessions_total } }
  | { kind: "guided_session"; times_played; last_played_at }
  | { kind: "guided_program_session"; program_id; program_name; sequence_number;
      adherence: { sessions_completed; sessions_total } };

SessionRun

Per-instance wrapper. run_id is created server-side on Start tap.

type SessionRun = {
  run_id: string;       // returned by POST /v1/session-runs
  session: SessionPayload;
  context: SessionContext;
};

CueManifest

Each exercise points at one. Composer-emitted; CDN-served. See exercise-library/cue-manifest.md for the full spec.

type CueManifest = {
  schema_version: 1;
  exercise: string;
  recipe_hash: string;
  language: string;
  asset_version: number;
  version: string;                    // sha256 prefix of the MP4
  video_id: string;                   // Bunny Stream GUID
  video_url: string;                  // HLS playlist URL
  video_mp4_url: string;              // 720p MP4 HLS-fallback URL (Portal only;
                                      // TV uses SessionPayload-level fields)
  duration_ms: number;
  cues: Cue[];                        // ordered by at_ms
};

type Cue =
  | { at_ms; kind: "intro" | "intro_with_instructions"
              | "rest_between_sets" | "rest_side_switch" | "outro" }
  | { at_ms; kind: "work"; side?: "left" | "right"; set: number; target_reps: number };

Session flow

URL space split into two unambiguous scopes (post 2026-05-21 rework):

• /sessions/[sessionId]/start/* — template scope. [sessionId] is the session template id; run not yet created.
• /runs/[runId]/* — run scope. [runId] is session_runs.id; run was created server-side (phone path: startPhoneRunAction; TV path: pair-claim handler).

/sessions                                (patient picker — normal Portal chrome)
   │                                     lists assignments + in-progress runs
   │
   ▼ Tap "Începe" on an assignment       (or "Continuă" on a run → /runs/{runId})
/sessions/{sessionId}/start              (kiosk shell — dark, full-bleed)
   │
   ▼ redirect
/sessions/{sessionId}/start/safety       "Atenție" reminder (hardcoded copy)
   │
   ▼
/sessions/{sessionId}/start/materials    pre-session checklist
   │
   ▼
/sessions/{sessionId}/start/pose-opt-in  camera consent + "Stai în cadru" framing
   │
   ▼
/sessions/{sessionId}/start/where-to-play   "Pe telefon" / "Pe TV" device picker
   │
   ├─ "Pe telefon" → startPhoneRunAction server action → POST /v1/session-runs →
   │                 redirect to /runs/{runId}/play
   │
   └─ "Pe TV"     → /sessions/{sessionId}/start/pair-tv → patient types code →
                    /v1/session-pairings/{pair_id}/claim creates run →
                    redirect to /runs/{runId}/companion

/runs/{runId}/play                       exercise loop (SessionStage)
   │       │       The conductor's internal currentExerciseIdx is the source
   │       │       of truth; no per-exercise URL. Single page = single mount =
   │       │       refs persist across the whole session.
   │       │
   │       ▼ pain event → PainSheet → continue / skip / end_session
   │       │
   │       ▼ Sari → next exercise
   │
   ▼ last exercise's terminal event arrives at /v1/session-runs/{id}/exercise-event
   ▼ server transitions status → ended_naturally (in same tx as event INSERT)
/runs/{runId}/feedback                   VAS + RPE + notes (post-terminal only)
   │
   ▼ submit
/runs/{runId}/done                       summary → back to /sessions picker

Routes live in apps/portal/app/(kiosk)/sessions/[sessionId]/start/ and apps/portal/app/(kiosk)/runs/[runId]/. Each dynamic segment has ONE unambiguous meaning across all sub-routes.

Resume on re-entry (don't restart from exercise 1)

A run resumes at the first unterminated exercise — GET /v1/session-runs/{id}/payload returns terminated_session_exercise_ids (events of kind completed / skipped / abandoned / failed, scoped to that run) and the conductor starts at firstUnterminatedIdx. The "Continuă" link has always used this.

startPhoneRunAction (actions.ts) now does too: before creating a run it checks active_runs for an in-progress run of the same session and, if found, redirects straight into it instead of calling CreateRun. Without this guard a patient who was knocked out mid-session (network drop) and re-enters via the Start flow rather than "Continuă" would hit CreateRun, which supersedes the live run and starts a fresh one at exercise 1 — forcing them to redo everything (the top confusion reported from production 2026-06). Switching to a different session still supersedes (the guard matches only the same session_id).

Network-stall recovery

A mid-play buffering stall (onWaiting past STALL_THRESHOLD_MS) or a code 2 video error pauses with pauseReason="network". Recovery is active: the conductor re-attempts playback every NETWORK_RETRY_INTERVAL_MS (5s) — a soft play() nudge for a buffer underrun, a full HLS re-attach when the element hit a fatal error — and the browser's playing event still auto-resumes the instant buffering catches up. The stall overlay shows a "Reîncearcă" button (requestResume — resumes in place, but if the run was auto-closed during a long stall it routes to /auto-closed instead of resuming into a dead run; falls through to conductor.resume() when offline) so a patient is never stranded waiting. The message reads off the real signal — truly offline (navigator.onLine === false) vs a weak link — and never the old misleading "connection restored — tap Continuă" (navigator.onLine stays true through a buffering stall). Before this, a stall that the browser didn't spontaneously clear left the patient stuck indefinitely (prod 2026-06-25: one sat 26 min, then ended the session with 0 exercises).

SessionConductor — the playback engine

Single class at apps/portal/lib/session/session-conductor.ts. One conductor per session run. Mounted once at (kiosk)/runs/[runId]/layout.tsx via SessionPlayerProvider; persists across per-step navigations.

Architectural choices

Settled 2026-05-17 after the cue-manifest / audio-bundle / multi-primitive model was abandoned (architecture decision in cue-manifest.md).

• One <video> element, not one per exercise. iOS Safari's autoplay gesture sticks to the element it was granted on; switching elements between exercises produced black screens on iPhone HLS transitions in field testing.
• Single HLS attachment at a time. Smart TVs and weak Androids cap simultaneous decoded streams; mounting N videos hurt them.
• video.currentTime is the master clock. No AudioContext, no Web Audio scheduling, no drift correction.
• Trade-off: can't pre-attach HLS for exercise N+1 while exercise N plays. Mitigation: pre-FETCH manifest N+1 in the background; the between-countdown serves as the HLS attach window for the next MP4.

State machine

   idle ─→ priming ─→ ready ─→ playing ──┬─→ paused (manual/visibility/external/network)
                         │       │       │      │
                         │       │       │      ▼
                         │       │       │   playing
                         │       │       │
                         │       │       ▼
                         │       │    between (inter-exercise countdown)
                         │       │       │
                         │       │       ▼
                         │       └────→ playing (next exercise)
                         │
                         │    end of last exercise
                         ▼
                       done
                         │
                       error (manifest_fetch_failed / media_failed / priming_timeout)

Snapshot shape

The conductor exposes a useSyncExternalStore-compatible subscription to a snapshot:

type SessionPlayerSnapshot = {
  status: SessionPlayerStatus;
  errorReason: ErrorReason | null;
  pauseReason: PauseReason | null;
  currentExerciseIdx: number | null;
  exerciseVideoTime: number;          // seconds
  exerciseVideoDuration: number;
  exercises: ReadonlyArray<{
    status: "pending" | "priming" | "ready" | "playing" | "done" | "failed";
    manifest: CueManifest | null;
    currentCueIdx: number | null;
    currentCue: Cue | null;
  }>;
  betweenSecondsRemaining: number;
  log: ReadonlyArray<{ t: number; msg: string }>;
};

The UI (SessionStage) reads this snapshot via useSyncExternalStore and renders overlays.

Cue model + overlays

Each cue carries timing + kind + (for work cues) set/side/target_reps. The conductor's RAF loop maps video.currentTime to the active cue via binary search, emits snapshot updates on transitions.

Overlays derive from cue + currentTime:

• Rep counter ("3 din 5") — for work cues. Interpolated as floor(time_in_cue / cue_duration × target_reps) + 1. Today this is approximate (the rep video is uniform-tempo by construction); when F10 pose tracking ships, the counter is driven by detected reps instead.
• Set indicator ("Set 1 din 2") — from cue.set + the exercise's sets.
• Side label ("stânga" / "dreapta") — from cue.side.
• Pain pill ("Mă doare") — always visible during playing/paused. Tap pauses the conductor and opens PainSheet.
• Between-exercise countdown — when status is "between", displays betweenSecondsRemaining + "Sari peste pauză" CTA.
• Fullscreen toggle — top-right; toggles document fullscreen.

Smart-TV fallback

Superseded as a standalone surface

The "TV as a degraded standalone player" model below is superseded by Phone–TV Companion Mode. In companion mode, the TV is no longer a degraded surface — it's a presentation slave to the phone, so the missing overlays and absent auto-advance no longer matter (input is on the phone; "next exercise" is commanded by the phone). The iframe rendering branch itself still exists for legacy smart TVs that can't run our <video> reliably; only the standalone-surface framing is retired.

UA detection regex in SessionStage: /web0S|webOS|SmartTV|SMART-TV|Tizen|HbbTV|NetCast/i (case-insensitive, also matches Web0S).

When matched, the stage renders Bunny's hosted iframe player (player.mediadelivery.net/embed/{library}/{video_id}) instead of our <video>. The iframe's compiled player handles the long tail of TV media stack quirks our path can't (codec profile picks, manifest variant selection, MP4 fallback). Trade-offs (legacy framing — in companion mode the first two are no longer relevant):

• No overlays. The iframe is opaque; postMessage round-trips for currentTime add latency. Rep counter, pain pill, between-card are disabled on TV.
• No auto-advance. The conductor is bypassed entirely on TV (it can't reach "ready" without a <video> element). One exercise per TV view; patient navigates between exercises via URL / browser back.
• Inline-styled. All chrome on the smart-TV path uses inline styles (not Tailwind) because older webOS Chromium (108-) can't parse Tailwind v4's oklch() colors.

Pose-tracking opt-in

Before the exercise loop, patient sees a three-state opt-in:

1. intro — explains pose tracking, two CTAs ("Nu acum" / "Da, urmărește-mă").
2. framing — mirrored front-camera feed with "STAI ÎN CADRU" overlay; patient confirms head-to-hip visibility.
3. denied — getUserMedia rejected or unsupported; "Continuă fără urmărire".

Choice persisted in SessionStateProvider.poseTrackingChoice: "opted_in" | "declined" | null.

The opt-in is session-scoped today (mock data). When real backend wiring lands, the choice will persist per (patient, program) so returning patients skip the screen unless they explicitly want to change it.

F10 (when it ships) reads poseTrackingChoice === "opted_in" at exercise start and (re-)initializes the camera + MediaPipe pipeline. The opt-in surface itself is done.

Lives at apps/portal/components/session/pose-opt-in-client.tsx.

Event surface

The player emits events to three distinct endpoint families, each with its own concern, consent flag, and retention.

1. Patient-session ingest — clinical record

POST /v1/session-runs                                → on Start tap
  body: { source_kind, source_id, idempotency_key }
  returns: { run_id }
  Creates the run row, records started_at + pose_tracking_choice.

POST /v1/session-runs/{run_id}/pain                  → on each pain event
  body: PainEvent {
    exercise_id, set_idx, side, seconds_into_set,
    severity: "mild" | "moderate" | "severe",
    action: "continue" | "skip_exercise" | "end_session",
  }
  Appended immediately so we keep events even if the patient bails
  mid-session.

POST /v1/session-runs/{run_id}/exercise-event        → on each milestone
  body: {
    session_exercise_id,
    kind: "started" | "completed" | "skipped"
        | "paused_for_pain" | "resumed",
    video_time_s?: number,
    set_count_completed?: int,
  }
  Patient progress milestones. Clinically-meaningful counterpart to the
  /v1/media/events heartbeat (which is video QoS analytics, not progress).
  Drop point is inferred server-side from "last `started` with no matching
  `completed`" by the auto-close cron — client never fires `dropped`.

POST /v1/session-runs/{run_id}/end-early             → on End Session tap
  body: {}
  Explicit-end terminal write. Server flips status to ended_explicit
  and derives completed / exercises_completed from events. Idempotent.

POST /v1/session-runs/{run_id}/feedback              → on feedback submit
  body: {
    feedback_pain_level_now?: number,
    feedback_perceived_effort?: number,
    feedback_notes?: string,
  }
  Post-terminal-only. 409 if status is still in_progress. Idempotent.
  Status was already flipped (to ended_naturally, ended_explicit, or
  auto_closed) by the time the kiosk reaches this endpoint.

• DB shape: session_runs is state (flat table, mutable until completed_at set). session_pain_events + session_exercise_events are events (range-partitioned monthly per P41).
• Source counters (e.g. patient_treatment_plans.sessions_completed_count) are denormalized caches, updated when a run completes.
• Consent: patient must be authenticated; per-tenant authz (is_patient_at_current_org) enforced.

2. Telemetry — engagement analytics + media QoS

POST /v1/media/events

10-second heartbeat during playback + buffering events + session start/end. Goes to the Layer 11 Telemetry API. Video QoS only — patient progress milestones (started / completed / dropped per exercise) ride the clinical-record family above, not this stream. Full spec in telemetry/media-events.md.

• Lawful basis: Art. 6(1)(f) legitimate interest. No per-run consent step, no UI toggle, no per-purpose flag. Gated by signed-session-token audience + signature. See decisions.md → Why engagement telemetry is legitimate interest, not consent.
• DB shape: media_session_metrics + media_buffering_events (monthly partitioned per P41). One row per session, aggregated server-side from heartbeats.

3. Pose — biometric stream

POST /v1/pose/frames

MediaPipe landmark frames batched as binary float32 + gzip, 1-second batches. Only emitted when poseTrackingChoice === "opted_in". Goes to the Layer 11 Telemetry API.

• Lawful basis: Art. 6(1)(a) explicit consent (Art. 9 special category — biometric). biometric per-purpose consent flag must be active; withdrawal takes effect immediately at the ingest tier (existing Start-flow pose consent screen is unchanged).
• DB shape: pose_session_metrics + pose_rep_metrics (Postgres) + replay blob in S3 at s3://restartix-telemetry/{org_id}/{session_id}.bin.gz.

Why three families, not one

• Different lawful bases. Clinical record (pain, exercise milestones) is authn-gated, Art. 6(1)(b) contract / 9(2)(h) healthcare. Media engagement is Art. 6(1)(f) legitimate interest — no per-purpose opt-in, no per-run UI step (see decisions.md → Why engagement telemetry is legitimate interest, not consent). Pose is Art. 6(1)(a) explicit consent + Art. 9 biometric — biometric per-purpose flag. Mixing them at the endpoint level confuses the lawful-basis story.
• Different retention. Clinical record retention (≥6 years per audit/compliance). Telemetry rollup retention is shorter. Pose replay blobs follow biometric data policy.
• Different downstream consumers. Clinical record drives specialist dashboards (one source of truth for "what happened in this session"). Telemetry drives engagement analytics. Pose drives rep counting + ROM measurement.
• Different scale. Clinical events are sparse (~10-20/session: pain + exercise milestones). Heartbeats are 10s cadence (~115/session). Pose frames are 30 fps batched to 1Hz (~600/session).

Why exercise milestones are clinical, not telemetry

The decision is documented in decisions.md. Short version: per-exercise progress (started / completed / dropped per exercise) determines whether the session was clinically completed, the same category as pain events. Putting milestones in telemetry would force the clinic UI to query two databases for one patient's clinical detail page and conflate clinically-meaningful data with the legitimate-interest playback-QoS stream. The video QoS heartbeat (load time, buffering, watch percentage, dropped frames) stays in telemetry — it's observational, processed under Art. 6(1)(f) legitimate interest, and unrelated to clinical decisions.

Debug overlay

Mounted when the URL has ?debug=1. Inline-styled (renders on browsers that can't parse Tailwind v4's oklch() colors). Shows:

• User agent + navigator.platform
• Smart-TV detection result + iframe URL when applicable
• Conductor status / errorReason / pauseReason
• Current exercise idx, video position, between countdown
• Per-exercise status (p=pending, r=ready, P=playing, d=done, f=failed)
• Last 12 conductor log lines

Component at apps/portal/components/session/debug-overlay.tsx.

What's NOT in scope

• Educational video player — single-video play-through for content like "Understanding your diagnosis", posture explainers. Different surface entirely (no cues, no overlays, no pain logging). Will be a separate component when needed; reuses HLS attach helper + smart-TV detection, nothing else.
• hold exercise mode — isometric ("hold for 30 seconds"). Spec exists in exercise-library/composition.md but the composer doesn't produce hold-mode manifests yet. When it does, the player adds a hold-cue branch (countdown overlay, no rep counter).
• video_only exercise mode — pre-baked single MP4, no composer recipe. Maps to duration_based exercises imported from the legacy platform. Would slot in as a simpler exercise branch (single cue covering the whole video).
• Real-time specialist monitoring — live view of patient executing a session. Requires WebSocket / SSE protocol (not 10-second heartbeat HTTP). Future scope.
• Offline playback — service-worker video caching + local event queue + sync-on-reconnect. The existing per-session-manifest URL scheme is compatible (content-addressed, immutable), but no offline runtime today.

File map

apps/portal/
├── app/
│   ├── (patient)/sessions/page.tsx             picker (assignments + active runs)
│   └── (kiosk)/
│       ├── layout.tsx                          dark kiosk shell + Clerk gate + SessionStateProvider
│       ├── sessions/[sessionId]/start/         template-scope prep
│       │   ├── page.tsx                        redirects to safety
│       │   ├── safety/page.tsx                 platform-level reminder
│       │   ├── materials/page.tsx              pre-session checklist
│       │   ├── pose-opt-in/page.tsx            opt-in route
│       │   ├── where-to-play/page.tsx          "Pe telefon" / "Pe TV" picker
│       │   └── pair-tv/page.tsx                TV-branch pairing entry
│       └── runs/[runId]/                       run-scope (run already exists)
│           ├── layout.tsx                      fetches /payload, mounts SessionPlayerProvider + SessionRunBridge
│           ├── page.tsx                        redirects to exercise/1
│           ├── exercise/[step]/page.tsx        renders SessionStage
│           ├── companion/page.tsx              phone-controller UI while TV plays
│           ├── feedback/page.tsx               VAS + RPE + notes
│           └── done/page.tsx                   summary
├── components/session/
│   ├── session-player-provider.tsx             React context wrapping conductor
│   ├── session-run-bridge.tsx                  binds run handle into SessionStateProvider
│   ├── session-stage.tsx                       overlays (smart-TV branch is dead code post-companion)
│   ├── session-state-provider.tsx              pain events + feedback + pose choice + run handle
│   ├── start-on-phone-button.tsx               fires startPhoneRunAction on click
│   ├── pair-tv-client.tsx                      pairing code entry + claim
│   ├── companion-client.tsx                    SSE consumer + command publisher
│   ├── pain-sheet.tsx                          "Mă doare" action sheet
│   ├── pose-opt-in-client.tsx                  three-state opt-in
│   └── simple-markdown.tsx                     trivial md renderer for safety
└── lib/session/
    ├── session-conductor.ts                    the playback engine
    ├── fetch.ts                                fetchAssignedSessions + fetchRunPayload
    ├── actions.ts                              startPhoneRunAction + ingest wrappers
    └── types.ts                                wire types

Related

• Exercise library — composition — how the full-bake MP4 is built.
• Exercise library — cue manifest — the metadata sidecar the player consumes.
• Telemetry — media events — engagement event spec (10-second heartbeat).
• Programs & Protocols — the shipped prescribe / enroll model. (The old treatment-plans/patient-session.md description was retired on 2026-08-02; this doc is authoritative for the player surface.)