Media Events Specification

Layer 2 feature, not yet implemented. See index.md for the architecture and rationale; api.md for the request envelope and auth.

This document defines all video QoS events that the Patient Portal sends to Telemetry API via POST /v1/media/events. Server-side these events drive in-memory aggregation per run; on receipt of the terminal session_end event the aggregator writes one media_session_metrics row + N media_buffering_events rows to the telemetry service's own Postgres — there is no callback to API.

Scope reminder — media QoS only. Per-exercise lifecycle (started / completed / skipped / paused_for_pain / resumed) is clinical record, not telemetry. Those events ride API's POST /v1/session-runs/{run_id}/exercise-event and live in session_exercise_events. The clinic UI reads everything clinical from API alone. See sessions for that schema. The taxonomy below is deliberately narrow: load events, buffering, watch-threshold milestones, presence, dropped frames, session-end. Nothing about exercise progress.

Media kind. The media_type field on each event is the on-the-wire discriminator. "video" (default for backward compatibility — pre-audio clients may omit the field) or "audio". The aggregator captures the kind from the first event for a given media_id and writes it to media_session_metrics.media_kind. The audio player MUST NOT emit quality_change or dropped_frames — those are video-only (ABR + decoder-level signals that don't apply to a static audio file). All other event kinds (video_played / video_paused / buffering_* / watch_pct_* / page_* / session_end) are kind-agnostic and apply to both. The legacy event names keep video_* prefixes for backward compatibility with the existing portal heartbeat client — they semantically mean "media transport," not "video specifically."

session_id semantic. The session_id field on every event payload below is the session_runs.id from the clinical record — the same identifier the player POSTs to POST /v1/session-runs/{run_id}/* on API, and the same identifier carried in the signed-session-token's run_id claim. One run = one signed token = one session_id value reused across every event for the whole multi-exercise playthrough. Per-exercise scope is carried in the media_id field, not in session_id. The wire field name stays session_id (it's the natural name from the player's perspective); the join key to the clinical record is unambiguous because the value space is the same.

Legal basis (no consent gate). Media QoS telemetry is processed under GDPR Art. 6(1)(f) legitimate interest — playback quality is operational metadata necessary to deliver and improve the service, not a separately consented purpose. The signed token carries no consents.engagement claim and Telemetry never returns 403 on this route; audience + signature are the full gate. Pose data, which is Art. 9 biometric special-category data, is handled separately and stays consent-gated via consents.pose. See index.md → Consent.

Why this taxonomy, not a heartbeat

Earlier versions of this spec defined a per-10-second heartbeat event carrying running totals (position, buffer count, bitrate, dropped frames, etc.). That model is rejected: it's expensive (115 events / session / patient at scale), most fields are redundant frame-to-frame, and the running totals belong in the server-side aggregator rather than on the wire.

Instead, the client fires sparse events on state transitions and thresholds, and the Telemetry service maintains per-run aggregates in memory until session_end. Trade-off: a server-side process crash mid-run loses that run's media metrics (the run itself completes — clinical record is on API and unaffected). Acceptable for QoS analytics; not acceptable for clinical signals (which is exactly why progress milestones moved off this surface).

Event lifecycle

run starts (server mints JWT)
  │
  ├── video_played          ← first frame rendered for the first exercise
  │     │
  │     ├── buffering_start ← video stalls
  │     ├── buffering_end   ← video resumes
  │     ├── quality_change  ← ABR switches resolution/bitrate
  │     │
  │     ├── watch_pct_25    ← per-exercise watch thresholds
  │     ├── watch_pct_50
  │     ├── watch_pct_75
  │     ├── watch_pct_95
  │     │
  │     ├── video_paused    ← system pause (pain dialog, tab hidden, etc.)
  │     ├── video_played    ← resume after pause OR start of next exercise
  │     │
  │     ├── page_hidden     ← tab/window backgrounded
  │     ├── page_visible    ← tab/window foregrounded
  │     │
  │     └── dropped_frames  ← periodic (delta ≥ threshold)
  │
  └── session_end           ← terminal event; triggers aggregator flush

video_played / video_paused can fire multiple times per run (one pair per exercise plus any in-exercise pauses). watch_pct_* events are per-exercise, scoped by media_id. buffering_start / buffering_end are pairs; the aggregator joins them on (session_id, buffering_start_position_seconds).

Event types

video_played

Fired when the player begins (or resumes) rendering frames. Both "started this exercise" and "resumed after pause" fire video_played; disambiguate via data.is_resume.

{
  "event": "video_played",
  "session_id": "uuid",
  "media_id": "exercise-uuid",
  "media_type": "video",
  "timestamp": "2026-05-07T10:00:00.000Z",
  "data": {
    "position_seconds": 0.0,
    "is_resume": false,
    "total_duration_seconds": 120.5,
    "video_load_time_ms": 1200,
    "ttfb_ms": 340,
    "cdn_response_time_ms": 280,
    "connection_type": "wifi",
    "effective_bandwidth": 12.5,
    "rtt_ms": 45,
    "initial_bitrate": 2500000,
    "initial_resolution": "720p"
  }
}

Load-perf fields (video_load_time_ms, ttfb_ms, cdn_response_time_ms, initial network state) are only meaningful on the first video_played for a given media_id — they describe the cold-start cost. Resumes carry is_resume=true and may omit those fields.

video_paused

Fired when the player suspends playback. Distinguishable causes:

{
  "event": "video_paused",
  "session_id": "uuid",
  "media_id": "exercise-uuid",
  "timestamp": "2026-05-07T10:01:15.000Z",
  "data": {
    "position_seconds": 45.2,
    "reason": "user" | "system" | "pain_dialog" | "tab_hidden"
  }
}

The reason field is best-effort metadata; the aggregator counts pause occurrences without dispatching on it.

buffering_start

Fired when the video stalls (buffer underrun). The aggregator opens a buffering interval keyed by (session_id, position_seconds).

{
  "event": "buffering_start",
  "session_id": "uuid",
  "media_id": "exercise-uuid",
  "timestamp": "2026-05-07T10:00:23.100Z",
  "data": {
    "position_seconds": 23.1,
    "bitrate_before": 2500000,
    "resolution_before": "720p",
    "cdn_response_time_ms": 1200,
    "connection_type": "wifi",
    "effective_bandwidth": 3.2
  }
}

buffering_end

Fired when playback resumes after a stall. The aggregator closes the matching interval and emits a media_buffering_events row at flush time.

{
  "event": "buffering_end",
  "session_id": "uuid",
  "media_id": "exercise-uuid",
  "timestamp": "2026-05-07T10:00:25.900Z",
  "data": {
    "position_seconds": 23.1,
    "duration_ms": 2800,
    "bitrate_after": 1500000,
    "resolution_after": "480p",
    "recovered": true
  }
}

quality_change

Fired when ABR switches video quality.

{
  "event": "quality_change",
  "session_id": "uuid",
  "media_id": "exercise-uuid",
  "timestamp": "2026-05-07T10:00:26.000Z",
  "data": {
    "position_seconds": 23.1,
    "from_bitrate": 2500000,
    "to_bitrate": 1500000,
    "from_resolution": "720p",
    "to_resolution": "480p",
    "reason": "bandwidth_decrease"
  }
}

Reasons: bandwidth_decrease, bandwidth_increase, buffer_low, user_manual, initial. The aggregator increments bitrate_switches and resolution_switches counters per run.

watch_pct_25 / watch_pct_50 / watch_pct_75 / watch_pct_95

Per-exercise watch-threshold milestones. Each fires at most once per media_id per run. The 95% threshold is the closest the spec gets to "exercise completed"; it's not completion-as-a-clinical-signal — that's session_exercise_events.kind = "completed" on API. The 95% bound exists for engagement analytics ("of patients who started exercise X, what % watched ≥95%?").

{
  "event": "watch_pct_75",
  "session_id": "uuid",
  "media_id": "exercise-uuid",
  "timestamp": "2026-05-07T10:01:30.000Z",
  "data": {
    "position_seconds": 90.4,
    "elapsed_real_seconds": 95.2
  }
}

position_seconds is in-media time; elapsed_real_seconds is wall-clock time since first video_played for the same media_id. The two diverge when there are pauses / buffering.

page_visible / page_hidden

Presence signal from Page Visibility API (document.hidden transitions). The aggregator uses these to attribute "background time" in the final watch-percentage calculation — a video that "played" while the tab was hidden doesn't count.

{
  "event": "page_hidden",
  "session_id": "uuid",
  "timestamp": "2026-05-07T10:01:00.000Z"
}

Same shape for page_visible. No media_id — these are run-scoped, not per-exercise.

dropped_frames

Periodic snapshot of HTMLVideoElement.getVideoPlaybackQuality().droppedVideoFrames. Client fires when the delta since last report exceeds a threshold (default: 5 new drops). The aggregator records the maximum cumulative count seen per run.

{
  "event": "dropped_frames",
  "session_id": "uuid",
  "media_id": "exercise-uuid",
  "timestamp": "2026-05-07T10:01:00.000Z",
  "data": {
    "cumulative_dropped": 12,
    "cumulative_total": 1800
  }
}

Browser support: Chrome / Edge / Firefox. Safari returns NaN — the client just doesn't emit on Safari (lost signal is acceptable; it's a QoS metric, not clinical).

session_end

Terminal event. Triggers the in-memory aggregator's flush: one media_session_metrics row + N media_buffering_events rows written to the telemetry-owned PG.

{
  "event": "session_end",
  "session_id": "uuid",
  "timestamp": "2026-05-07T10:30:00.000Z",
  "data": {
    "client_status": "completed" | "ended_early" | "tab_closed"
  }
}

client_status is a hint — the server doesn't reject on it; the aggregator stores it for diagnostic context. A run that exits without firing session_end (network loss, browser kill, OS force-quit, dead battery) is finalized by the silence-sweep reaper with client_status = "auto_closed_unknown". The reaper ticks every minute and finalizes any aggregator that hasn't received an event in ≥10 minutes of wall-clock time — partial run state (watched seconds, watch thresholds reached, buffering events, etc.) is preserved exactly as it was at the last received event. Server clock, not client clock: client timestamps can be skewed; the operator wants "have we heard from this run recently."

The event is idempotent: a duplicate session_end after flush is dropped (the aggregator records "flushed" per run, indexed by session_id).

Frontend implementation notes

• page_visible / page_hidden — use document.visibilitychange event listener.
• buffering_start / buffering_end — bind to <video> waiting and playing events respectively. Track the time delta client-side and emit on resume.
• watch_pct_* — track cumulative watched-time (excluding pauses + background-tab time) per media_id. Fire when crossing each threshold.
• dropped_frames — sample getVideoPlaybackQuality() every ~10s; emit when delta crosses threshold. Track cumulative count locally to dedupe.
• quality_change — bind to HLS.js's Hls.Events.LEVEL_SWITCHED or equivalent.
• session_end — fire on the run-completion paths (feedback submit, ended-early, browser-unload via pagehide).

Browser-API compatibility

Metric	API	Support
TTFB / load time	PerformanceResourceTiming	All modern browsers
Connection type	navigator.connection.effectiveType	Chrome / Edge (NOT Safari/Firefox)
Bandwidth estimate	navigator.connection.downlink	Chrome / Edge
RTT	navigator.connection.rtt	Chrome / Edge
Dropped frames	getVideoPlaybackQuality()	Chrome / Edge / Firefox (NOT Safari)
Page visibility	document.visibilityState	All modern browsers

Fallback strategy: when an API is unavailable, omit the field. The server's aggregator handles missing fields (they reduce signal, not validity).

Aggregation contract (server side)

The Telemetry service maintains one in-memory aggregator per active session_id. On each event:

• video_played (first per media_id): record load-perf fields; mark exercise as started.
• video_played / video_paused: maintain watched-time accumulator per media_id.
• buffering_start: open interval keyed by (session_id, position_seconds); start clock.
• buffering_end: close matching interval; emit row at flush.
• quality_change: increment bitrate_switches / resolution_switches; update peak/avg bitrate.
• watch_pct_*: record threshold timestamp per media_id (for funnel queries).
• page_hidden / page_visible: maintain hidden-time accumulator; subtract from watched-time at flush.
• dropped_frames: track max cumulative count seen.
• session_end: flush.

On session_end, write one media_session_metrics row with: per-run totals (watch time, completion %, replay count, buffering count + ms, switches, dropped frames, peak/avg bitrate, peak resolution, connection metadata), client_status, media_kind, flushed_at. And N media_buffering_events rows (one per buffering interval, monthly-partitioned per P41).

Replay count tracks non-resume video_played events for a given media_id after the first. Each non-resume replay (user pressed restart, auto-replay loop) increments by one; resumes (data.is_resume = true) and the initial play do not. Surfaces in F9's per-exercise breakdown as "retries per exercise."

Specialist & analytics queries (post-launch)

All reads through API. Telemetry PG is read-only-via-Core-API; no direct browser access. These queries become a real surface when analytics dashboards ship (post-launch, not MVP). Tenant scoping is enforced at the API endpoint via organization_id filtering.

"Why is this patient's video not loading?"

SELECT
    session_id, media_id, started_at,
    video_load_time_ms, ttfb_ms,
    buffering_count, total_buffering_duration_ms,
    avg_bitrate, peak_resolution,
    connection_type
FROM media_session_metrics
WHERE patient_id = $1
  AND organization_id = $2  -- enforced by API endpoint
  AND started_at > now() - INTERVAL '7 days'
ORDER BY started_at DESC
LIMIT 20;

"Which exercises buffer the most for this clinic?"

SELECT
    media_id,
    count(*) AS buffer_events,
    avg(duration_ms) AS avg_buffer_duration,
    percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_buffer_duration
FROM media_buffering_events
WHERE organization_id = $1
  AND started_at > now() - INTERVAL '7 days'
GROUP BY media_id
ORDER BY buffer_events DESC
LIMIT 20;

Materialized views (media_session_weekly_agg per clinic) refresh nightly to serve cohort dashboards without scanning millions of rows on every page load. Cross-tenant queries are not exposed today — readers are clinic-scoped per the processor rule. If/when Tier 3 fires (cross-tenant analytical workload), those queries move to ClickHouse via the swap-point interfaces — see index.md → Scaling roadmap.

What this taxonomy intentionally excludes

• seek events. The kiosk player doesn't expose scrub. If a future surface needs it, add it then.
• error events. Playback errors go to Sentry / observability, not engagement analytics. A failed playback already shows up as a started exercise with no watch_pct_* events — that's signal enough for "this exercise is broken for this patient."
• Per-10s heartbeats. Replaced by sparse events + server-side aggregation. See rationale at top of doc.
• Per-exercise lifecycle events (exercise_started, exercise_completed, etc.). Those are clinical record on session_exercise_events (API), not telemetry.
• session_start. Run start is clinical (POST /v1/session-runs on API). No telemetry counterpart needed.