Data Flow

This document sits between UI Design (which defines what the user sees) and Contracts (which formalise the API shapes). For each screen and interaction, it draws what calls what: which service initiates, which responds, what data crosses each boundary. Read each arrow two ways: it is a contract boundary (what shape the data takes) and a sequencing constraint (downstream cannot build until the upstream contract is published).

System Context

---

Part 1: Live Session

Flows that run automatically during an active recording session — audio capture, ML insights, and system resilience.

Flow 1: Start Recording

The user opens the New Meeting ▾ dropdown and selects Start Live Recording. After the browser grants microphone access, the app creates a meeting and initiates the recording session across all three services.

Pre-conditions: The client checks for an active recording session before enabling the button. If one exists, "Start Live Recording" is disabled with a tooltip. Core enforces this server-side — if StartRecordingCommand arrives while a session is already running, it responds with RecordingErrorEvent (session_conflict). If the browser denies microphone access, the app shows a blocking modal with a link to browser audio settings — no data flow occurs.

Flow 2: Live Audio → Transcription (Lowest Latency Path)

Audio flows from the browser microphone through Core and ML to AssemblyAI. Transcript segments return via the ML WebSocket — the same long-lived connection Core uses to send audio. Audio chunks flow upstream as binary frames, segments and insights flow downstream as CloudEvents text frames.

OPFS shadow buffer: Every audio chunk is simultaneously written to an always-on shadow buffer maintained by a dedicated Web Worker using the Origin Private File System (OPFS) createSyncAccessHandle() API. Each chunk carries a monotonically incrementing sequence number assigned in the browser. This buffer runs unconditionally — it captures audio regardless of Core or GCS connectivity. It is cleared only after Core confirms all chunks are safely in GCS (see Flow 16 and Flow 9).

Chunk-based GCS writes: Instead of a single streaming write to one file, each audio chunk is stored as a separate GCS object keyed by sequence number: meetings/{id}/chunks/{seq:08d}.webm. WebM encodes its EBML header in the first chunk; subsequent chunks contain raw Cluster data. This structure enables gap recovery — any chunk missed due to a connectivity failure can be backfilled from OPFS by sequence number. At session end, Core composes the chunk objects into the final audio.webm (see Flow 9).

Core streams segments directly to the client via WebSocket for minimum latency, and persists them to the database asynchronously in the background. The app distinguishes interim segments from final segments and replaces them in-place when the final version arrives — no layout shift.

Flow 3: Live Insights Pipeline

Talking points and tasks are extracted by the same LLM query, batched together as a single structured output call. This avoids redundant token spending — the transcript context is loaded once into the prompt cache and both extraction tasks run against it. All insights stream back through the ML WebSocket as CloudEvents text frames, following the dual-write pattern: Core fans out to the browser via WebSocket for latency, and persists to DB asynchronously for durability.

Context management: ML maintains a rolling transcript buffer in memory, appending each finalised segment as it arrives. The full buffer is included in the LLM prompt as cached context. The prompt is always ordered as: [system instructions] [schema] [anchor segments] [recent window] [latest segment]. When the context budget is exceeded, segments are dropped from the oldest non-anchored position — the boundary between the anchor and the recent window — never from the front. Removing from the front would change the content immediately after the static cached prefix, invalidating every transcript token in the cache. Because the anchor only ever grows, the cached region ([system] + [schema] + [anchor]) expands over the course of the meeting and is never invalidated by trimming.

Flow 3a: Live Talking Points & Tasks (Batched — Per Finalised Segment)

On every finalised transcript segment, ML sends the full rolling transcript buffer to the LLM requesting both the latest talking point and any new tasks in a single structured output call. The LLM returns both in one response. Talking points update immediately, and tasks are extracted opportunistically from the same call.

LLM-native task deduplication: The current list of extracted tasks is appended to the dynamic suffix of each prompt. The LLM is instructed to return only tasks that are genuinely new — not already represented in the existing list. This delegates deduplication to the model, which handles paraphrase and semantic overlap naturally without a separate post-processing step.

The prompt is structured for OpenAI prompt caching: [system instructions] [output schema] [anchor segments] forms the stable cached prefix that grows as the session progresses. [recent window] [latest segment] [existing task list] is the dynamic suffix appended on each call. Placing the task list in the dynamic suffix (not the cached prefix) keeps the cache hit rate high — the stable cached region is never invalidated by task accumulation.

Flow 3b: Live Speaker Identification (Per Diarised Speaker)

AssemblyAI's transcript segments arrive pre-diarised — each segment carries a speaker_label (e.g. speaker_1, speaker_2). ML's job is to resolve each speaker_label to a known Person by matching voice embeddings against enrolled profiles.

Every segment gets a voice embedding. Regardless of whether the speaker has been identified, ML extracts a voice embedding from the segment's audio and stores it on the segment. This happens unconditionally — embeddings are required for post-meeting RAG and future retrieval, not only for speaker matching.

Matching strategy: Speaker matching runs separately, gated on the per-session map speaker_label → { status, person_id?, attempts }. This map lives in ML's memory for the hot path but is mirrored to a meeting_speaker_states table in Postgres on meaningful transitions. On session start — and on reconnect after a pod restart — Core pushes the current speaker states and voice profiles to ML via StreamStartEvent, so ML reconstructs its in-memory map without needing a pull endpoint.

---

Part 2: User Mutations

Flows initiated by the user during or after a recording session. All follow the Optimistic Mutation with Echo-Suppressed Streaming pattern: the client updates local state immediately, sends the mutation via REST, and suppresses the returning WebSocket echo.

Flow 4: Notes Auto-Save

The Private Notes scratchpad is the primary surface of the live recording view — it occupies the left column. Notes auto-save continuously with no explicit save button. The app debounces keystrokes and patches the meeting's notes field.

Flow 5: User Creates Task

The user's task is written via REST (not the streaming path) since it's a user-initiated mutation. Tasks have a description (required), assignee (optional), and due date (optional).

Flow 6: Task Mutations (Full CRUD)

Flow 5 covers task creation. The UI design specifies a full set of task mutations: edit, delete, toggle completion, nest under other tasks, assign a person, and set a due date. Editing a system-generated task converts it to user-owned.

Flow 7: User Labels Speaker as Person

When a user identifies "Speaker A" as a known Person (by clicking the speaker label on any transcript segment), the system reassigns all segments from that speaker and records the mapping in meeting_speaker_states as a manual override so that ML respects it immediately on any pod recovery. Voice profile enrichment from the session's embeddings happens during post-meeting processing, not here.

Flow 8: Create New Person During Speaker Labelling

Flow 7 assumes the person already exists. The UI design says the user can "reassign to a known person or add a new one." When creating a new person, the UI handles this as two sequential operations: first create the person, then assign them to the speaker label using the same endpoint as Flow 7. The speaker-labels endpoint always receives an existing person reference — it has no knowledge of whether that person was just created or long-established. Voice profile enrichment happens during post-meeting processing.

---

Part 3: Session End & Post-Meeting

Flows triggered when a recording stops (user-initiated or auto) and the subsequent background processing that upgrades all artefacts to final quality.

Flow 9: Stop Recording

The user presses Stop Recording (or the system auto-stops at the duration limit). The stop sequence is strictly ordered: drain ML first (to flush final transcript segments), then collect any remaining OPFS gaps, then compose the final audio file, then trigger post-meeting processing. This ordering ensures no audio is lost and the composed file includes all chunks.

Flow 10: Duration Warning & Auto-Stop

A configurable maximum recording duration (default: 4 hours) is enforced server-side. Core sends a warning at T-10 minutes and auto-stops at the limit. The auto-stop triggers the same post-meeting pipeline as a user-initiated stop.

Flow 11: Post-Meeting Processing (Automatic, via Pub/Sub)

Post-meeting processing runs automatically via the shared TranscriptionJob Pub/Sub worker. For live recordings, the job is published flagging tasks to be skipped, preserving tasks captured during the session.

The worker:

1. Batch-transcribes the full audio from GCS (higher accuracy)
2. Replaces transcript segments with the improved results
3. Generates headline, summary, topics, and finalises talking points
4. Extracts tasks (file upload flow only)

The Meeting Summary page shows a subtle progress indicator during re-processing and updates each artefact in real time as it completes.

Flow 12: Transcription Processing Lifecycle

The transcriptions table tracks processing status through a defined state machine: pending → transcribing → synthesizing → completed (or failed). The client uses this to show re-processing progress on the Meeting Summary page.

Flow 13: Audio Playback (Signed URL Direct to GCS)

Core generates a short-lived signed URL. The client streams audio directly from Cloud Storage, with standard HTTP range requests for seeking. The audio player appears on the Transcript tab of the Meeting Summary page. If the audio file is still being processed, the endpoint returns 404 and the client retries with exponential backoff.

Flow 14: Degraded Mode — Layered Resilience

The system has five independent failure domains. Each degrades gracefully — the OPFS shadow buffer ensures audio is never lost regardless of what fails on the backend. Core detects failures and notifies the client via RecordingErrorEvent with a specific error code. Recovery is automatic: when the broken link restores, Core sends a recovery signal and the client clears the warning. Gaps in GCS are filled via the gap upload sequence (Flow 16).

The diagram above shows the ml_unavailable path in detail. The remaining failure domains follow the same notification pattern:

App → Core WS drop: The browser's WebSocket onclose event fires. The OPFS shadow buffer captures all audio produced during the outage. On reconnect, the client sends ResumeRecordingCommand and Flow 16 backfills any missing GCS chunks before audio forwarding to ML resumes.

Core → GCS failure: Chunk writes fail — Core sends RecordingErrorEvent (storage_unavailable). Audio continues flowing through the WebSocket; the OPFS shadow buffer captures the gap locally. On GCS recovery, Core sends RecordingErrorEvent (storage_recovered, last_stored_seq) and gap chunks are uploaded via Flow 16.

ML → AssemblyAI failure: Transcript segments stop arriving. Core sends RecordingErrorEvent (transcoder_error). Voice embeddings are unaffected. Audio and notes continue. Missing transcript is rebuilt during post-meeting processing from the full audio in GCS.

ML → OpenAI failure: Talking points and task extraction stop. Core sends RecordingErrorEvent (insight_warning). Transcript and speaker ID are unaffected. Missing insights are rebuilt during post-meeting processing.

Flow 15: Audio Silence Detection

Two layers detect audio problems: the browser catches microphone issues locally, and Core catches broken streams server-side.

Client-side (primary): The browser monitors the MediaStream via Web Audio API AnalyserNode. If the RMS level falls below a threshold for 10 consecutive seconds, the client shows an inline notice. No server round-trip needed — this is purely a UX signal. Clears automatically when audio levels recover or the first transcript segment arrives.

Server-side (secondary): Core tracks time since the last audio chunk was received on the WebSocket. If no chunks arrive for 10 seconds while a session is active, Core sends a RecordingErrorEvent. This catches the case where the browser believes it's sending audio but the WebSocket stream is silently broken.

Flow 16: Audio Gap Recovery

When a connectivity gap occurs — either the App→Core WebSocket drops or Core cannot write to GCS — the OPFS shadow buffer accumulates all audio produced during the outage. On recovery, the client compares its OPFS buffer against the last sequence number Core successfully stored in GCS, then uploads any missing chunks via REST. Core writes each to GCS by sequence number and deduplicates — chunks already stored are skipped. This same flow runs at session stop time if any gaps remain (see Flow 9).

Flow 17: Server-Side Inactivity Timeout — New

If no audio chunks arrive for a configurable period (default: 5 minutes) while a recording session is active, Core treats the session as abandoned and triggers the same stop sequence as Flow 9. This covers the case where the user closes their laptop lid, loses power, or otherwise disappears without explicitly stopping — the WebSocket heartbeat timeout (~60 seconds) transitions the connection to closed, but the recording resource would remain in active state indefinitely without this secondary timeout. The inactivity timeout prevents abandoned sessions from blocking the concurrent-session guard.

Flow 18: Background Tab Audio Continuity — New

Chrome and other browsers aggressively throttle background tabs — JavaScript timers are capped at 1 execution per minute, and some WebSocket activity may be delayed. However, MediaRecorder itself runs on a browser-internal thread and is not throttled when the tab is backgrounded. The critical design choice: all audio chunk processing (sequence numbering, OPFS writes, and WebSocket sends) runs in a dedicated Web Worker, which is exempt from background tab throttling. The main thread only receives notifications for UI updates.

This means audio capture and transmission continue uninterrupted when the user switches to another tab. The page title changes to "● Recording…" so the user can find the tab.

Flow 19: Batched Gap Upload — New

When a large connectivity gap occurs (e.g., 30 minutes offline = ~18,000 chunks), the client uploads gap chunks in batches rather than one-at-a-time. The client reads chunks from OPFS in batches of 50, uploads each batch as a single multipart request to POST /meetings/{id}/recording/chunks, and uses the remaining_missing_sequences in the response to drive the next batch. A determinate progress indicator shows upload progress on the Meeting Summary page. If the browser closes mid-upload, the upload resumes from where it left off on next page load (OPFS retains all chunks until AudioChunkStoredEvent confirms GCS receipt).

Flow 20: Soft-Deleted Meeting During Post-Processing — New

Meetings are soft-deleted (flagged with deleted_at, not removed from the database). If a user soft-deletes a meeting while post-meeting processing is running, ML's write-back calls to Core REST will encounter a soft-deleted resource. Core handles this gracefully: write-back endpoints (PUT /transcriptions/{id}/segments, PUT /meetings/{id}/synthesis, PATCH /transcriptions/{id}/status) check deleted_at and return 204 No Content if the meeting is soft-deleted. ML treats this as success (no retry). The post-meeting processing completes silently — artefacts are written to the soft-deleted meeting's rows in the database but are never visible to the user. This avoids 404 errors, unnecessary retries, and DLQ noise.

---

Design Decisions

Key architectural choices and their rationale. These are the "why" behind the flows above.

---

Boundary Inventory

Every boundary shown in the diagrams above. Each becomes a contract on the Contracts page.