diff --git a/docs/SERVER_SIDE_WEBRTC_ARCHITECTURE.md b/docs/SERVER_SIDE_WEBRTC_ARCHITECTURE.md deleted file mode 100644 index 5af83b932..000000000 --- a/docs/SERVER_SIDE_WEBRTC_ARCHITECTURE.md +++ /dev/null @@ -1,1504 +0,0 @@ -# Server-Side WebRTC Architecture for WhatsApp Calling - -## Executive Summary - -This document designs a migration from browser-based WebRTC to server-side WebRTC for -Chatwoot's WhatsApp Calling feature. The new architecture places a **Pion-based Go -sidecar** (called `chatwoot-media-server`) between Meta's media servers and the agent's -browser. This gives us call persistence across page reloads, server-side recording, -and centralized call lifecycle management -- all without modifying the existing Rails -application server in a language-incompatible way. - ---- - -## Table of Contents - -1. [Technology Recommendation](#1-technology-recommendation) -2. [System Architecture](#2-system-architecture) -3. [Component Interaction Flows](#3-component-interaction-flows) -4. [Inbound Call Flow](#4-inbound-call-flow) -5. [Outbound Call Flow](#5-outbound-call-flow) -6. [Recording Pipeline](#6-recording-pipeline) -7. [Frontend-to-Backend Audio Relay](#7-frontend-to-backend-audio-relay) -8. [Call Lifecycle and Reconnection](#8-call-lifecycle-and-reconnection) -9. [Database Schema Changes](#9-database-schema-changes) -10. [File Structure](#10-file-structure) -11. [Deployment Architecture](#11-deployment-architecture) -12. [Scalability Analysis](#12-scalability-analysis) -13. [Security Considerations](#13-security-considerations) -14. [Pros, Cons, and Trade-offs](#14-pros-cons-and-trade-offs) -15. [Migration Path](#15-migration-path) - ---- - -## 1. Technology Recommendation - -### Evaluation Matrix - -| Criterion | Janus (C) | mediasoup (Node) | Pion (Go) | GStreamer+WebRTC (C) | -|------------------------------|------------|-------------------|----------------|----------------------| -| WebRTC standard compliance | Full | Full (SFU only) | Full | Full | -| Can act as WebRTC endpoint | Yes | No (SFU, not endpoint) | Yes | Yes | -| SDP offer/answer handling | Plugin API | JS API | Native Go API | GStreamer pipeline | -| Audio mixing/recording | Plugins | External | Manual + Opus | Native pipelines | -| Operational complexity | High (C, plugins, Lua) | Medium | Low | High (pipelines) | -| Docker image size | ~200 MB | ~150 MB | ~15-30 MB | ~300 MB | -| Concurrent call capacity | High | High | Very high | High | -| Memory per call | ~5 MB | ~3 MB | ~1-2 MB | ~8 MB | -| Language interop with Rails | HTTP/WS | HTTP/WS | HTTP/gRPC | CLI/HTTP | -| Community / maintenance | Active | Active | Very active | Active | -| Learning curve for team | Steep | Moderate | Moderate | Steep | - -### Recommendation: Pion (Go sidecar) - -**Pion** is the best fit for this use case. Here is the rationale: - -1. **Full WebRTC endpoint capability.** Pion can hold an `RTCPeerConnection` on the - server side -- it is not just an SFU. It can receive Meta's SDP offer, generate an - SDP answer, and establish a direct SRTP audio session with Meta's media servers. - mediasoup cannot do this; it is an SFU that only forwards packets between peers. - -2. **Lightweight.** A statically-compiled Go binary produces a Docker image under 30 MB. - No plugin system, no Lua, no native dependency chain. This matters for Chatwoot's - self-hosted deployment model. - -3. **Audio recording is straightforward.** Pion gives access to raw RTP packets and - decoded Opus frames. Writing them to an OGG/Opus container (or decoding to PCM for - WAV) is well-documented in Pion's examples. No GStreamer pipeline needed. - -4. **Excellent concurrency.** Go's goroutine model handles thousands of concurrent - connections with minimal memory overhead. Each active call consumes approximately - 1-2 MB of memory (RTP buffers + SRTP context). - -5. **Clean integration surface.** The Go sidecar exposes a simple HTTP/gRPC API that - Rails calls. No shared memory, no FFI, no process spawning. - -6. **Active ecosystem.** Pion has 13k+ GitHub stars, weekly releases, and an active - Discord. The `pion/webrtc`, `pion/rtp`, `pion/opus`, and `pion/interceptor` - packages cover all requirements. - -### Why not the others - -- **Janus**: Overkill. Requires maintaining C plugins, Lua scripting, and a complex - configuration. Designed for multi-party video conferencing, not 1:1 audio relay. -- **mediasoup**: Cannot act as a WebRTC endpoint. It is an SFU that forwards media - between browser peers. Meta expects to negotiate with a single WebRTC peer, not a - forwarding unit. -- **GStreamer**: Powerful but heavy. Pipeline-based architecture is great for complex - media processing, but adds significant operational complexity for what is - fundamentally a simple audio relay + recording task. - ---- - -## 2. System Architecture - -### High-Level Diagram - -``` - CHATWOOT SERVER - ┌─────────────────────────────────────────────────────┐ - │ │ - ┌──────────────┐ │ ┌────────────────┐ ┌──────────────────────┐ │ - │ WhatsApp │ Webhook │ │ │ HTTP │ │ │ - │ Contact │ (signaling) │ │ Rails App │◄─────►│ chatwoot-media- │ │ - │ │─────────────────────►│ (Puma/ │ │ server (Pion Go) │ │ - └──────┬───────┘ │ │ Sidekiq) │ │ │ │ - │ │ │ │ │ Holds WebRTC peers: │ │ - │ │ └───────┬────────┘ │ - Meta-side peer │ │ - │ │ │ │ - Agent-side peer │ │ - │ │ │ ActionCable │ - Recording engine │ │ - │ │ │ (WebSocket) │ │ │ - │ ┌────────────────────┐ │ │ └──────────┬───────────┘ │ - │ │ Meta Media │ │ │ │ │ - │ │ Servers │ │ │ SRTP │ SRTP │ - │ │ │ │ │ (Meta side) │ (Agent side) │ - └──►│ Routes audio │◄═══════════╪══════════════════════════╡ │ - │ between endpoints │ │ │ │ │ - └────────────────────┘ │ ┌───────▼────────┐ │ │ - │ │ Agent's │◄══════════════╡ │ - │ │ Browser │ WebRTC audio │ │ - │ │ (Vue.js) │ (SRTP) │ │ - │ │ │ │ │ - │ └────────────────┘ │ │ - │ │ │ - │ ┌─────────────▼───────────┐ │ - │ │ Recording Pipeline │ │ - │ │ │ │ - │ │ OGG/Opus file │ │ - │ │ → ActiveStorage │ │ - │ └─────────────────────────┘ │ - └─────────────────────────────────────────────────────┘ -``` - -### Two-Peer Bridge Architecture - -The media server maintains **two independent WebRTC peer connections per call** and -bridges audio between them: - -``` - Meta Media Servers chatwoot-media-server Agent Browser - ════════════════ ═══════════════════ ══════════════ - SRTP SRTP - ┌──────────────┐ ◄═══════════► ┌──────────────┐ ◄═══════════► ┌──────────────┐ - │ │ │ │ │ │ - │ Meta's │ Peer A │ Pion Go │ Peer B │ Browser │ - │ WebRTC │ (Meta-side │ process │ (Agent-side │ WebRTC │ - │ endpoint │ connection) │ │ connection) │ RTCPeerConn │ - │ │ │ │ │ │ - └──────────────┘ │ ┌────────┐ │ └──────────────┘ - │ │ Audio │ │ - │ │ Bridge │ │ - │ │ │ │ - │ │ A→B │ │ - │ │ B→A │ │ - │ │ │ │ - │ │ +Record │ │ - │ └────────┘ │ - └──────────────┘ -``` - -**Why two peers instead of one:** - -Meta's WebRTC endpoint negotiates a standard peer connection. The agent's browser also -needs a standard peer connection. The media server sits in the middle as a "back-to-back -user agent" (B2BUA), forwarding RTP packets between the two peers while also tapping -into the streams for recording. - -This is a well-established pattern in VoIP (analogous to a SIP B2BUA). Each peer -connection is independent: Meta does not know about the agent, the agent does not know -about Meta. The media server handles codec negotiation, SRTP encryption/decryption, and -RTP forwarding for both sides. - ---- - -## 3. Component Interaction Flows - -### Component Responsibilities - -``` -┌─────────────────────────────────────────────────────────────────────────────────────┐ -│ │ -│ RAILS APP (existing, modified) │ -│ ───────────────────────────── │ -│ - Receives Meta webhooks (call connect/terminate events) │ -│ - Manages Call model lifecycle (DB records, status transitions) │ -│ - Delegates SDP handling to media server via internal HTTP API │ -│ - Broadcasts ActionCable events to agent browsers │ -│ - Stores recordings in ActiveStorage after media server delivers them │ -│ - Runs transcription jobs (unchanged) │ -│ │ -│ CHATWOOT-MEDIA-SERVER (new, Go/Pion) │ -│ ──────────────────────────────────── │ -│ - Holds Meta-side WebRTC peer connection (Peer A) │ -│ - Holds Agent-side WebRTC peer connection (Peer B) │ -│ - Bridges audio packets between Peer A and Peer B │ -│ - Records both audio streams to disk (OGG/Opus) │ -│ - Manages ICE/STUN/TURN for both peers │ -│ - Exposes HTTP API for Rails to control call sessions │ -│ - Reports call events back to Rails via HTTP callbacks │ -│ │ -│ AGENT BROWSER (modified) │ -│ ──────────────────────── │ -│ - Establishes WebRTC peer connection to media server (NOT to Meta directly) │ -│ - Sends/receives audio via standard WebRTC (getUserMedia + RTCPeerConnection) │ -│ - No longer handles recording (server does it) │ -│ - Can reconnect to an active call after page reload │ -│ - Receives SDP offer from media server via ActionCable, responds with answer │ -│ │ -└─────────────────────────────────────────────────────────────────────────────────────┘ -``` - -### Media Server Internal HTTP API - -The Go sidecar exposes the following HTTP endpoints, accessible only from the Rails -process (bound to localhost or internal network): - -``` -POST /sessions Create a new call session -POST /sessions/:id/meta-sdp Set Meta's SDP offer and get server's SDP answer -POST /sessions/:id/agent-offer Get SDP offer for agent-side peer -POST /sessions/:id/agent-answer Set agent's SDP answer for agent-side peer -POST /sessions/:id/terminate Tear down both peers and finalize recording -GET /sessions/:id/status Get session status (peers connected, ICE state) -GET /sessions/:id/recording Download the finalized recording file -DELETE /sessions/:id Force-destroy a session - -POST /sessions/:id/agent-reconnect Generate new agent-side offer for reconnection -``` - -Authentication between Rails and the media server uses a shared secret token passed as -a Bearer header, configured via environment variable `MEDIA_SERVER_AUTH_TOKEN`. - ---- - -## 4. Inbound Call Flow - -``` - WhatsApp Meta Chatwoot Media Agent - Contact Cloud Backend Server Browser - │ │ │ │ │ - 1. │── Dials ──────────►│ │ │ │ - │ business # │ │ │ │ - │ │ │ │ │ - 2. │ │── Webhook ──────────►│ │ │ - │ │ event: connect │ │ │ - │ │ {id, from, │ │ │ - │ │ sdp_offer} │ │ │ - │ │ │ │ │ - 3. │ │ ┌─────┴──────┐ │ │ - │ │ │ Incoming │ │ │ - │ │ │ CallService │ │ │ - │ │ │ │ │ │ - │ │ │ a. Contact │ │ │ - │ │ │ b. Convo │ │ │ - │ │ │ c. Call rec │ │ │ - │ │ └─────┬──────┘ │ │ - │ │ │ │ │ - 4. │ │ │── POST /sessions ─►│ │ - │ │ │ {call_id, │ │ - │ │ │ meta_sdp_offer} │ │ - │ │ │ │ │ - 5. │ │ │ ┌─────┴──────┐ │ - │ │ │ │ Create │ │ - │ │ │ │ Peer A │ │ - │ │ │ │ (Meta-side)│ │ - │ │ │ │ │ │ - │ │ │ │ Set remote │ │ - │ │ │ │ desc (SDP │ │ - │ │ │ │ offer) │ │ - │ │ │ │ │ │ - │ │ │ │ Create │ │ - │ │ │ │ answer │ │ - │ │ │ │ + ICE │ │ - │ │ │ │ gather │ │ - │ │ │ └─────┬──────┘ │ - │ │ │ │ │ - 6. │ │ │◄── {sdp_answer, │ │ - │ │ │ session_id} │ │ - │ │ │ │ │ - 7. │ │ │ Store session_id │ │ - │ │ │ in Call.meta │ │ - │ │ │ │ │ - │ │ │ NOTE: Do NOT send │ │ - │ │ │ SDP answer to Meta│ │ - │ │ │ yet. Wait for │ │ - │ │ │ agent to accept. │ │ - │ │ │ │ │ - 8. │ │ │── ActionCable ─────────────────────────►│ - │ │ │ whatsapp_call.incoming │ - │ │ │ {id, call_id, caller, conversation_id} │ - │ │ │ (NO sdp_offer -- agent doesn't need it)│ - │ │ │ │ │ - 9. │ │ │ │ ┌──────┴───┐ - │ │ │ │ │ Widget: │ - │ │ │ │ │ "Incoming│ - │ │ │ │ │ call" │ - │ │ │ │ │ [Accept] │ - │ │ │ │ │ [Reject] │ - │ │ │ │ └──────┬───┘ - │ │ │ │ │ - │ │ │ │ Agent clicks │ - │ │ │ │ "Accept" │ - │ │ │ │ │ -10. │ │ │◄── POST /accept ──────────────────────│ - │ │ │ {call_id} │ │ - │ │ │ (NO sdp_answer │ │ - │ │ │ from browser!) │ │ - │ │ │ │ │ -11. │ │ ┌─────┴──────┐ │ │ - │ │ │ CallService│ │ │ - │ │ │ (row lock) │ │ │ - │ │ │ │ │ │ - │ │ │ Validate │ │ │ - │ │ │ still ring │ │ │ - │ │ └─────┬──────┘ │ │ - │ │ │ │ │ -12. │ │◄── pre_accept ──────│ │ │ - │ │ + accept │ │ │ - │ │ (SDP from media │ │ │ - │ │ server, stored │ │ │ - │ │ in Call.meta) │ │ │ - │ │ │ │ │ -13. │◄════════════ Audio (SRTP) ═══════════════════════════════════►│ (Peer A) │ - │ │ │ │ │ - │ │ │ │ │ -14. │ │ │── POST /sessions/ │ │ - │ │ │ :id/agent-offer ►│ │ - │ │ │ │ │ -15. │ │ │ ┌─────┴──────┐ │ - │ │ │ │ Create │ │ - │ │ │ │ Peer B │ │ - │ │ │ │ (agent- │ │ - │ │ │ │ side) │ │ - │ │ │ │ │ │ - │ │ │ │ Create │ │ - │ │ │ │ SDP offer │ │ - │ │ │ │ for agent │ │ - │ │ │ └─────┬──────┘ │ - │ │ │ │ │ -16. │ │ │◄── {sdp_offer} ───│ │ - │ │ │ │ │ -17. │ │ │── ActionCable ─────────────────────────►│ - │ │ │ whatsapp_call.agent_offer │ - │ │ │ {sdp_offer, ice_servers} │ - │ │ │ │ │ -18. │ │ │ │ ┌──────┴───┐ - │ │ │ │ │getUserMe-│ - │ │ │ │ │dia(audio)│ - │ │ │ │ │ │ - │ │ │ │ │setRemote-│ - │ │ │ │ │Desc(offer│ - │ │ │ │ │from media│ - │ │ │ │ │server) │ - │ │ │ │ │ │ - │ │ │ │ │createAns-│ - │ │ │ │ │wer + ICE │ - │ │ │ │ └──────┬───┘ - │ │ │ │ │ -19. │ │ │◄── POST /agent-answer ────────────────│ - │ │ │ {sdp_answer} │ │ - │ │ │ │ │ -20. │ │ │── POST /sessions/ │ │ - │ │ │ :id/agent-answer►│ │ - │ │ │ │ │ -21. │ │ │ ┌─────┴──────┐ │ - │ │ │ │ Set agent │ │ - │ │ │ │ SDP answer │ │ - │ │ │ │ on Peer B │ │ - │ │ │ │ │ │ - │ │ │ │ Bridge │ │ - │ │ │ │ audio: │ │ - │ │ │ │ A <-> B │ │ - │ │ │ │ │ │ - │ │ │ │ Start │ │ - │ │ │ │ recording │ │ - │ │ │ └─────┬──────┘ │ - │ │ │ │ │ -22. │◄══════ Audio (Meta ↔ Media Server) ════►│◄══ Audio (Media Server ↔ Browser) ══►│ - │ │ │ │ │ - │ Agent hears customer, customer hears agent. Recording captures both. │ -``` - -### Key Differences from Current Architecture - -| Aspect | Current (browser WebRTC) | New (server WebRTC) | -|------------------------------|---------------------------------------|----------------------------------------| -| Who answers Meta's SDP | Agent's browser | Media server (Pion) | -| Audio path | Meta <-> Browser (direct) | Meta <-> Media Server <-> Browser | -| Accept API payload | Browser sends `sdp_answer` | Browser sends nothing (just call ID) | -| SDP answer source | Browser's RTCPeerConnection | Media server's Pion peer | -| Agent-side connection | N/A (direct to Meta) | Separate WebRTC peer (Peer B) | -| Recording | Browser MediaRecorder | Media server OGG/Opus writer | -| Survives page reload | No (active call dies) | Yes (Peer A stays up, Peer B reconnects)| -| SDP fix (actpass -> active) | Backend string replacement | Media server generates correct SDP | - ---- - -## 5. Outbound Call Flow - -``` - Agent Chatwoot Media Meta WhatsApp - Browser Backend Server Cloud Contact - │ │ │ │ │ - 1. │── Click phone icon │ │ │ │ - │ │ │ │ │ - 2. │── POST /initiate ──►│ │ │ │ - │ {conversation_id} │ │ │ │ - │ (NO sdp_offer │ │ │ │ - │ from browser!) │ │ │ │ - │ │ │ │ │ - 3. │ │── POST /sessions ──►│ │ │ - │ │ {call_id: temp, │ │ │ - │ │ direction: out} │ │ │ - │ │ │ │ │ - 4. │ │ ┌──────┴───────┐ │ │ - │ │ │ Create Peer A │ │ │ - │ │ │ (Meta-side) │ │ │ - │ │ │ │ │ │ - │ │ │ Create SDP │ │ │ - │ │ │ offer for │ │ │ - │ │ │ Meta │ │ │ - │ │ └──────┬───────┘ │ │ - │ │ │ │ │ - 5. │ │◄── {sdp_offer, │ │ │ - │ │ session_id} │ │ │ - │ │ │ │ │ - 6. │ │── initiate_call ────────────────────────►│ │ - │ │ {to, sdp_offer │ │── Rings phone ──►│ - │ │ from media server}│ │ │ - │ │ │ │ │ - │ │◄── {call_id} ───────────────────────────│ │ - │ │ │ │ │ - 7. │ │ Create Call record │ │ │ - │ │ (outgoing, ringing) │ │ │ - │ │ Store session_id │ │ │ - │ │ │ │ │ - 8. │◄── {status: calling,│ │ │ │ - │ call_id, id} │ │ │ │ - │ │ │ │ │ - │ Widget: "Ringing.."│ │ │ │ - │ │ │ │ │ - │ │ │ Contact answers │ - │ │ │ │◄─────────────────│ - │ │ │ │ │ - 9. │ │◄── Webhook ─────────────────────────────│ │ - │ │ event: connect │ │ │ - │ │ {call_id, │ │ │ - │ │ sdp_answer} │ │ │ - │ │ │ │ │ -10. │ │── POST /sessions/ │ │ │ - │ │ :id/meta-sdp │ │ │ - │ │ {sdp_answer} ►│ │ │ - │ │ │ │ │ -11. │ │ ┌──────┴───────┐ │ │ - │ │ │ Set Meta's │ │ │ - │ │ │ SDP answer │ │ │ - │ │ │ on Peer A │ │ │ - │ │ └──────┬───────┘ │ │ - │ │ │ │ │ -12. │◄══════════ Audio (Meta ↔ Media Server, Peer A) ══════════════►│ │ - │ │ │ │ │ - │ (Now set up Peer B for agent) │ │ │ - │ │ │ │ │ -13-21. [Same agent-side SDP exchange as inbound steps 14-22] │ │ - │ │ │ │ │ -22. │◄══ Audio (Media Server ↔ Browser, Peer B) ══►│ │ │ - │ │ │ │ │ - │ Full duplex audio: Agent <-> Media Server <-> Meta <-> Contact │ -``` - ---- - -## 6. Recording Pipeline - -### Architecture - -``` - chatwoot-media-server - ┌──────────────────────────────────────────────────────┐ - │ │ - │ Peer A (Meta) Peer B (Agent) │ - │ ┌─────────┐ ┌─────────┐ │ - │ │ Remote │ │ Remote │ │ - │ │ track │─────┐ ┌────│ track │ │ - │ │ (customer│ │ │ │ (agent │ │ - │ │ audio) │ │ │ │ audio) │ │ - │ └─────────┘ │ │ └─────────┘ │ - │ │ │ │ - │ ┌─────▼─────▼──────┐ │ - │ │ Recording Mux │ │ - │ │ │ │ - │ │ Option A: │ │ - │ │ Single mixed │ │ - │ │ OGG/Opus file │ │ - │ │ │ │ - │ │ Option B: │ │ - │ │ Two separate │ │ - │ │ tracks (stereo: │ │ - │ │ L=customer, │ │ - │ │ R=agent) │ │ - │ │ │ │ - │ └────────┬─────────┘ │ - │ │ │ - │ ▼ │ - │ ┌─────────────────┐ │ - │ │ /recordings/ │ │ - │ │ {session_id}.ogg│ │ - │ │ │ │ - │ │ Written in real │ │ - │ │ time as packets │ │ - │ │ arrive │ │ - │ └─────────────────┘ │ - │ │ - └──────────────────────────────────────────────────────┘ - │ - On call end - │ - ▼ - ┌─────────────────┐ - │ Rails callback │ - │ │ - │ Media server │ - │ POSTs to Rails: │ - │ /callbacks/ │ - │ recording_ready │ - │ {session_id, │ - │ file_path} │ - └────────┬────────┘ - │ - ▼ - ┌─────────────────┐ - │ Rails downloads │ - │ file from media │ - │ server via │ - │ GET /sessions/ │ - │ :id/recording │ - │ │ - │ Attaches to │ - │ ActiveStorage │ - │ │ - │ Enqueues │ - │ transcription │ - │ job │ - └─────────────────┘ -``` - -### Recording Implementation Details - -**Format:** OGG container with Opus codec. This is the native codec for WebRTC audio, -so no transcoding is needed. The Pion interceptor taps RTP packets before -encryption/after decryption and writes Opus frames into OGG pages. - -**Real-time writing:** Recording starts when both peers are connected and audio is -bridged. Opus frames are written to disk as they arrive (not buffered in memory). This -means even if the media server crashes, the recording up to that point is preserved on -disk. - -**Stereo separation (recommended):** Record customer audio on the left channel and -agent audio on the right channel. This enables: -- Better transcription accuracy (speaker diarization is trivial) -- Post-processing flexibility (adjust volumes, filter noise per channel) -- Compliance with regulations that require separate party recordings - -**File lifecycle:** -1. File created when recording starts: `/recordings/{session_id}.ogg` -2. Appended in real time as audio packets arrive -3. Finalized (OGG trailer written) when call ends -4. Rails fetches via HTTP, attaches to ActiveStorage -5. Media server deletes local file after Rails confirms receipt - -**Failure mode:** If the media server process crashes mid-recording, the OGG file on -disk is incomplete but still playable (most players handle truncated OGG). A startup -recovery routine scans for orphaned recording files and reports them to Rails. - ---- - -## 7. Frontend-to-Backend Audio Relay - -### Mechanism: Standard WebRTC (Peer B) - -The agent's browser establishes a normal WebRTC peer connection -- but to the media -server instead of Meta. From the browser's perspective, it looks identical to the -current implementation: - -``` - Current: Browser <--WebRTC--> Meta Media Servers - New: Browser <--WebRTC--> chatwoot-media-server <--WebRTC--> Meta Media Servers -``` - -The browser still uses `getUserMedia`, `RTCPeerConnection`, `ontrack`, etc. The only -difference is the SDP it receives comes from the media server rather than Meta. - -### SDP Exchange for Agent-Side Peer (Peer B) - -``` - Rails Media Server Browser - │ │ │ - │── POST /sessions/:id/ │ │ - │ agent-offer ──────────────────►│ │ - │ │── Create Peer B │ - │ │── Generate SDP offer │ - │ │── Gather ICE candidates │ - │◄── {sdp_offer, ice_servers} ─────│ │ - │ │ │ - │── ActionCable: agent_offer ──────────────────────────────────►│ - │ {sdp_offer, ice_servers} │ │ - │ │ ┌────────┴──────┐ - │ │ │getUserMedia │ - │ │ │setRemoteDesc │ - │ │ │createAnswer │ - │ │ │ICE gather │ - │ │ └────────┬──────┘ - │ │ │ - │◄── POST /whatsapp_calls/:id/agent-answer ──────────────────────│ - │ {sdp_answer} │ │ - │ │ │ - │── POST /sessions/:id/ │ │ - │ agent-answer ─────────────────►│ │ - │ │── Set remote desc on Peer B │ - │ │── Bridge audio A <-> B │ - │ │── Start recording │ - │ │ │ - │ │◄═══════ SRTP audio ════════►│ -``` - -### Why WebRTC (not WebSocket audio streaming) - -Alternatives considered: - -| Approach | Latency | Complexity | Browser support | Quality | -|-----------------------|------------|------------|-----------------|----------------| -| **WebRTC (Peer B)** | ~20-50ms | Low | Native | Opus, adaptive | -| WebSocket + PCM | ~100-200ms | High | Manual codec | Fixed bitrate | -| WebSocket + Opus | ~80-150ms | Medium | Manual decode | Good | -| HTTP chunked | ~200-500ms | Medium | Standard | Variable | - -WebRTC is the clear winner: -- **Latency**: Sub-50ms, critical for real-time voice conversation -- **Codec negotiation**: Opus bitrate adaptation is automatic -- **Echo cancellation**: Built into browser WebRTC stack -- **NAT traversal**: ICE/STUN/TURN handled automatically -- **No custom audio code**: Browser handles decode/playback natively - -WebSocket audio would require building a custom audio pipeline in the browser: -decoding Opus, managing jitter buffers, handling echo cancellation, and scheduling -audio playback -- all of which WebRTC already handles. - ---- - -## 8. Call Lifecycle and Reconnection - -### Call State Machine (Server-Side) - -``` - ┌─────────────┐ - │ created │ Media server session allocated - └──────┬──────┘ - │ - Meta SDP exchanged (Peer A ready) - │ - ┌──────▼──────┐ - │ ringing │ Waiting for agent to accept - └──┬───┬───┬──┘ - │ │ │ - accepted │ │ │ timeout / reject - │ │ │ - ┌──────▼┐ │ ┌▼──────────┐ - │ meta_ │ │ │ no_answer │ - │ conn │ │ │ / failed │ - └──┬────┘ │ └────────────┘ - │ │ - agent SDP │ │ agent never connected - exchanged │ │ - │ │ - ┌──────▼──────┐│ - │ in_progress ││ Both peers connected, audio bridged - └──────┬──────┘│ - │ │ - ┌──────▼──────┐│ - │agent_disconn││ Agent page reload / network issue - │(Peer B down)││ Peer A still holds Meta audio - └──┬───┬──────┘│ - │ │ │ - reconnect│ │timeout│ - │ │(30s) │ - ┌──────▼┐ │ │ - │in_prog│ │ │ - │(reconn│ │ │ - │ected) │ │ │ - └──┬────┘ │ │ - │ │ │ - ├───────┤ │ - │ │ │ - ┌──────▼───────▼───────▼──┐ - │ completed │ Call ended (any reason) - └─────────────────────────┘ -``` - -### Reconnection After Page Reload - -This is the primary motivation for the architecture change. Here is the reconnection -flow: - -``` - Agent Browser Rails Media Server - │ │ │ - │ ═══ Active call ══════════╪═════════════════════════════╡ - │ (Peer B established) │ │ Peer A: active - │ │ │ Peer B: active - │ │ │ Audio: bridged - │ │ │ Recording: active - │ │ │ - │── Page refresh ──► │ │ - │ (Peer B connection dies) │ │ - │ │ │ - │ │◄── Callback: agent_disconn ─│ (Peer B ICE fails) - │ │ │ - │ │ Call status stays │ Peer A: STILL ACTIVE - │ │ "in_progress" │ Audio from Meta: still - │ │ Start 30s reconnect timer │ flowing (into void) - │ │ │ Recording: paused - │ │ │ (agent track silent) - │ │ │ - │── Page loads │ │ - │ Vue mounts │ │ - │ │ │ - │── GET /whatsapp_calls/ │ │ - │ active ─────────────────►│ │ - │ │ │ - │◄── {call in progress, │ │ - │ call_id, id} │ │ - │ │ │ - │ Widget auto-shows: │ │ - │ "Call in progress - │ │ - │ Reconnecting..." │ │ - │ │ │ - │── POST /whatsapp_calls/ │ │ - │ :id/reconnect ──────────►│ │ - │ │ │ - │ │── POST /sessions/:id/ │ - │ │ agent-reconnect ──────────►│ - │ │ │ - │ │ ┌──────┴──────┐ - │ │ │ Tear down │ - │ │ │ old Peer B │ - │ │ │ │ - │ │ │ Create new │ - │ │ │ Peer B │ - │ │ │ │ - │ │ │ Generate │ - │ │ │ new SDP │ - │ │ │ offer │ - │ │ └──────┬──────┘ - │ │ │ - │ │◄── {sdp_offer} ────────────│ - │ │ │ - │◄── ActionCable: agent_offer│ │ - │ {sdp_offer, ice_servers}│ │ - │ │ │ - │ getUserMedia (mic) │ │ - │ setRemoteDescription │ │ - │ createAnswer + ICE │ │ - │ │ │ - │── POST /agent-answer ─────►│ │ - │ {sdp_answer} │ │ - │ │── POST /sessions/:id/ │ - │ │ agent-answer ─────────────►│ - │ │ │ - │ │ ┌──────┴──────┐ - │ │ │ Set answer │ - │ │ │ on new │ - │ │ │ Peer B │ - │ │ │ │ - │ │ │ Re-bridge │ - │ │ │ audio │ - │ │ │ │ - │ │ │ Resume │ - │ │ │ recording │ - │ │ └──────┴──────┘ - │ │ │ - │◄══════════ Audio restored (< 2 seconds gap) ════════════╡ - │ │ │ - │ Widget: call timer │ │ - │ resumes from server- │ │ - │ tracked duration │ │ -``` - -### Key Reconnection Details - -1. **Reconnect window:** The media server holds Peer A (Meta-side) for 30 seconds - after Peer B (agent-side) disconnects. If the agent reconnects within that window, - the call continues seamlessly. If the timeout expires, the media server tells Rails - to terminate the call with Meta. - -2. **Audio gap:** The customer hears silence during the reconnection (typically 1-3 - seconds for a page reload). This is acceptable for a page reload scenario. The - media server could optionally play a hold tone to the Meta side during disconnection. - -3. **Timer continuity:** The call duration timer is tracked server-side - (`Call.started_at`). The frontend calculates elapsed time from `started_at` rather - than running a local timer. On reconnection, the widget immediately shows the correct - elapsed time. - -4. **Recording continuity:** The recording file on disk continues to receive the - customer's audio during the agent disconnect period. The agent's channel is silent - during the gap. When the agent reconnects, their audio resumes in the recording. - -5. **Multiple reconnections:** Each reconnection creates a new Peer B. There is no - limit on the number of reconnections during a single call. - -### Multi-Agent Visibility - -The existing pattern is preserved: - -- When a call is ringing, all agents in the account see the incoming call widget -- When one agent accepts, ActionCable `whatsapp_call.accepted` dismisses the call from - all other agents' widgets -- The accepting agent's browser is the only one that establishes Peer B -- If the accepting agent's Peer B disconnects and they do not reconnect within 30s, - the call could optionally be re-offered to other agents (future enhancement) - -### Media Server Restart - -If the Go media server process crashes or restarts: - -1. All active Peer A and Peer B connections are lost -2. The media server has no persistent state -- all sessions are in-memory -3. On startup, it scans `/recordings/` for orphaned files and reports them to Rails -4. Rails detects the loss via HTTP health check failure or callback timeout -5. Rails terminates all `in_progress` calls associated with the failed media server -6. Recordings up to the crash point are preserved on disk - -For high availability, multiple media server instances can run behind a load balancer -with session affinity (sticky sessions by `session_id`). Redis can optionally store -session metadata for cross-instance recovery, but this adds complexity and is not -recommended for the initial implementation. - ---- - -## 9. Database Schema Changes - -### Migration: Add media_session_id to calls - -``` -add_column :calls, :media_session_id, :string -add_index :calls, :media_session_id -``` - -The `media_session_id` is the unique identifier for the session on the Go media server. -Rails uses it to make API calls to the media server for a given call. - -### Updated Call.meta structure - -```json -{ - "media_session_id": "sess_abc123", - "meta_sdp_offer": "v=0\r\n...", - "meta_sdp_answer": "v=0\r\n...", - "agent_sdp_offer": "v=0\r\n...", - "agent_sdp_answer": "v=0\r\n...", - "ice_servers": [{"urls": "stun:stun.l.google.com:19302"}], - "agent_reconnect_count": 0, - "recording_file": "sess_abc123.ogg" -} -``` - -### New InstallationConfig keys - -| Key | Type | Description | -|------------------------------|---------|------------------------------------------------| -| `MEDIA_SERVER_URL` | string | Internal URL of the media server (e.g., `http://media:4000`) | -| `MEDIA_SERVER_AUTH_TOKEN` | string | Shared secret for Rails <-> media server auth | -| `MEDIA_SERVER_STUN_SERVERS` | string | Comma-separated STUN server URLs | -| `MEDIA_SERVER_TURN_SERVERS` | string | Comma-separated TURN server URLs | -| `MEDIA_SERVER_TURN_USERNAME` | string | TURN server username | -| `MEDIA_SERVER_TURN_PASSWORD` | string | TURN server password | - ---- - -## 10. File Structure - -### New Go sidecar (separate repository or monorepo subdirectory) - -``` -enterprise/media-server/ -├── Dockerfile -├── go.mod -├── go.sum -├── main.go # Entry point, HTTP server setup -├── config/ -│ └── config.go # Environment variable parsing -├── api/ -│ ├── handler.go # HTTP route definitions -│ ├── middleware.go # Auth token validation -│ ├── session_handler.go # POST /sessions, GET /sessions/:id -│ ├── meta_sdp_handler.go # POST /sessions/:id/meta-sdp -│ ├── agent_handler.go # agent-offer, agent-answer, agent-reconnect -│ └── recording_handler.go # GET /sessions/:id/recording -├── session/ -│ ├── manager.go # Session lifecycle (create, find, cleanup) -│ ├── session.go # Single call session (two peers + bridge) -│ ├── peer.go # WebRTC peer connection wrapper -│ └── bridge.go # RTP packet forwarding between peers -├── recording/ -│ ├── recorder.go # OGG/Opus file writer -│ ├── ogg_writer.go # Low-level OGG container writing -│ └── cleanup.go # Orphaned file detection on startup -├── callback/ -│ └── rails_client.go # HTTP client for callbacks to Rails -└── recordings/ # Runtime directory for recording files - └── .gitkeep -``` - -### Modified Rails files (enterprise) - -``` -enterprise/ -├── app/ -│ ├── controllers/api/v1/accounts/ -│ │ └── whatsapp_calls_controller.rb # MODIFIED: new endpoints -│ │ # - accept: no longer receives sdp_answer from browser -│ │ # - new: reconnect action -│ │ # - new: agent_answer action -│ │ # - remove: upload_recording (server handles it) -│ │ -│ ├── services/whatsapp/ -│ │ ├── call_service.rb # MODIFIED: delegates SDP to media server -│ │ ├── incoming_call_service.rb # MODIFIED: creates media session -│ │ ├── media_server_client.rb # NEW: HTTP client for media server API -│ │ ├── call_recording_service.rb # NEW: fetches recording from media server -│ │ └── call_reconnect_service.rb # NEW: handles agent reconnection -│ │ -│ ├── jobs/whatsapp/ -│ │ ├── call_recording_fetch_job.rb # NEW: async fetch recording on call end -│ │ └── call_transcription_job.rb # UNCHANGED -│ │ -│ └── models/ -│ └── call.rb # MODIFIED: add media_session_id accessor -│ -├── config/ -│ └── media_server.yml # Media server connection config -``` - -### Modified frontend files - -``` -app/javascript/dashboard/ -├── composables/ -│ └── useWhatsappCallSession.js # HEAVILY MODIFIED -│ # - No longer sends SDP to Meta -│ # - Receives SDP offer from media server via ActionCable -│ # - Sends SDP answer to Rails (which forwards to media server) -│ # - No longer handles recording -│ # - Adds reconnection logic -│ # - beforeunload no longer terminates call (just disconnects Peer B) -│ -├── stores/ -│ └── whatsappCalls.js # MODIFIED -│ # - New state: isReconnecting, reconnectAttempts -│ # - New action: handleAgentOffer (for reconnection) -│ # - Remove: outbound WebRTC state (moved to server) -│ -├── api/ -│ └── whatsappCalls.js # MODIFIED -│ # - accept: no sdp_answer parameter -│ # - new: agentAnswer(callId, sdpAnswer) -│ # - new: reconnect(callId) -│ # - remove: uploadRecording -│ # - new: getActiveCall() -│ -├── components/widgets/ -│ └── WhatsappCallWidget.vue # MODIFIED -│ # - Reconnection UI state -│ # - Timer from server-tracked started_at -│ -├── helper/ -│ └── actionCable.js # MODIFIED -│ # - Handle new event: whatsapp_call.agent_offer -│ # - Handle new event: whatsapp_call.agent_disconnected -``` - ---- - -## 11. Deployment Architecture - -### Docker Compose Addition - -```yaml -# Added to docker-compose.yml -services: - media-server: - build: - context: ./enterprise/media-server - dockerfile: Dockerfile - ports: - - "4000:4000" # Internal HTTP API (Rails communication) - - "10000-10100:10000-10100/udp" # RTP/SRTP media ports - environment: - - AUTH_TOKEN=${MEDIA_SERVER_AUTH_TOKEN} - - STUN_SERVERS=stun:stun.l.google.com:19302 - - TURN_SERVERS=${TURN_SERVERS:-} - - TURN_USERNAME=${TURN_USERNAME:-} - - TURN_PASSWORD=${TURN_PASSWORD:-} - - RAILS_CALLBACK_URL=http://web:3000 - - RECORDINGS_DIR=/recordings - - LOG_LEVEL=info - - UDP_PORT_MIN=10000 - - UDP_PORT_MAX=10100 - volumes: - - media-recordings:/recordings - networks: - - chatwoot - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "--spider", "-q", "http://localhost:4000/health"] - interval: 10s - timeout: 5s - retries: 3 - -volumes: - media-recordings: -``` - -### Procfile Addition - -``` -media: ./enterprise/media-server/chatwoot-media-server -``` - -### Network Architecture - -``` - ┌─────────────────────────────────────────────────────────────────────┐ - │ Docker Network │ - │ │ - │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ ┌────────────┐ │ - │ │ web │ │ worker │ │ media-server │ │ redis │ │ - │ │ (Rails) │ │ (Sidekiq)│ │ (Pion Go) │ │ │ │ - │ │ :3000 │ │ │ │ :4000 HTTP │ │ :6379 │ │ - │ │ ◄────────────────────► │ │ │ │ - │ │ │ │ │ │ :10000-10100│ │ │ │ - │ └────┬─────┘ └──────────┘ │ UDP (media) │ └────────────┘ │ - │ │ └───────┬──────┘ │ - │ │ │ │ - └───────┼────────────────────────────────┼───────────────────────────┘ - │ │ - │ :3000 (HTTPS via LB) │ :10000-10100/UDP - │ │ (must be exposed to internet - │ │ for WebRTC ICE connectivity) - ▼ ▼ - ┌─────────────┐ ┌──────────────────┐ - │ Internet │ │ Internet │ - │ (browsers, │ │ (Meta servers, │ - │ webhooks) │ │ agent browsers) │ - └─────────────┘ └──────────────────┘ -``` - -### Critical: UDP Port Exposure - -The media server MUST have UDP ports exposed to the internet for WebRTC ICE to work. -This is because: - -1. **Meta-side (Peer A):** Meta's media servers send SRTP packets to the media server's - public IP/port. The media server must be reachable from Meta's infrastructure. - -2. **Agent-side (Peer B):** The agent's browser sends SRTP packets to the media server. - If the agent is behind a NAT, STUN handles discovery, but the media server must still - have publicly-accessible UDP ports. - -If UDP ports cannot be directly exposed (e.g., behind a restrictive firewall or -cloud load balancer that does not support UDP), a **TURN server** is required as a -relay. The media server's ICE configuration should include TURN credentials for both -Peer A and Peer B. - -### STUN/TURN Configuration - -| Component | STUN needed? | TURN needed? | -|----------------|----------------------------------|--------------------------------------| -| Peer A (Meta) | Yes (discover server's public IP)| Only if Meta can't reach server UDP | -| Peer B (Agent) | Yes (standard) | If agent is behind restrictive NAT | - -Recommended setup: -- Deploy a TURN server (coturn) alongside the media server -- Or use a managed TURN service (Twilio Network Traversal, Cloudflare TURN) -- Configure TURN credentials in environment variables - ---- - -## 12. Scalability Analysis - -### Per-Call Resource Consumption (Media Server) - -| Resource | Per call | Notes | -|----------------|--------------|--------------------------------------------------| -| Memory | ~2-3 MB | Two peer connections, RTP buffers, SRTP contexts | -| CPU | ~0.5-1% | Packet forwarding + Opus frame copying (no transcoding) | -| Bandwidth | ~100 kbps | Opus audio, both directions (~50kbps each way) | -| Disk I/O | ~12 KB/s | Recording write (Opus @ 48kbps / 8 = 6KB/s * 2 streams) | -| UDP ports | 4 per call | 2 for Peer A (RTP + RTCP), 2 for Peer B | -| File handles | 3 per call | 2 sockets + 1 recording file | - -### Capacity Estimates (Single Media Server Instance) - -| Server Size | RAM | CPU | Concurrent Calls | Notes | -|-----------------|--------|-------|-------------------|------------------------------| -| Small (2 CPU) | 2 GB | 2 core| ~50 | Suitable for most deployments| -| Medium (4 CPU) | 4 GB | 4 core| ~200 | Mid-size contact centers | -| Large (8 CPU) | 8 GB | 8 core| ~500 | Large deployments | - -The bottleneck is typically network bandwidth, not CPU or memory. At 100 kbps per -call, 500 concurrent calls require ~50 Mbps of sustained bandwidth. - -### Horizontal Scaling - -For deployments exceeding 500 concurrent calls: - -``` - ┌─────────────────┐ - │ Rails App │ - │ │ - │ Routes calls to │ - │ media servers │ - │ via consistent │ - │ hashing on │ - │ call_id │ - └────────┬────────┘ - │ - ┌──────────────┼──────────────┐ - │ │ │ - ┌──────▼──────┐ ┌────▼────────┐ ┌──▼──────────┐ - │ media-srv-1 │ │ media-srv-2 │ │ media-srv-3 │ - │ :4000 │ │ :4001 │ │ :4002 │ - │ UDP 10000- │ │ UDP 10100- │ │ UDP 10200- │ - │ 10099 │ │ 10199 │ │ 10299 │ - └─────────────┘ └─────────────┘ └─────────────┘ -``` - -Each media server instance gets a unique range of UDP ports. Rails assigns calls to -instances using consistent hashing on `call_id`. The media server URL is stored in -`Call.meta['media_server_url']` so Rails always routes to the correct instance. - ---- - -## 13. Security Considerations - -### Internal API Authentication - -``` -Rails ──── Bearer {MEDIA_SERVER_AUTH_TOKEN} ────► Media Server -``` - -The shared secret is configured via environment variable. The media server validates -the token on every request. This is sufficient for co-located services on the same -Docker network. - -### SRTP Encryption - -All audio is encrypted via SRTP (Secure Real-time Transport Protocol). Pion handles -DTLS key exchange automatically as part of the WebRTC handshake. The media server -decrypts incoming SRTP from one peer and re-encrypts it for the other peer. Audio -is briefly in plaintext within the media server's memory during forwarding. - -### Recording Security - -- Recording files are stored on a Docker volume, not in a web-accessible directory -- Files are transferred to Rails via internal HTTP and stored in ActiveStorage -- ActiveStorage enforces authentication for blob downloads -- Recording files are deleted from the media server after successful transfer - -### ICE Candidate Filtering - -The media server should filter ICE candidates to avoid exposing internal network -topology. Only `srflx` (server-reflexive) and `relay` (TURN) candidates should be -included in SDP offers to agent browsers. `host` candidates revealing internal IPs -should be stripped. - -### Rate Limiting - -The media server should enforce: -- Maximum concurrent sessions per account (configurable, default: 10) -- Maximum session duration (configurable, default: 2 hours) -- Session creation rate limit (configurable, default: 5 per minute per account) - ---- - -## 14. Pros, Cons, and Trade-offs - -### Pros - -1. **Call persistence across page reloads.** The primary goal. The Meta-side WebRTC - connection is held by the server; the agent can reconnect within 30 seconds. - -2. **Reliable server-side recording.** No dependency on browser memory. Recording is - written to disk in real time. Even if the browser crashes, the recording up to that - point is saved. - -3. **Centralized call lifecycle.** Rails and the media server fully control the call. - The browser is a "dumb terminal" for audio I/O. This simplifies state management - and reduces frontend complexity. - -4. **No SDP fix needed in Rails.** The media server generates its own SDP answers using - Pion, which produces correct `a=setup:active` natively. The `actpass` -> `active` - string replacement hack is eliminated. - -5. **Foundation for advanced features.** Server-side audio access enables: - - Real-time transcription (stream audio to STT service) - - Call transfer (redirect Peer B to a different agent) - - Conference calls (mix multiple Peer B connections) - - Hold music (play audio file into Peer A while agent is disconnected) - - Supervisory monitoring (add a listen-only Peer C) - -6. **Recording format upgrade.** OGG/Opus with stereo channel separation (customer L, - agent R) improves transcription accuracy and post-processing flexibility vs the - current single-channel WebM blob. - -### Cons - -1. **Increased latency.** Audio passes through an additional hop (media server). Expected - additional latency: 1-5ms within the same datacenter. Unlikely to be perceptible - for voice calls (human perception threshold ~40ms). - -2. **New infrastructure component.** The Go sidecar is a new process to deploy, monitor, - and maintain. It adds operational complexity. Teams unfamiliar with Go need to learn - the codebase for troubleshooting. - -3. **UDP port management.** WebRTC requires UDP ports exposed to the internet. This is - non-trivial in cloud environments with TCP-only load balancers. May require - additional infrastructure (dedicated UDP-capable load balancer, or direct IP exposure). - -4. **Double encryption overhead.** Audio is SRTP-encrypted on both Peer A and Peer B. - The media server decrypts and re-encrypts every packet. This adds minimal CPU - overhead (~0.5% per call) but is architecturally unavoidable for a B2BUA. - -5. **Single point of failure.** If the media server crashes, all active calls drop. The - recovery is recordings on disk and Rail terminating lingering calls, but the calls - themselves cannot be recovered. Mitigated by health checks and automatic restart. - -6. **Increased bandwidth.** The media server doubles bandwidth consumption because it - receives audio from Meta AND sends it to the agent (and vice versa). In the browser- - direct model, Chatwoot's server uses zero media bandwidth. - -### Trade-offs Accepted - -| Trade-off | Justification | -|-------------------------------------|------------------------------------------------| -| +1 network hop latency | <5ms, imperceptible for voice | -| New Go service to maintain | Pion is stable; service is small (~2000 LoC) | -| UDP ports exposed | Standard WebRTC requirement; TURN as fallback | -| Double bandwidth | ~100kbps/call is trivial for modern infra | -| Recordings briefly on local disk | Cleaned up after ActiveStorage transfer | - ---- - -## 15. Migration Path - -### Phase 1: Build Media Server (2-3 weeks) - -**Goal:** Standalone Go binary that can hold a WebRTC peer and record audio. - -- Implement session management (create, find, destroy) -- Implement Peer A: accept SDP offer, generate SDP answer, hold SRTP connection -- Implement Peer B: generate SDP offer, accept SDP answer -- Implement audio bridge (forward RTP packets A <-> B) -- Implement OGG/Opus recording -- Implement HTTP API with auth token -- Implement health check endpoint -- Write integration tests with Pion's built-in test utilities -- Dockerize - -### Phase 2: Rails Integration (1-2 weeks) - -**Goal:** Rails delegates SDP handling to media server; recording fetched server-side. - -- Add `Whatsapp::MediaServerClient` service -- Modify `IncomingCallService` to create media session on webhook -- Modify `CallService#pre_accept_and_accept` to use server-generated SDP -- Add `CallReconnectService` -- Add `CallRecordingFetchJob` -- Add new controller actions: `reconnect`, `agent_answer` -- Remove `upload_recording` action -- Add ActionCable event: `whatsapp_call.agent_offer` -- Add media server health check to deployment monitoring -- Feature flag: `whatsapp_call_server_media` (separate from `whatsapp_call`) - -### Phase 3: Frontend Migration (1-2 weeks) - -**Goal:** Browser connects to media server (Peer B) instead of Meta directly. - -- Modify `useWhatsappCallSession.js`: - - Accept flow: wait for `agent_offer` ActionCable event instead of using Meta's SDP - - Send `agent_answer` to Rails instead of Meta's SDP answer - - Remove `MediaRecorder` logic entirely - - Add reconnection on page load: detect active call, trigger reconnect - - `beforeunload`: do NOT terminate call, just clean up Peer B locally -- Modify `whatsappCalls.js` store: - - Add reconnection state - - Handle `agent_offer` event - - Timer from `started_at` instead of local counter -- Modify `ConversationHeader.vue` (outbound): - - No longer generate SDP offer in browser - - Just POST to `/initiate` without `sdp_offer` - - Wait for `agent_offer` event after contact answers -- Remove recording-related code from frontend - -### Phase 4: Cutover Strategy - -**Approach:** Feature flag per account. - -```ruby -# In IncomingCallService and CallService: -if account.feature_enabled?('whatsapp_call_server_media') - # New path: delegate to media server -else - # Legacy path: browser-direct WebRTC -end -``` - -```javascript -// In useWhatsappCallSession.js: -const useServerMedia = computed(() => - currentAccount.value?.features?.includes('whatsapp_call_server_media') -); -``` - -This allows: -1. Gradual rollout to test accounts first -2. Instant rollback by disabling the feature flag -3. Both paths coexist during the migration period -4. Legacy path removed once all accounts are migrated - -### Phase 5: Cleanup (1 week) - -After all accounts are migrated: -- Remove browser-direct WebRTC code path -- Remove `upload_recording` endpoint -- Remove `MediaRecorder` and `startCallRecording` from frontend -- Remove feature flag check (`whatsapp_call_server_media` becomes default) -- Update documentation - -### Timeline Summary - -| Phase | Duration | Deliverable | -|-------|----------|-----------------------------------------------------| -| 1 | 2-3 wks | Standalone media server binary, tested | -| 2 | 1-2 wks | Rails integration, new endpoints | -| 3 | 1-2 wks | Frontend migration, reconnection | -| 4 | 1 wk | Feature-flagged rollout, validation | -| 5 | 1 wk | Cleanup legacy code | -| **Total** | **6-9 wks** | **Full migration complete** | - ---- - -## Appendix A: Alternative Approaches Considered and Rejected - -### A1. WebSocket Audio Streaming (No Peer B WebRTC) - -Instead of a second WebRTC peer connection to the browser, stream raw audio over -WebSocket. - -**Rejected because:** -- 3-5x higher latency (WebSocket has no jitter buffer, no adaptive bitrate) -- Must build custom audio decode/playback pipeline in browser -- No browser echo cancellation (WebRTC AEC is tied to RTCPeerConnection) -- No NAT traversal (WebSocket is TCP, goes through proxies) -- Significantly more frontend code complexity - -### A2. SIP Trunking (No WebRTC to Meta) - -Meta supports SIP as an alternative to WebRTC for the business endpoint. Use a SIP -server (Asterisk/FreeSWITCH) instead of WebRTC. - -**Rejected for initial implementation because:** -- Adds another large infrastructure component (Asterisk/FreeSWITCH) -- SIP requires dedicated infrastructure for NAT traversal (SIP ALG, RTP proxy) -- More complex debugging (SIP signaling + RTP + codec negotiation) -- May be a good Phase 2 option for enterprises already running PBX infrastructure - -### A3. Browser Service Worker for Call Persistence - -Use a Service Worker to hold the WebRTC peer connection, surviving page reloads. - -**Rejected because:** -- Service Workers cannot access `RTCPeerConnection` (not available in SW context) -- `SharedWorker` can hold JS objects across tabs but not across page reloads -- No browser API supports WebRTC connections that survive navigation -- This is a fundamental browser architecture limitation, not a coding problem - -### A4. Iframe-Based WebRTC Isolation - -Put WebRTC code in a hidden iframe that persists across SPA navigation. - -**Rejected because:** -- Only works for SPA navigation (Vue Router), not actual page reloads -- Fragile: iframe can be garbage-collected by browser memory pressure -- Does not solve recording reliability (still browser-dependent) -- Adds significant complexity for marginal benefit - ---- - -## Appendix B: Media Server API Specification (Draft) - -### POST /sessions - -Create a new call session. - -**Request:** -```json -{ - "call_id": "chatwoot_call_42", - "account_id": 1, - "direction": "incoming", - "meta_sdp_offer": "v=0\r\no=- ...", - "ice_servers": [ - {"urls": "stun:stun.l.google.com:19302"}, - {"urls": "turn:turn.example.com:3478", "username": "user", "credential": "pass"} - ] -} -``` - -**Response (201):** -```json -{ - "session_id": "sess_abc123", - "meta_sdp_answer": "v=0\r\no=- ...", - "status": "meta_peer_ready" -} -``` - -For incoming calls, the response includes the SDP answer to send to Meta. For -outgoing calls, `meta_sdp_offer` is null initially; the server generates an offer -that Rails sends to Meta via `initiate_call`. - -### POST /sessions/:id/agent-offer - -Generate an SDP offer for the agent-side peer connection. - -**Response (200):** -```json -{ - "sdp_offer": "v=0\r\no=- ...", - "ice_servers": [...] -} -``` - -### POST /sessions/:id/agent-answer - -Set the agent's SDP answer and begin audio bridging. - -**Request:** -```json -{ - "sdp_answer": "v=0\r\no=- ..." -} -``` - -**Response (200):** -```json -{ - "status": "bridged", - "recording": true -} -``` - -### POST /sessions/:id/agent-reconnect - -Tear down old agent peer and create a new one. - -**Response (200):** -```json -{ - "sdp_offer": "v=0\r\no=- ...", - "ice_servers": [...] -} -``` - -### POST /sessions/:id/terminate - -End the session, finalize recording. - -**Response (200):** -```json -{ - "status": "terminated", - "recording_file": "sess_abc123.ogg", - "recording_size_bytes": 245760, - "duration_seconds": 135 -} -``` - -### GET /sessions/:id/recording - -Download the finalized recording file. - -**Response:** Binary OGG/Opus file with `Content-Type: audio/ogg`. - -### GET /health - -**Response (200):** -```json -{ - "status": "ok", - "active_sessions": 12, - "uptime_seconds": 86400 -} -``` diff --git a/docs/SERVER_SIDE_WEBRTC_IMPLEMENTATION_PLAN.md b/docs/SERVER_SIDE_WEBRTC_IMPLEMENTATION_PLAN.md deleted file mode 100644 index 8b665098c..000000000 --- a/docs/SERVER_SIDE_WEBRTC_IMPLEMENTATION_PLAN.md +++ /dev/null @@ -1,1314 +0,0 @@ -# Server-Side WebRTC Migration -- Implementation Plan & Tracking - -> **Feature branch:** `feat/whatsapp-call` | **Base branch:** `develop` -> **Feature flag (existing):** `whatsapp_call` | **Feature flag (new):** `whatsapp_call_server_media` -> **Created:** 2026-04-21 | **Status:** Planning - ---- - -## Table of Contents - -1. [Feature Requirements](#1-feature-requirements) -2. [Architecture Decision: WebRTC Peer B](#2-architecture-decision-webrtc-peer-b-not-websocket-audio) -3. [Current State Assessment](#3-current-state-assessment) -4. [Implementation Phases](#4-implementation-phases) - - [Phase 1: Foundation](#phase-1-foundation--database--configuration--go-scaffold) - - [Phase 2: Core Call Flow](#phase-2-core-call-flow) - - [Phase 3: Recording](#phase-3-server-side-recording) - - [Phase 4: Call Persistence & Reconnection](#phase-4-call-persistence--reconnection) - - [Phase 5: Multi-Participant](#phase-5-multi-participant) - - [Phase 6: Audio Injection](#phase-6-audio-injection--hold-music) - - [Phase 7: AI Foundation](#phase-7-ai-captain-foundation) -5. [API Surface Changes](#5-api-surface-changes) -6. [Database Changes](#6-database-changes) -7. [Configuration & Environment Variables](#7-configuration--environment-variables) -8. [Risk Register](#8-risk-register) -9. [Testing Strategy](#9-testing-strategy) -10. [Rollout Plan](#10-rollout-plan) - ---- - -## 1. Feature Requirements - -Six core requirements drive this migration. Each is a hard prerequisite for considering the project complete. - -### R1: End-to-End Calling Works Correctly - -Inbound and outbound calls must complete the full lifecycle: ring, accept, two-way audio, hang up. The audio path changes from `Browser <-> Meta` to `Browser <-> Media Server <-> Meta`, but from the user's perspective behavior is identical. Latency increase must stay below 10ms (datacenter hop). - -**Acceptance criteria:** -- Agent hears customer and customer hears agent with no perceptible quality loss. -- Outbound call flow (initiate, ring, connect, terminate) works end to end. -- Inbound call flow (webhook, ring, accept, audio, terminate) works end to end. -- SDP negotiation no longer requires the `actpass -> active` string replacement hack. - -### R2: Call Persists When Browser Closes or Refreshes - -The Meta-side WebRTC connection (Peer A) is held by the Go media server. When the agent's browser closes or refreshes, only the agent-side connection (Peer B) drops. The call stays alive for a configurable window (default 30 seconds), and the agent can reconnect automatically on page reload. - -**Acceptance criteria:** -- Agent refreshes the page during an active call. The call does not terminate. -- On reload, the dashboard detects the active call and reconnects within 3 seconds. -- Customer hears silence (or hold tone) during the reconnection gap, not a disconnect. -- If the agent does not reconnect within 30s, the call terminates gracefully. -- The `beforeunload` handler no longer calls `terminateCallOnUnload`. - -### R3: Call Recording Saved to Database / S3 - -Recording happens server-side in the Go media server. Both audio streams (customer via Peer A, agent via Peer B) are captured as OGG/Opus with stereo channel separation (L=customer, R=agent). On call end, Rails fetches the recording file from the media server via internal HTTP and attaches it to ActiveStorage. - -**Acceptance criteria:** -- Recording is captured entirely server-side. No `MediaRecorder`, `AudioContext`, or WebM blob logic in the browser. -- Recording survives agent browser crash (server was recording the whole time). -- Recording is stereo: left channel = customer, right channel = agent. -- Recording is available in ActiveStorage within 60 seconds of call end. -- Existing transcription pipeline (`CallTranscriptionJob`) works with the new OGG/Opus format. - -### R4: Multiple Frontends Can Join the Same Call - -Multiple agent browser tabs or different agents can connect to the same call session. The media server supports multiple Peer B connections with roles: `active` (sends and receives audio), `listen_only` (receives audio, mic muted server-side), and `inject_only` (sends audio only, for playback systems). - -**Acceptance criteria:** -- A supervisor can join an active call in listen-only mode and hear both sides. -- The primary agent's audio is not affected by a listener joining. -- A `call_participants` table tracks who is connected and their role. -- Frontend shows a participant list during active calls. - -### R5: Play Song/Message to Caller - -The media server can inject audio from a file (MP3, OGG, WAV) into the Meta-side peer (Peer A) as RTP packets. This enables hold music when the agent disconnects and pre-recorded announcements. - -**Acceptance criteria:** -- When the agent disconnects, the customer automatically hears hold music instead of silence. -- An API endpoint allows playing an audio file to the caller mid-call. -- Audio injection does not interrupt the agent's audio when the agent is connected. -- Default hold music is configurable per account or inbox. - -### R6: AI Captain Integration Readiness - -The media server architecture supports tapping the audio stream for real-time AI processing. A plugin interface (`AudioConsumer`) allows external systems to receive a copy of the audio in real time. A "virtual peer" concept allows AI to inject audio (e.g., suggested responses, automated greetings) into the call. - -**Acceptance criteria:** -- An `AudioConsumer` plugin interface exists in the Go media server. -- An RTP tap can stream audio to an external service (WebSocket or gRPC). -- A virtual peer can inject synthesized audio into the call. -- Rails has service hooks for AI streaming integration. -- No AI features ship in this migration -- this is the foundation only. - ---- - -## 2. Architecture Decision: WebRTC Peer B (NOT WebSocket Audio) - -### The Decision - -The agent's browser connects to the Go media server via a standard WebRTC peer connection (Peer B). The browser still uses `RTCPeerConnection`, `getUserMedia`, and native audio playback -- but it peers with the Go server, not with Meta directly. - -### Why WebRTC Peer B Over Binary WebSocket Relay - -The alternative considered was streaming raw audio over WebSocket (binary frames of PCM or Opus). This was rejected for five reasons: - -**1. Latency.** WebRTC delivers sub-50ms end-to-end latency via UDP with jitter buffers, adaptive bitrate, and congestion control built into the protocol. WebSocket runs over TCP, which adds 100-200ms of latency due to head-of-line blocking, Nagle's algorithm, and the lack of purpose-built jitter compensation. For real-time voice, this difference is audible. - -**2. Echo cancellation.** Browser echo cancellation (AEC) is tightly coupled to the WebRTC stack. It operates on the `RTCPeerConnection` audio pipeline. With WebSocket audio, there is no native AEC -- it would need to be implemented manually or disabled entirely, causing echo for agents without headsets. - -**3. No custom audio pipeline needed.** WebRTC handles decode, jitter buffering, volume normalization, and playback natively. WebSocket audio requires building all of this in JavaScript: decoding Opus frames, managing a playback buffer, scheduling `AudioContext` output, and handling underruns. This is hundreds of lines of fragile browser code. - -**4. NAT traversal.** WebRTC handles NAT traversal automatically via ICE/STUN/TURN. WebSocket requires the server to be directly addressable on a stable URL (typically behind a reverse proxy), and does not support UDP fallback for restrictive networks. - -**5. Code reuse.** The browser already has WebRTC code for the current direct-to-Meta flow. Switching to Peer B requires minimal frontend changes (different SDP source, remove recording). Switching to WebSocket audio would be a complete rewrite of the audio pipeline. - -### What Changes for the Browser - -| Aspect | Current (direct to Meta) | New (Peer B to media server) | -|--------|--------------------------|------------------------------| -| `RTCPeerConnection` target | Meta media servers | Go media server | -| SDP offer source | Meta webhook payload | Media server (via ActionCable) | -| SDP answer destination | Meta API (via Rails) | Media server (via Rails) | -| `MediaRecorder` | Active (browser records) | Removed (server records) | -| `beforeunload` behavior | Terminates call | Does nothing (call persists) | -| Reconnect on reload | Not possible | Automatic via `GET /active` | - -### What Does NOT Change for the Browser - -- Still uses `getUserMedia({ audio: true })` for microphone access. -- Still creates `RTCPeerConnection` with ICE servers. -- Still uses `ontrack` to play remote audio. -- Still uses `setRemoteDescription` and `createAnswer` for SDP exchange. -- The Vue composable API surface (`acceptCall`, `endActiveCall`, `toggleMute`) stays the same. - ---- - -## 3. Current State Assessment - -### What Exists Today (on `feat/whatsapp-call` branch) - -The browser-direct WhatsApp calling feature is substantially implemented across 9 planned PRs. The following is already built or merged: - -**Merged to `develop`:** -- PR-1: `Call` model, migration (`20260408170902_create_calls.rb`), error classes, feature flag definition. - -**On `feat/whatsapp-call` branch (not yet merged):** -- `Whatsapp::IncomingCallService` -- processes Meta webhooks, creates Call records, broadcasts ActionCable events. -- `Whatsapp::CallService` -- accept/reject/terminate orchestration with `pre_accept_and_accept(sdp_answer)`. -- `Whatsapp::CallMessageBuilder` -- creates `voice_call` message content type. -- `Whatsapp::CallTranscriptionService` + `CallTranscriptionJob` -- OpenAI Whisper integration. -- `Whatsapp::CallPermissionReplyService` -- handles outbound call permission opt-in. -- `WhatsappCallsController` -- 6 endpoints (show, accept, reject, terminate, initiate, upload_recording). -- `WhatsappCloudCallMethods` -- Meta Cloud API provider layer. -- Frontend: API client, Pinia store, WebRTC composable, WhatsappCallWidget, VoiceCall bubble, ActionCable handlers, outbound UI in ConversationHeader. - -### What Needs to Change for Server-Side WebRTC - -The migration modifies existing code rather than replacing it entirely. The key changes are: - -**Rails services (modify):** -- `IncomingCallService`: instead of storing Meta's SDP offer for the browser, create a media server session and store `media_session_id`. -- `CallService#pre_accept_and_accept`: no longer receives `sdp_answer` from the browser. Instead, the SDP answer comes from the media server. Orchestrates Peer B setup after accepting Meta's call. -- Controller: `accept` no longer requires `sdp_answer` parameter. New actions: `active`, `agent_answer`, `reconnect`. Remove `upload_recording`. - -**Rails services (new):** -- `Whatsapp::MediaServerClient` -- HTTP client for the Go media server internal API. -- `Whatsapp::CallReconnectService` -- handles agent reconnection after page reload. -- `Whatsapp::CallRecordingFetchJob` -- Sidekiq job to fetch recording from media server on call end. - -**Frontend (modify):** -- `useWhatsappCallSession.js`: remove `MediaRecorder` logic, remove `terminateCallOnUnload`, add reconnection flow, receive SDP from media server instead of Meta. -- `whatsappCalls.js` store: add reconnection state, handle `agent_offer` event. -- `whatsappCalls.js` API: remove `uploadRecording`, add `agentAnswer`, `reconnect`, `getActiveCall`. -- `ConversationHeader.vue`: outbound initiate no longer sends `sdp_offer` from browser. - -**New component (Go):** -- `chatwoot-media-server` -- Pion-based sidecar handling Peer A (Meta), Peer B (agent), audio bridge, and recording. - -### Known Technical Debt in Current Implementation - -These issues exist in the current branch and should be resolved during or before the migration: - -1. **Duplicated ICE gathering logic** -- `waitForOutboundIceGathering` in ConversationHeader duplicates `waitForIceGatheringComplete` in the composable. With Peer B, ICE gathering moves to the media server for Peer A and stays in the browser only for Peer B. -2. **No `onUnmounted` cleanup** for outbound calls in ConversationHeader. The outbound WebRTC objects (`outboundCall.pc`, `.stream`) leak if the component unmounts mid-call. -3. **Hardcoded STUN server** -- `stun:stun.l.google.com:19302` with no backend-provided configuration. The media server migration introduces proper ICE server configuration via environment variables. -4. **`console.log` left in** -- ICE state logging in ConversationHeader. -5. **Recording mimeType assumption** -- `audio/webm;codecs=opus` with no browser feature detection. Moot after migration (server records). -6. **Bare string in ActionCable** -- `onWhatsappCallPermissionGranted` uses template literal instead of i18n key. -7. **Feature flag unused on frontend** -- `FEATURE_FLAGS.WHATSAPP_CALL` declared but no component checks it. - ---- - -## 4. Implementation Phases - -### Phase 1: Foundation -- Database, Configuration, Go Scaffold - -**Goal:** Infrastructure prerequisites. The media server has a health endpoint, Rails can talk to it, and the database supports media session tracking. - -**Estimated effort:** 1 week - -#### Database Migration - -- [ ] **`db/migrate/YYYYMMDDHHMMSS_add_media_session_id_to_calls.rb`** (NEW) - - Add `media_session_id` string column to `calls` table - - Add index on `media_session_id` - - Migration is backward-compatible (nullable column, no default) - -#### Rails Configuration - -- [ ] **`enterprise/app/services/whatsapp/media_server_client.rb`** (NEW) - - HTTP client wrapping Faraday for communication with the Go media server - - Methods: `create_session`, `meta_sdp`, `agent_offer`, `agent_answer`, `agent_reconnect`, `terminate`, `session_status`, `download_recording`, `health` - - Authentication via Bearer token from `ENV['MEDIA_SERVER_AUTH_TOKEN']` - - Timeout configuration: 5s connect, 30s read (SDP generation may involve ICE gathering) - - Error handling: raise `Whatsapp::CallErrors::MediaServerError` on failures - - Circuit breaker pattern: if health check fails 3 times, stop attempting new sessions - -- [ ] **`config/media_server.yml`** (NEW) or environment variable approach - - `MEDIA_SERVER_URL` (default: `http://localhost:4000`) - - `MEDIA_SERVER_AUTH_TOKEN` - - `MEDIA_SERVER_STUN_SERVERS` (comma-separated, default: `stun:stun.l.google.com:19302`) - - `MEDIA_SERVER_TURN_SERVERS` (optional) - - `MEDIA_SERVER_TURN_USERNAME` (optional) - - `MEDIA_SERVER_TURN_PASSWORD` (optional) - - `MEDIA_SERVER_PUBLIC_IP` (required for ICE candidate in SDP) - -- [ ] **`enterprise/app/models/call.rb`** (MODIFY) - - Add `media_session_id` accessor and convenience methods - - Add `media_session_active?` method that checks media server status - - Add `meta_server_url` method (for multi-instance routing in the future) - -#### Go Media Server Scaffold - -- [ ] **`enterprise/media-server/main.go`** (NEW) - - HTTP server on configurable port (default 4000) - - Graceful shutdown on SIGTERM/SIGINT - - Structured logging (JSON format for production) - -- [ ] **`enterprise/media-server/config/config.go`** (NEW) - - Parse environment variables: `AUTH_TOKEN`, `STUN_SERVERS`, `TURN_SERVERS`, `PUBLIC_IP`, `UDP_PORT_MIN`, `UDP_PORT_MAX`, `RECORDINGS_DIR`, `RAILS_CALLBACK_URL`, `LOG_LEVEL` - -- [ ] **`enterprise/media-server/api/handler.go`** (NEW) - - Route definitions for all endpoints - - `GET /health` returns `{"status": "ok", "active_sessions": N, "uptime_seconds": N}` - -- [ ] **`enterprise/media-server/api/middleware.go`** (NEW) - - Bearer token authentication middleware - - Request logging middleware - - Recovery/panic middleware - -- [ ] **`enterprise/media-server/go.mod`** (NEW) - - Module: `github.com/chatwoot/chatwoot-media-server` - - Dependencies: `pion/webrtc/v4`, `pion/interceptor`, standard library HTTP - -- [ ] **`enterprise/media-server/Dockerfile`** (NEW) - - Multi-stage build: Go build stage + scratch/alpine runtime - - Target image size: under 30 MB - - Expose port 4000 (HTTP) and UDP port range - -- [ ] **`docker-compose.yml`** (MODIFY) or **`docker-compose.deploy.yaml`** (MODIFY) - - Add `media-server` service definition - - UDP port range exposure: `10000-10100:10000-10100/udp` - - Shared volume for recordings - - Health check pointing to `/health` - - Environment variables from `.env` - -- [ ] **`Procfile.dev`** (MODIFY) - - Add media server process: `media: ./enterprise/media-server/chatwoot-media-server` - -#### Verification - -- [ ] `GET /health` returns 200 from inside Docker network -- [ ] `MediaServerClient.new.health` returns successfully from Rails console -- [ ] Migration runs cleanly: `bundle exec rails db:migrate` -- [ ] `Call.new(media_session_id: 'test')` works - ---- - -### Phase 2: Core Call Flow - -**Goal:** Inbound and outbound calls work end-to-end through the media server. The browser peers with the Go server, not Meta directly. - -**Estimated effort:** 2-3 weeks - -#### Go Media Server: Peer A (Meta-side WebRTC) - -- [ ] **`enterprise/media-server/session/manager.go`** (NEW) - - Thread-safe session map (sync.RWMutex) - - `CreateSession(callID, metaSDP, iceServers)` -- creates session and Peer A - - `FindSession(sessionID)` -- lookup by ID - - `DestroySession(sessionID)` -- teardown both peers - - Periodic cleanup goroutine: destroy sessions idle for >2 hours - - Session creation rate limit per account_id - -- [ ] **`enterprise/media-server/session/session.go`** (NEW) - - Holds Peer A, Peer B (optional), bridge, recording state - - State machine: `created -> meta_peer_ready -> bridged -> agent_disconnected -> terminated` - - `SetMetaSDP(offer) -> answer` -- set Meta's SDP on Peer A, return generated answer - - `CreateAgentOffer() -> sdpOffer` -- create Peer B, generate SDP offer for agent - - `SetAgentAnswer(answer)` -- complete Peer B, start bridge and recording - - `AgentReconnect() -> sdpOffer` -- tear down old Peer B, create new one - - `Terminate()` -- close both peers, finalize recording, notify Rails - -- [ ] **`enterprise/media-server/session/peer.go`** (NEW) - - Wraps `pion/webrtc.PeerConnection` - - Configures audio-only media (no video) - - ICE candidate handling: trickle or full gather (configurable) - - ICE candidate filtering: strip `host` candidates from agent-facing offers - - `OnTrack` callback registration for the bridge - - `OnICEConnectionStateChange` callback for disconnect detection - -- [ ] **`enterprise/media-server/session/bridge.go`** (NEW) - - Forward RTP packets from Peer A remote track to Peer B local track - - Forward RTP packets from Peer B remote track to Peer A local track - - Tap both streams for the recording pipeline - - Packet copying (not pointer sharing) to avoid races between bridge and recorder - - Graceful handling of one peer disconnecting while the other stays up - -#### Go Media Server: Peer B (Agent-side WebRTC) - -- [ ] **Peer B creation in `session.go`** (part of session implementation above) - - Create `PeerConnection` with STUN/TURN configuration - - Add audio transceiver (sendrecv) - - Generate SDP offer with ICE candidates - - Public IP override in SDP if `MEDIA_SERVER_PUBLIC_IP` is set - - On Peer B ICE failure: transition to `agent_disconnected`, notify Rails via callback - -#### Go Media Server: HTTP API Endpoints - -- [ ] **`enterprise/media-server/api/session_handler.go`** (NEW) - - `POST /sessions` -- create session with Meta SDP, return session_id + SDP answer - - `GET /sessions/:id/status` -- return session state, peer connection states, duration - - `DELETE /sessions/:id` -- force-destroy session - -- [ ] **`enterprise/media-server/api/meta_sdp_handler.go`** (NEW) - - `POST /sessions/:id/meta-sdp` -- for outbound calls: set Meta's SDP answer on Peer A - -- [ ] **`enterprise/media-server/api/agent_handler.go`** (NEW) - - `POST /sessions/:id/agent-offer` -- generate SDP offer for agent browser - - `POST /sessions/:id/agent-answer` -- set agent's SDP answer, start bridge - - `POST /sessions/:id/agent-reconnect` -- tear down old Peer B, create new offer - -- [ ] **`enterprise/media-server/api/recording_handler.go`** (NEW, stub for Phase 3) - - `GET /sessions/:id/recording` -- download recording file (returns 404 until Phase 3) - -- [ ] **`enterprise/media-server/callback/rails_client.go`** (NEW) - - HTTP client for callbacks to Rails - - `POST /api/internal/media_server/callbacks` with events: `agent_disconnected`, `agent_reconnected`, `session_terminated`, `recording_ready` - - Retry logic: 3 attempts with exponential backoff - - Authentication: same shared secret token - -#### Rails: Modify IncomingCallService - -- [ ] **`enterprise/app/services/whatsapp/incoming_call_service.rb`** (MODIFY) - - In `handle_call_connect` for inbound calls: - - After creating the Call record, call `MediaServerClient.create_session` with Meta's SDP offer - - Store `media_session_id` on the Call record - - Store the media server's SDP answer (to send to Meta) instead of Meta's SDP offer (for the browser) - - The SDP answer sent to Meta comes from the media server, not the browser - - Feature-flagged: check `whatsapp_call_server_media` flag to use new path vs legacy - - In `broadcast_incoming_call`: do NOT include `sdp_offer` in the ActionCable payload (the browser no longer needs it for direct peering) - -#### Rails: Modify CallService - -- [ ] **`enterprise/app/services/whatsapp/call_service.rb`** (MODIFY) - - `pre_accept_and_accept` no longer receives `sdp_answer` from the browser - - New flow: - 1. Lock the call row, validate ringing state - 2. Call Meta's `pre_accept_call` with the SDP answer stored in `Call.meta` (from the media server) - 3. Call Meta's `accept_call` with the same SDP answer - 4. Update call status to `in_progress` - 5. Request agent-side SDP offer from media server: `MediaServerClient.agent_offer(media_session_id)` - 6. Broadcast `whatsapp_call.agent_offer` via ActionCable with the SDP offer and ICE servers - - Outbound `initiate` flow: - 1. Create media session (direction: outbound, no Meta SDP yet) - 2. Get SDP offer from media server for Meta - 3. Call Meta's `initiate_call` with the media server's SDP offer - 4. On Meta's webhook with SDP answer, call `MediaServerClient.meta_sdp(session_id, sdp_answer)` to complete Peer A - 5. Then set up Peer B same as inbound - - New method: `agent_answer(sdp_answer)` -- forwards agent's SDP answer to media server - -#### New Controller Actions - -- [ ] **`enterprise/app/controllers/api/v1/accounts/whatsapp_calls_controller.rb`** (MODIFY) - - `GET /active` (NEW action) -- return the agent's currently active call (if any), used for reconnection on page load - ``` - GET /api/v1/accounts/:account_id/whatsapp_calls/active - Response: { id, call_id, status, conversation_id, started_at, duration_seconds } or 204 No Content - ``` - - `POST /:id/agent_answer` (NEW action) -- receive SDP answer from agent browser, forward to media server - ``` - POST /api/v1/accounts/:account_id/whatsapp_calls/:id/agent_answer - Body: { sdp_answer: "..." } - ``` - - `POST /:id/reconnect` (NEW action) -- agent reconnects to an active call after page reload - ``` - POST /api/v1/accounts/:account_id/whatsapp_calls/:id/reconnect - Response: { sdp_offer: "...", ice_servers: [...] } - ``` - - `POST /:id/accept` (MODIFY) -- remove `sdp_answer` requirement from params - - `POST /initiate` (MODIFY) -- remove `sdp_offer` requirement from params - -- [ ] **`config/routes.rb`** (MODIFY) - - Add `member` routes: `agent_answer`, `reconnect` - - Add `collection` route: `active` - -#### Rails: Internal Callbacks Controller - -- [ ] **`enterprise/app/controllers/api/internal/media_server/callbacks_controller.rb`** (NEW) - - Receives events from the Go media server - - Authentication: validate shared secret token - - Events handled: - - `agent_disconnected`: broadcast `whatsapp_call.agent_disconnected` via ActionCable, start reconnect timer - - `recording_ready`: enqueue `CallRecordingFetchJob` - - `session_terminated`: if call not already terminated, update status and broadcast - -- [ ] **`config/routes.rb`** (MODIFY) - - Add internal API namespace for media server callbacks - -#### ActionCable Event Changes - -- [ ] **`app/javascript/dashboard/helper/actionCable.js`** (MODIFY) - - Add handler for `whatsapp_call.agent_offer` -- delivers SDP offer from media server to the agent browser for Peer B setup - - Add handler for `whatsapp_call.agent_disconnected` -- notifies other tabs/agents that the primary agent's connection dropped - - Modify `whatsapp_call.incoming` handler -- no longer includes `sdp_offer` in payload - - Modify `whatsapp_call.outbound_connected` handler -- no longer includes `sdp_answer` in payload (SDP exchange happens via media server) - -#### Frontend Composable Refactor - -- [ ] **`app/javascript/dashboard/composables/useWhatsappCallSession.js`** (MODIFY) - - **Remove:** `startCallRecording`, `stopAndUploadRecording`, `mediaRecorder`, `recordedChunks`, `recordingCallId` -- all recording logic - - **Remove:** `terminateCallOnUnload` -- call persists on page close - - **Modify `doAcceptCall`:** - - No longer receives `sdp_offer` from Meta - - No longer calls `pc.setRemoteDescription` with Meta's offer - - No longer calls `WhatsappCallsAPI.accept(call.id, completeSdp)` - - New flow: call `WhatsappCallsAPI.accept(call.id)` (no SDP), then wait for `agent_offer` ActionCable event - - **Add `handleAgentOffer(sdpOffer, iceServers)`:** - - Called when `whatsapp_call.agent_offer` event arrives - - `getUserMedia({ audio: true })` - - Create `RTCPeerConnection` with provided `iceServers` - - `setRemoteDescription(offer)` with media server's SDP offer - - `createAnswer()` + ICE gathering - - POST SDP answer to `WhatsappCallsAPI.agentAnswer(callId, sdpAnswer)` - - **Modify `beforeunload` handler:** - - Do NOT call `terminateCallOnUnload` - - Just clean up the local `RTCPeerConnection` and `MediaStream` (Peer B dies naturally) - - **Remove** `startCallRecording` call from `pc.ontrack` handler - -- [ ] **`app/javascript/dashboard/stores/whatsappCalls.js`** (MODIFY) - - Add state: `isReconnecting` (boolean), `reconnectAttempts` (number) - - Add action: `handleAgentOffer(payload)` -- stores the SDP offer and triggers the composable - - Add action: `setReconnecting(value)` -- manages reconnection UI state - - Remove: outbound call `sdp_offer` handling (the browser no longer generates the initial offer for outbound) - - Add action: `handleAgentDisconnected(callId)` -- handles notification that agent connection dropped (for multi-tab awareness) - -- [ ] **`app/javascript/dashboard/api/whatsappCalls.js`** (MODIFY) - - `accept(callId)` -- remove `sdpAnswer` parameter - - `initiate(conversationId)` -- remove `sdpOffer` parameter - - **Add:** `agentAnswer(callId, sdpAnswer)` -- POST SDP answer to new endpoint - - **Add:** `reconnect(callId)` -- POST to reconnect endpoint - - **Add:** `getActiveCall()` -- GET active call for current agent - - **Remove:** `uploadRecording(callId, blob)` - -- [ ] **`app/javascript/dashboard/components/widgets/conversation/ConversationHeader.vue`** (MODIFY) - - Remove outbound `RTCPeerConnection` creation and SDP offer generation - - Remove `waitForOutboundIceGathering` (duplicated ICE logic) - - Remove `console.log` statements for ICE state - - Outbound initiate now just calls `WhatsappCallsAPI.initiate(conversationId)` with no SDP - - Wait for `agent_offer` event (same as inbound) after contact answers - -#### Verification - -- [ ] Inbound call: Meta webhook -> media server session -> agent accepts -> audio flows through media server -- [ ] Outbound call: agent initiates -> media server creates offer -> Meta receives -> contact answers -> audio flows -- [ ] SDP exchange does not require `actpass -> active` string replacement -- [ ] Both browser-direct (legacy) and server-media paths work when feature-flagged - ---- - -### Phase 3: Server-Side Recording - -**Goal:** Recording happens entirely on the Go media server. Browser recording code is removed. Recordings are fetched by Rails on call end. - -**Estimated effort:** 1 week - -#### Go Media Server: Recording Pipeline - -- [ ] **`enterprise/media-server/recording/recorder.go`** (NEW) - - Receives decoded Opus frames from both Peer A and Peer B via the bridge's tap - - Writes frames into OGG/Opus container in real time - - Stereo separation: Peer A (customer) on left channel, Peer B (agent) on right channel - - File path: `/recordings/{session_id}.ogg` - - Starts when both peers are connected and bridge is active - - Pauses agent channel (writes silence) when Peer B is disconnected (reconnection gap) - - Finalizes (writes OGG trailer) when session terminates - -- [ ] **`enterprise/media-server/recording/ogg_writer.go`** (NEW) - - Low-level OGG page construction - - Handles page sequencing, checksums, and granule position tracking - - Based on Pion's `oggreader`/`oggwriter` examples - -- [ ] **`enterprise/media-server/recording/cleanup.go`** (NEW) - - On startup: scan `/recordings/` for orphaned files (sessions that no longer exist) - - Report orphaned files to Rails via callback - - Delete orphaned files older than 24 hours - -- [ ] **`enterprise/media-server/api/recording_handler.go`** (MODIFY -- implement the stub from Phase 2) - - `GET /sessions/:id/recording` -- stream the finalized OGG file with `Content-Type: audio/ogg` - - `POST /sessions/:id/terminate` response now includes `recording_file`, `recording_size_bytes`, `duration_seconds` - -#### Rails: Recording Fetch Job - -- [ ] **`enterprise/app/jobs/whatsapp/call_recording_fetch_job.rb`** (NEW) - - Triggered by `recording_ready` callback from media server - - Downloads recording from `MediaServerClient.download_recording(session_id)` - - Attaches to `call.recording` via ActiveStorage - - Updates the message bubble with recording URL via `CallMessageBuilder.update_recording_url!` - - Enqueues `CallTranscriptionJob` after successful attachment - - Retry: 3 attempts with exponential backoff - - On final failure: log error, mark call with `recording_fetch_failed` in meta - -#### Rails: Remove Browser Upload Path - -- [ ] **`enterprise/app/controllers/api/v1/accounts/whatsapp_calls_controller.rb`** (MODIFY) - - Remove `upload_recording` action (behind feature flag -- keep for legacy path) - - Remove `attach_recording_and_enqueue_transcription` private method (behind feature flag) - -#### Frontend: Remove Recording Code - -- [ ] **`app/javascript/dashboard/composables/useWhatsappCallSession.js`** (MODIFY) - - Remove `startCallRecording` export - - Remove `stopAndUploadRecording` function - - Remove `mediaRecorder`, `recordedChunks`, `recordingCallId` module-level variables - - Remove `startCallRecording(pc, stream, call.id)` call from `ontrack` handler - - Remove `stopAndUploadRecording()` from cleanup callback and `endActiveCall` - -- [ ] **`app/javascript/dashboard/api/whatsappCalls.js`** (MODIFY) - - Remove `uploadRecording` method (if not already removed in Phase 2) - -#### Verification - -- [ ] Call completes -> recording appears in ActiveStorage within 60 seconds -- [ ] Recording is OGG/Opus format, playable in browser audio player -- [ ] Recording has stereo channels (customer left, agent right) -- [ ] Transcription job runs successfully on the new format -- [ ] Browser has zero recording-related code when `whatsapp_call_server_media` is enabled -- [ ] Agent browser crash mid-call -> recording is still captured up to the crash point - ---- - -### Phase 4: Call Persistence & Reconnection - -**Goal:** Calls survive browser close/refresh. Automatic reconnection on page reload. - -**Estimated effort:** 1-2 weeks - -#### Go Media Server: Disconnect Detection & Hold - -- [ ] **`enterprise/media-server/session/session.go`** (MODIFY) - - On Peer B ICE connection state `disconnected` or `failed`: - - Transition session state to `agent_disconnected` - - Keep Peer A alive (Meta audio continues to flow into the bridge's buffer) - - Start reconnect timeout (configurable, default 30s) - - Callback to Rails: `agent_disconnected` event - - If hold music is configured (Phase 6), start playing it to Peer A - - On reconnect timeout expiry: - - Callback to Rails: `session_terminated` with reason `agent_reconnect_timeout` - - Terminate session - -- [ ] **`enterprise/media-server/session/session.go`** (MODIFY -- reconnect logic) - - `AgentReconnect()`: - - Tear down old Peer B (close connection, release resources) - - Create new Peer B with fresh SDP offer - - On successful answer: re-bridge audio, resume recording - - Callback to Rails: `agent_reconnected` event - - Increment reconnect counter on session - -#### Rails: Reconnection Support - -- [ ] **`enterprise/app/services/whatsapp/call_reconnect_service.rb`** (NEW) - - Called from `WhatsappCallsController#reconnect` - - Validates: call is `in_progress`, call has `media_session_id`, requesting agent is the accepted agent - - Calls `MediaServerClient.agent_reconnect(media_session_id)` - - Returns new SDP offer for the agent browser - - Broadcasts `whatsapp_call.agent_offer` via ActionCable - -- [ ] **`enterprise/app/controllers/api/v1/accounts/whatsapp_calls_controller.rb`** (MODIFY -- `active` action) - - `GET /active`: find the current user's active call (status `in_progress`, `accepted_by_agent_id` = current_user.id) - - Return call details including `media_session_id`, `started_at`, `conversation_id` - - Used by the frontend on page load to detect if a call needs reconnection - -#### Frontend: Reconnection Composable - -- [ ] **`app/javascript/dashboard/composables/useCallReconnection.js`** (NEW) - - On mount: call `WhatsappCallsAPI.getActiveCall()` - - If active call exists: - - Set store state: `isReconnecting = true` - - Show reconnection UI: "Call in progress -- Reconnecting..." - - Call `WhatsappCallsAPI.reconnect(callId)` - - Handle `agent_offer` event (same flow as initial accept) - - On success: `isReconnecting = false`, show active call UI with correct timer - - On failure: offer "Retry" button or end call - - Timer continuity: calculate elapsed time from `started_at` returned by server, not local state - -- [ ] **`app/javascript/dashboard/composables/useWhatsappCallSession.js`** (MODIFY) - - **Remove `terminateCallOnUnload`** function entirely - - **Modify `handleBeforeUnload`:** - - Do NOT terminate the call - - Clean up local WebRTC resources (close `RTCPeerConnection`, stop `MediaStream` tracks) - - The media server detects Peer B disconnect via ICE and handles it - - **Add reconnection integration:** - - On composable mount, check for active call via `useCallReconnection` - - If reconnecting, skip the incoming call widget and go straight to active call UI - -- [ ] **`app/javascript/dashboard/stores/whatsappCalls.js`** (MODIFY) - - Add `isReconnecting` state - - Add `reconnectAttempts` counter - - `handleAgentDisconnected(callId)`: set `isReconnecting = true` if this is our active call - -- [ ] **`app/javascript/dashboard/components/widgets/WhatsappCallWidget.vue`** (MODIFY) - - Add reconnecting state to the widget UI - - Show "Reconnecting..." with a spinner instead of the normal call controls - - Display server-tracked call duration (from `started_at`) instead of local timer - -#### Verification - -- [ ] Agent refreshes page during active call -> call does not terminate -- [ ] On reload, dashboard shows "Reconnecting..." and audio resumes within 3 seconds -- [ ] Call duration timer is continuous (no reset on reconnect) -- [ ] Customer hears silence (or hold tone) during the 1-3 second gap -- [ ] If agent does not reconnect within 30s, call terminates and Meta receives hangup -- [ ] Multiple reconnections work (refresh 5 times in a row) -- [ ] Recording is continuous across reconnections (single file, agent channel silent during gaps) - ---- - -### Phase 5: Multi-Participant - -**Goal:** Multiple agents can join the same call with different roles. - -**Estimated effort:** 2 weeks - -#### Database - -- [ ] **`db/migrate/YYYYMMDDHHMMSS_create_call_participants.rb`** (NEW) - ``` - create_table :call_participants do |t| - t.bigint :call_id, null: false - t.bigint :user_id, null: false - t.string :role, null: false, default: 'active' # active, listen_only, inject_only - t.string :peer_id # media server peer identifier - t.datetime :joined_at - t.datetime :left_at - t.timestamps - end - add_index :call_participants, [:call_id, :user_id], unique: true - add_index :call_participants, :call_id - ``` - -- [ ] **`enterprise/app/models/call_participant.rb`** (NEW) - - `belongs_to :call` - - `belongs_to :user` - - Validations: role inclusion, uniqueness of user per call - - Scopes: `active`, `connected` (where `left_at` is nil) - -- [ ] **`enterprise/app/models/call.rb`** (MODIFY) - - `has_many :call_participants` - - `has_many :participants, through: :call_participants, source: :user` - -#### Go Media Server: Multi-Peer Support - -- [ ] **`enterprise/media-server/session/session.go`** (MODIFY) - - Support multiple Peer B connections per session (keyed by `peer_id`) - - Each peer has a role: - - `active`: bidirectional audio (current single-agent behavior) - - `listen_only`: receives mixed audio from Peer A + all active peers, does not transmit - - `inject_only`: can send audio into the mix but does not receive - - Audio bridge modification: mix audio from Peer A + all active peer B connections, send mixed output to each peer - -- [ ] **`enterprise/media-server/session/mixer.go`** (NEW) - - Audio mixer: combine Opus frames from multiple sources - - For each output peer: mix all other sources (exclude self to prevent echo) - - Efficient: only active when multiple active peers exist, otherwise pass-through - -#### Rails: Join/Leave Endpoints - -- [ ] **`enterprise/app/controllers/api/v1/accounts/whatsapp_calls_controller.rb`** (MODIFY) - - `POST /:id/join` -- join an active call as listener or active participant - - `POST /:id/leave` -- leave the call (close your Peer B) - - `GET /:id/participants` -- list current participants and their roles - -- [ ] **`enterprise/app/services/whatsapp/call_participant_service.rb`** (NEW) - - `join(call, user, role)`: create participant record, request peer from media server - - `leave(call, user)`: update participant record, disconnect peer on media server - - Authorization: only agents with access to the conversation can join - -#### Frontend: Participant List - -- [ ] **`app/javascript/dashboard/components/widgets/WhatsappCallParticipants.vue`** (NEW) - - Show list of connected participants with their roles - - "Join as listener" button for supervisors - - Uses Tailwind for styling, Composition API with `