chatwoot/stats.md

Captain Assistant Metrics — Exploration

Candidate metrics for a per-assistant overview page, sourced from messages, conversations, reporting_events, and the captain_* tables.

Grounding facts

• Assistant → messages: Captain::Assistant has_many :messages, as: :sender → Captain messages are rows in messages with sender_type = 'Captain::Assistant', sender_id = <assistant.id>. Most reliable per-assistant attribution.
• Assistant → inboxes → conversations: captain_inboxes(captain_assistant_id, inbox_id).
• Reporting events (in reporting_events, scoped by conversation_id/inbox_id, no assistant column): conversation_captain_inference_resolved, conversation_captain_inference_handoff, conversation_bot_resolved, conversation_bot_handoff, conversation_resolved, conversation_opened (value > 0 = a reopen).
• Enums: messages.message_type → incoming:0, outgoing:1; conversations.status → open:0, resolved:1, pending:2, snoozed:3; captain_assistant_responses.status → pending:0, approved:1.

All queries take :assistant_id, :account_id, and a :start/:end window.

---

Conversations Handled

Definition: How many distinct conversations this assistant actually participated in (sent at least one message). Your denominator for every rate below, and the headline "is anyone using this assistant" volume number.

SELECT COUNT(DISTINCT m.conversation_id) AS conversations_handled
FROM messages m
WHERE m.account_id = :account_id
  AND m.sender_type = 'Captain::Assistant'
  AND m.sender_id   = :assistant_id
  AND m.created_at BETWEEN :start AND :end;

Auto-Resolution (Deflection) Rate

Definition: Of the conversations Captain handled, the share it closed on its own with no human reply. This is the core ROI/deflection signal — higher means more tickets the team never had to touch.

WITH handled AS (
  SELECT DISTINCT conversation_id
  FROM messages
  WHERE account_id = :account_id
    AND sender_type = 'Captain::Assistant'
    AND sender_id   = :assistant_id
    AND created_at BETWEEN :start AND :end
)
SELECT
  COUNT(DISTINCT re.conversation_id)::float
    / NULLIF((SELECT COUNT(*) FROM handled), 0) AS auto_resolution_rate
FROM reporting_events re
JOIN handled h ON h.conversation_id = re.conversation_id
WHERE re.account_id = :account_id
  AND re.name IN ('conversation_captain_inference_resolved',
                  'conversation_bot_resolved');

Handoff Rate

Definition: Of the conversations Captain handled, the share it escalated to a human (either an explicit handoff tool call mid-chat, or the auto-resolve job deciding the customer still needs clarification). Inverse signal to deflection — tells you how often Captain hit its limits.

WITH handled AS (
  SELECT DISTINCT conversation_id
  FROM messages
  WHERE account_id = :account_id
    AND sender_type = 'Captain::Assistant'
    AND sender_id   = :assistant_id
    AND created_at BETWEEN :start AND :end
)
SELECT
  COUNT(DISTINCT re.conversation_id)::float
    / NULLIF((SELECT COUNT(*) FROM handled), 0) AS handoff_rate
FROM reporting_events re
JOIN handled h ON h.conversation_id = re.conversation_id
WHERE re.account_id = :account_id
  AND re.name IN ('conversation_captain_inference_handoff',
                  'conversation_bot_handoff');

Reopen-After-Auto-Resolve Rate (premature closures)

Definition: Of the conversations Captain auto-resolved, the share that got reopened afterwards. A quality/over-eagerness signal — high values mean Captain is closing tickets the customer wasn't actually done with. (conversation_opened with value > 0 is a reopen, not the first open.)

WITH resolved AS (
  SELECT DISTINCT re.conversation_id
  FROM reporting_events re
  JOIN captain_inboxes ci ON ci.inbox_id = re.inbox_id
  WHERE re.account_id = :account_id
    AND ci.captain_assistant_id = :assistant_id
    AND re.name = 'conversation_captain_inference_resolved'
    AND re.created_at BETWEEN :start AND :end
)
SELECT
  COUNT(DISTINCT o.conversation_id)::float
    / NULLIF((SELECT COUNT(*) FROM resolved), 0) AS reopen_rate
FROM reporting_events o
JOIN resolved r ON r.conversation_id = o.conversation_id
WHERE o.account_id = :account_id
  AND o.name = 'conversation_opened'
  AND o.value > 0;

Flagged-Response Rate

Definition: How often human agents reported a Captain message as bad (captain_message_reports), as a share of the public answers Captain produced. A direct human-in-the-loop quality score — rising values mean agents are losing trust in the answers.

WITH captain_msgs AS (
  SELECT id
  FROM messages
  WHERE account_id = :account_id
    AND sender_type = 'Captain::Assistant'
    AND sender_id   = :assistant_id
    AND message_type = 1
    AND private = false
    AND created_at BETWEEN :start AND :end
)
SELECT
  (SELECT COUNT(*) FROM captain_message_reports r
     WHERE r.message_id IN (SELECT id FROM captain_msgs))::float
  / NULLIF((SELECT COUNT(*) FROM captain_msgs), 0) AS flagged_rate;

Conversation Depth (messages per handled conversation)

Definition: Average number of public replies Captain sends per conversation it handles. Low (~1) = quick one-shot answers; high = Captain is grinding through long back-and-forths, which often correlates with the cases it ends up handing off.

SELECT
  COUNT(*)::float / NULLIF(COUNT(DISTINCT conversation_id), 0) AS msgs_per_conversation
FROM messages
WHERE account_id = :account_id
  AND sender_type = 'Captain::Assistant'
  AND sender_id   = :assistant_id
  AND message_type = 1
  AND private = false
  AND created_at BETWEEN :start AND :end;

Hours Saved

Definition: A headline "what did Captain save us" number for execs: the volume of replies Captain sent multiplied by the team's average response time. Reads as agent time deflected.

Caveat (important): the multiplier here is the team's average response time (first_response / reply_time), which is how long a customer waited, not how long an agent actually worked on a reply. Wait time is typically much larger than handling time, so this number overstates true labor saved by a wide margin and should be presented as a directional/estimate figure, not measured labor. Counting every Captain message (including ones in conversations later handed off to a human) also inflates it. See the conversation that produced this file for the more conservative "deflected conversations × handling time" alternative if a defensible number is needed.

SELECT
  (
    SELECT COUNT(*)
    FROM messages m
    WHERE m.account_id = :account_id
      AND m.sender_type = 'Captain::Assistant'
      AND m.sender_id   = :assistant_id
      AND m.message_type = 1
      AND m.private = false
      AND m.created_at BETWEEN :start AND :end
  )
  * (
    SELECT AVG(re.value)
    FROM reporting_events re
    WHERE re.account_id = :account_id
      AND re.name = 'reply_time'
      AND re.created_at BETWEEN :start AND :end
  ) / 3600.0 AS hours_saved;

Knowledge Base Coverage

Definition: How much knowledge backs this assistant — approved vs. still-pending FAQ responses, plus synced documents. Not a performance metric but the leading indicator: thin/low-approval knowledge usually explains a low deflection rate. (Time-independent; it's the assistant's current state.)

SELECT
  COUNT(*) FILTER (WHERE status = 1) AS approved_responses,
  COUNT(*) FILTER (WHERE status = 0) AS pending_responses,
  (SELECT COUNT(*) FROM captain_documents d
     WHERE d.assistant_id = :assistant_id) AS documents
FROM captain_assistant_responses
WHERE assistant_id = :assistant_id;

---

Notes for building the overview page

• Two attribution paths exist and answer slightly different questions. Message-based (sender_id = assistant) = "conversations Captain actually engaged." Inbox-based (captain_inboxes) = "conversations that flowed through Captain's inbox, engaged or not." Message-based is used for the handled/rate metrics (tighter); inbox-based only for the reopen metric since inference_resolved is emitted by the auto-resolve job. If an inbox has exactly one assistant, the two converge.
• reporting_events has no assistant_id — everything is joined back through conversation_id or inbox_id. Fine today, but multi-assistant-per-inbox support would want assistant attribution stamped on the event at write time.
• value on the captain inference events is "seconds from conversation creation to the event" (see create_captain_inference_event), so time-to-resolve / time-to-handoff are almost free if you want a couple more timing metrics.