From 5c39d896d9356873d968997f76b83b032ae44082 Mon Sep 17 00:00:00 2001 From: Shivam Mishra Date: Tue, 30 Jun 2026 15:22:12 +0530 Subject: [PATCH] feat: remove scratch files --- queries.md | 214 ++++++++++++++++++++++++++++++++++++++++++++++++++ stats.md | 224 ----------------------------------------------------- 2 files changed, 214 insertions(+), 224 deletions(-) create mode 100644 queries.md delete mode 100644 stats.md diff --git a/queries.md b/queries.md new file mode 100644 index 000000000..50a71f543 --- /dev/null +++ b/queries.md @@ -0,0 +1,214 @@ +# Captain Overview — effective SQL per stat + +These are the queries `Captain::AssistantStatsBuilder` effectively runs for +`account_id = 1`, `assistant_id = 1`, over a **30-day current window** +(`created_at >= NOW() - INTERVAL '30 days'`). Each metric runs once for the +current window and once for the previous equal-length window +(`NOW() - INTERVAL '60 days'` to `NOW() - INTERVAL '30 days'`); only the current +window is shown below. Swap the interval bounds for the previous window. + +Knowledge stats are point-in-time (no window, no previous). + +--- + +## 1. Conversations handled + +Distinct conversations the assistant authored any message in. + +```sql +SELECT COUNT(DISTINCT conversation_id) +FROM messages +WHERE account_id = 1 + AND sender_type = 'Captain::Assistant' + AND sender_id = 1 + AND created_at >= NOW() - INTERVAL '30 days'; +``` + +Indexes that matter: `messages (account_id, ...)`, plus the `sender_type/sender_id` +filter. This is the "handled set" reused as a subquery by stats 2 and 3. + +--- + +## 2 & 3. Auto-resolution rate + handoff rate (combined) + +`auto_resolution = resolved / handled * 100` and `handoff = handoff / handled * 100` +share the same handled-set subquery, so they fold into a single pass with +conditional aggregation. This runs the (costly) `messages` handled-set subquery +**once** and scans `reporting_events` once, instead of twice each. + +```sql +SELECT + COUNT(DISTINCT conversation_id) + FILTER (WHERE name IN ('conversation_captain_inference_resolved', + 'conversation_bot_resolved')) AS resolved, + COUNT(DISTINCT conversation_id) + FILTER (WHERE name IN ('conversation_captain_inference_handoff', + 'conversation_bot_handoff')) AS handoff +FROM reporting_events +WHERE account_id = 1 + AND name IN ('conversation_captain_inference_resolved', 'conversation_bot_resolved', + 'conversation_captain_inference_handoff', 'conversation_bot_handoff') + AND conversation_id IN ( + SELECT conversation_id + FROM messages + WHERE account_id = 1 + AND sender_type = 'Captain::Assistant' + AND sender_id = 1 + AND created_at >= NOW() - INTERVAL '30 days' + ); +``` + +The rates are still computed in Ruby (`resolved / handled`, `handoff / handled`), +so divide-by-zero guarding stays in the builder. The two rates do **not** sum to +100%: a handled conversation may carry neither event (still open) or both +(handed off then later resolved). + +Note: the event rows themselves are **not** date-filtered; the window is applied +via the handled-set subquery (the conversations the assistant touched in the +window). + +### Optional: fold in "handled" too (one query for all three) + +Materialize the handled set once in a CTE and derive all three numbers from it. +Useful because the page also displays `handled`, and every metric runs for both +the current and previous window. + +```sql +WITH handled AS ( + SELECT DISTINCT conversation_id + FROM messages + WHERE account_id = 1 + AND sender_type = 'Captain::Assistant' + AND sender_id = 1 + AND created_at >= NOW() - INTERVAL '30 days' +) +SELECT + (SELECT COUNT(*) FROM handled) AS handled, + COUNT(DISTINCT re.conversation_id) + FILTER (WHERE re.name IN ('conversation_captain_inference_resolved', + 'conversation_bot_resolved')) AS resolved, + COUNT(DISTINCT re.conversation_id) + FILTER (WHERE re.name IN ('conversation_captain_inference_handoff', + 'conversation_bot_handoff')) AS handoff +FROM reporting_events re +JOIN handled h ON h.conversation_id = re.conversation_id +WHERE re.account_id = 1 + AND re.name IN ('conversation_captain_inference_resolved', 'conversation_bot_resolved', + 'conversation_captain_inference_handoff', 'conversation_bot_handoff'); +``` + +Keep `handled` sourced from the CTE count (not `COUNT(*)` over the join), so +conversations with no event row still count toward `handled`. + +--- + +## 4. Hours saved + +`public_outgoing_count * avg_reply_time_seconds / 3600`, rounded. Two queries. + +Public outgoing replies the assistant sent: + +```sql +SELECT COUNT(*) +FROM messages +WHERE account_id = 1 + AND sender_type = 'Captain::Assistant' + AND sender_id = 1 + AND message_type = 1 -- outgoing + AND private = false + AND created_at >= NOW() - INTERVAL '30 days'; +``` + +Average reply time across the account in the window (seconds): + +```sql +SELECT AVG(value) +FROM reporting_events +WHERE account_id = 1 + AND name = 'reply_time' + AND created_at >= NOW() - INTERVAL '30 days'; +``` + +--- + +## 5. Conversation depth (messages / conversation) + +`public_outgoing_count / distinct_conversations`. Reuses query 4's count plus: + +```sql +SELECT COUNT(DISTINCT conversation_id) +FROM messages +WHERE account_id = 1 + AND sender_type = 'Captain::Assistant' + AND sender_id = 1 + AND message_type = 1 -- outgoing + AND private = false + AND created_at >= NOW() - INTERVAL '30 days'; +``` + +--- + +## 6. Reopen-after-resolve rate + +`reopened / resolved * 100`, where "resolved" is the inbox-based Captain-resolved +set. Two queries. + +Resolved (inbox-scoped) in the window: + +```sql +SELECT COUNT(DISTINCT conversation_id) +FROM reporting_events +WHERE account_id = 1 + AND name = 'conversation_captain_inference_resolved' + AND inbox_id IN ( + SELECT inbox_id FROM captain_inboxes WHERE captain_assistant_id = 1 + ) + AND created_at >= NOW() - INTERVAL '30 days'; +``` + +Of those, the ones later reopened (`conversation_opened` with `value > 0`): + +```sql +SELECT COUNT(DISTINCT conversation_id) +FROM reporting_events +WHERE account_id = 1 + AND name = 'conversation_opened' + AND value > 0 + AND conversation_id IN ( + SELECT conversation_id + FROM reporting_events + WHERE account_id = 1 + AND name = 'conversation_captain_inference_resolved' + AND inbox_id IN ( + SELECT inbox_id FROM captain_inboxes WHERE captain_assistant_id = 1 + ) + AND created_at >= NOW() - INTERVAL '30 days' + ); +``` + +Note: the `conversation_opened` events are **not** date-filtered; a reopen counts +whenever it happened, as long as the resolve fell in the window. + +--- + +## 7. Knowledge (point-in-time, no window) + +Approved / pending FAQ counts: + +```sql +SELECT status, COUNT(*) +FROM captain_assistant_responses +WHERE assistant_id = 1 +GROUP BY status; +-- status: 0 = pending, 1 = approved +``` + +Document count: + +```sql +SELECT COUNT(*) +FROM captain_documents +WHERE assistant_id = 1; +``` + +`coverage = approved / (approved + pending) * 100`, computed in Ruby. diff --git a/stats.md b/stats.md deleted file mode 100644 index 67348418e..000000000 --- a/stats.md +++ /dev/null @@ -1,224 +0,0 @@ -# Captain Assistant Metrics — Exploration - -Candidate metrics for a per-assistant overview page, sourced from `messages`, -`conversations`, `reporting_events`, and the `captain_*` tables. - -## Grounding facts - -- **Assistant → messages**: `Captain::Assistant has_many :messages, as: :sender` → Captain - messages are rows in `messages` with `sender_type = 'Captain::Assistant'`, - `sender_id = `. Most reliable per-assistant attribution. -- **Assistant → inboxes → conversations**: `captain_inboxes(captain_assistant_id, inbox_id)`. -- **Reporting events** (in `reporting_events`, scoped by `conversation_id`/`inbox_id`, no - assistant column): `conversation_captain_inference_resolved`, - `conversation_captain_inference_handoff`, `conversation_bot_resolved`, - `conversation_bot_handoff`, `conversation_resolved`, `conversation_opened` - (value > 0 = a reopen). -- **Enums**: `messages.message_type` → `incoming:0, outgoing:1`; `conversations.status` → - `open:0, resolved:1, pending:2, snoozed:3`; `captain_assistant_responses.status` → - `pending:0, approved:1`. - -All queries take `:assistant_id`, `:account_id`, and a `:start`/`:end` window. - ---- - -## Conversations Handled - -Definition: How many distinct conversations this assistant actually participated in (sent at -least one message). Your denominator for every rate below, and the headline "is anyone using -this assistant" volume number. - -```sql -SELECT COUNT(DISTINCT m.conversation_id) AS conversations_handled -FROM messages m -WHERE m.account_id = :account_id - AND m.sender_type = 'Captain::Assistant' - AND m.sender_id = :assistant_id - AND m.created_at BETWEEN :start AND :end; -``` - -## Auto-Resolution (Deflection) Rate - -Definition: Of the conversations Captain handled, the share it closed on its own with no human -reply. This is the core ROI/deflection signal — higher means more tickets the team never had -to touch. - -```sql -WITH handled AS ( - SELECT DISTINCT conversation_id - FROM messages - WHERE account_id = :account_id - AND sender_type = 'Captain::Assistant' - AND sender_id = :assistant_id - AND created_at BETWEEN :start AND :end -) -SELECT - COUNT(DISTINCT re.conversation_id)::float - / NULLIF((SELECT COUNT(*) FROM handled), 0) AS auto_resolution_rate -FROM reporting_events re -JOIN handled h ON h.conversation_id = re.conversation_id -WHERE re.account_id = :account_id - AND re.name IN ('conversation_captain_inference_resolved', - 'conversation_bot_resolved'); -``` - -## Handoff Rate - -Definition: Of the conversations Captain handled, the share it escalated to a human (either an -explicit handoff tool call mid-chat, or the auto-resolve job deciding the customer still needs -clarification). Inverse signal to deflection — tells you how often Captain hit its limits. - -```sql -WITH handled AS ( - SELECT DISTINCT conversation_id - FROM messages - WHERE account_id = :account_id - AND sender_type = 'Captain::Assistant' - AND sender_id = :assistant_id - AND created_at BETWEEN :start AND :end -) -SELECT - COUNT(DISTINCT re.conversation_id)::float - / NULLIF((SELECT COUNT(*) FROM handled), 0) AS handoff_rate -FROM reporting_events re -JOIN handled h ON h.conversation_id = re.conversation_id -WHERE re.account_id = :account_id - AND re.name IN ('conversation_captain_inference_handoff', - 'conversation_bot_handoff'); -``` - -## Reopen-After-Auto-Resolve Rate (premature closures) - -Definition: Of the conversations Captain auto-resolved, the share that got reopened afterwards. -A quality/over-eagerness signal — high values mean Captain is closing tickets the customer -wasn't actually done with. (`conversation_opened` with `value > 0` is a reopen, not the first -open.) - -```sql -WITH resolved AS ( - SELECT DISTINCT re.conversation_id - FROM reporting_events re - JOIN captain_inboxes ci ON ci.inbox_id = re.inbox_id - WHERE re.account_id = :account_id - AND ci.captain_assistant_id = :assistant_id - AND re.name = 'conversation_captain_inference_resolved' - AND re.created_at BETWEEN :start AND :end -) -SELECT - COUNT(DISTINCT o.conversation_id)::float - / NULLIF((SELECT COUNT(*) FROM resolved), 0) AS reopen_rate -FROM reporting_events o -JOIN resolved r ON r.conversation_id = o.conversation_id -WHERE o.account_id = :account_id - AND o.name = 'conversation_opened' - AND o.value > 0; -``` - -## Flagged-Response Rate - -Definition: How often human agents reported a Captain message as bad -(`captain_message_reports`), as a share of the public answers Captain produced. A direct -human-in-the-loop quality score — rising values mean agents are losing trust in the answers. - -```sql -WITH captain_msgs AS ( - SELECT id - FROM messages - WHERE account_id = :account_id - AND sender_type = 'Captain::Assistant' - AND sender_id = :assistant_id - AND message_type = 1 - AND private = false - AND created_at BETWEEN :start AND :end -) -SELECT - (SELECT COUNT(*) FROM captain_message_reports r - WHERE r.message_id IN (SELECT id FROM captain_msgs))::float - / NULLIF((SELECT COUNT(*) FROM captain_msgs), 0) AS flagged_rate; -``` - -## Conversation Depth (messages per handled conversation) - -Definition: Average number of public replies Captain sends per conversation it handles. Low -(~1) = quick one-shot answers; high = Captain is grinding through long back-and-forths, which -often correlates with the cases it ends up handing off. - -```sql -SELECT - COUNT(*)::float / NULLIF(COUNT(DISTINCT conversation_id), 0) AS msgs_per_conversation -FROM messages -WHERE account_id = :account_id - AND sender_type = 'Captain::Assistant' - AND sender_id = :assistant_id - AND message_type = 1 - AND private = false - AND created_at BETWEEN :start AND :end; -``` - -## Hours Saved - -Definition: A headline "what did Captain save us" number for execs: the volume of replies Captain -sent multiplied by the team's average response time. Reads as agent time deflected. - -Caveat (important): the multiplier here is the team's average *response time* (`first_response` / -`reply_time`), which is how long a customer *waited*, not how long an agent actually *worked* on a -reply. Wait time is typically much larger than handling time, so this number overstates true labor -saved by a wide margin and should be presented as a directional/estimate figure, not measured -labor. Counting every Captain message (including ones in conversations later handed off to a human) -also inflates it. See the conversation that produced this file for the more conservative -"deflected conversations × handling time" alternative if a defensible number is needed. - -```sql -SELECT - ( - SELECT COUNT(*) - FROM messages m - WHERE m.account_id = :account_id - AND m.sender_type = 'Captain::Assistant' - AND m.sender_id = :assistant_id - AND m.message_type = 1 - AND m.private = false - AND m.created_at BETWEEN :start AND :end - ) - * ( - SELECT AVG(re.value) - FROM reporting_events re - WHERE re.account_id = :account_id - AND re.name = 'reply_time' - AND re.created_at BETWEEN :start AND :end - ) / 3600.0 AS hours_saved; -``` - -## Knowledge Base Coverage - -Definition: How much knowledge backs this assistant — approved vs. still-pending FAQ responses, -plus synced documents. Not a performance metric but the leading indicator: thin/low-approval -knowledge usually explains a low deflection rate. (Time-independent; it's the assistant's -current state.) - -```sql -SELECT - COUNT(*) FILTER (WHERE status = 1) AS approved_responses, - COUNT(*) FILTER (WHERE status = 0) AS pending_responses, - (SELECT COUNT(*) FROM captain_documents d - WHERE d.assistant_id = :assistant_id) AS documents -FROM captain_assistant_responses -WHERE assistant_id = :assistant_id; -``` - ---- - -## Notes for building the overview page - -- **Two attribution paths exist** and answer slightly different questions. Message-based - (`sender_id = assistant`) = "conversations Captain actually engaged." Inbox-based - (`captain_inboxes`) = "conversations that flowed through Captain's inbox, engaged or not." - Message-based is used for the handled/rate metrics (tighter); inbox-based only for the reopen - metric since `inference_resolved` is emitted by the auto-resolve job. If an inbox has exactly - one assistant, the two converge. -- **`reporting_events` has no `assistant_id`** — everything is joined back through - `conversation_id` or `inbox_id`. Fine today, but multi-assistant-per-inbox support would want - assistant attribution stamped on the event at write time. -- **`value` on the captain inference events** is "seconds from conversation creation to the - event" (see `create_captain_inference_event`), so time-to-resolve / time-to-handoff are - almost free if you want a couple more timing metrics.