feat: assistant overview page [CW-7408] (#14889)

This PR adds a Captain Assistant **Overview** page to show some KPI
metrics (conversations handled, auto-resolution, handoff, hours saved,
reopen-after-resolve, conversation depth) with trend deltas vs the
previous window, a real knowledge card, and a lazily-loaded, cached LLM
welcome summary.

### Highlights

- **Two contextual banners** on the overview:
- **Inbox banner** — prompts the user to connect an inbox when the
assistant has none, so it can actually do work.
- **Coverage banner** — warns when FAQ coverage is below 85% with more
than 100 responses pending review, linking straight to the pending
queue. Dismissal persists per-assistant for 24h via localStorage.
- **Batched stats builder** (`Captain::AssistantStatsBuilder`) computes
both windows in single FILTER-aggregated scans to cut round trips,
behind new `stats`/`summary` endpoints.
- **Cards included but intentionally left dummy / not rendered yet:**
`ResponseQualityCard` (flagged responses) and `CreditUsageCard` (credit
usage + daily chart). Credits are an account-wide counter with no
per-assistant or daily history, so there is no real data to back them
yet; they ship in the codebase but are not wired into the page.

### Index migration

- Replaces `index_messages_on_sender_type_and_sender_id` with
`index_messages_on_sender_and_created` `(sender_type, sender_id,
created_at)`.
- **Why it helps:** the per-assistant windowed lookups filter `sender_*`
*and* a `created_at` range. The old 2-column index matched every
lifetime row for the assistant and filtered the time slice at the heap
(~89% of rows discarded); adding `created_at` as a range column lets
Postgres scan only the window, and fixes the row-count estimate so the
planner picks a hash join over a nested loop on `reporting_events`.
- **Why dropping the old index is safe:** the new index is a left-prefix
superset `(sender_type, sender_id, ...)`, so every query the old one
served is still served. No code references it by name, and dropping it
keeps write amplification on `messages` neutral. Built/dropped with
`CONCURRENTLY` and `if_not_exists`/`if_exists` guards.


## Preview

<img width="2572" height="1754" alt="CleanShot 2026-06-29 at 22 38
51@2x"
src="https://github.com/user-attachments/assets/3798d09e-7850-48e4-b2cd-508533f15cea"
/>

## Banners

#### Inbox connect alert

<img width="2178" height="612" alt="CleanShot 2026-06-30 at 14 26 55@2x"
src="https://github.com/user-attachments/assets/373c371c-bb7d-4291-a0f9-620673078302"
/>

#### Coverage alert
<img width="2178" height="612" alt="CleanShot 2026-06-30 at 14 25 41@2x"
src="https://github.com/user-attachments/assets/e12d6308-11b6-4ba2-88a2-8a3077dd3e8f"
/>

---------

Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
This commit is contained in:
Shivam Mishra
2026-07-07 17:54:13 +05:30
committed by GitHub
co-authored by Sivin Varghese
parent 1e7218d439
commit a5fcecb3f6
33 changed files with 2090 additions and 10 deletions
+83
View File
@@ -0,0 +1,83 @@
# Generates the LLM welcome summary for the Captain Overview page from the
# assistant's stats hash (see Captain::AssistantStatsBuilder). Renders the
# captain_overview_summary.liquid prompt and returns markdown.
class Captain::OverviewSummaryService < Captain::BaseTaskService
pattr_initialize [:account!, :assistant!, :first_name!, :stats!, :period!]
def perform
api_response = make_api_call(
feature: 'editor',
messages: [
{ role: 'system', content: system_prompt },
{ role: 'user', content: 'Write the summary.' }
]
)
return api_response if api_response[:error]
{ message: api_response[:message] }
end
private
def system_prompt
Liquid::Template.parse(prompt_from_file('captain_overview_summary')).render(prompt_variables)
end
def prompt_variables
stat_variables.merge(period_variables)
end
def stat_variables
{
'first_name' => first_name.to_s,
'assistant_name' => assistant.name.to_s,
'conversations_handled' => current(:conversations_handled),
'hours_saved' => current(:hours_saved),
'auto_resolution_rate' => current(:auto_resolution_rate),
'auto_resolution_trend' => trend(:auto_resolution_rate),
'handoff_rate' => current(:handoff_rate),
'handoff_trend' => trend(:handoff_rate),
'reopen_rate' => current(:reopen_rate),
'reopen_trend' => trend(:reopen_rate),
'knowledge_coverage' => stats.dig(:knowledge, :coverage).to_s,
'knowledge_approved' => stats.dig(:knowledge, :approved).to_s,
'knowledge_documents' => stats.dig(:knowledge, :documents).to_s
}
end
def period_variables
{
'today' => formatted_date(Time.zone.today),
'period_label' => period[:label].to_s,
'period_start' => formatted_date(period[:starts_on]),
'period_end' => formatted_date(period[:ends_on])
}
end
def formatted_date(date)
date.strftime('%B %-d, %Y')
end
def current(key)
stats.dig(key, :current).to_s
end
def trend(key)
stats.dig(key, :trend).to_s
end
def event_name
'captain_overview_summary'
end
def use_account_openai_hook?
true
end
# The overview summary is an internal analytics readout, not a customer-facing
# response, so it should not consume or be blocked by the captain_responses quota.
def counts_toward_usage?
false
end
end
@@ -0,0 +1,38 @@
You are writing a short, warm summary of how an AI support assistant named "{{ assistant_name }}" performed over a reporting period, for {{ first_name }}, the person who manages it.
Voice and format:
- Address {{ first_name }} directly and open with "Hey {{ first_name }},". Be conversational, never robotic.
- Always call the assistant by its name, {{ assistant_name }}. Never call it "Captain", "the assistant", or "your assistant".
- This is a static, read-only poster on an analytics dashboard, not a chat. The reader cannot reply or ask you for anything. Never ask a question, invite a reply, offer further help, or say things like "let me know" or "I can dive in".
- Write 2 to 4 sentences in one short paragraph. Add a second short paragraph only for a genuinely useful heads-up.
- Output plain markdown only: no headings, lists, preamble, or sign-off. Do not use em dashes.
- Never state an exact figure. This summary is cached and the live numbers keep moving, so a precise value would quickly look wrong. Round every number down to a clean approximation and soften it with words like "around", "roughly", "about", "nearly", "just over", or "upwards of". For example, render **1,248** as "upwards of **1,200**", **63.2%** as "around **60%**", and **612** hours as "roughly **600** hours". For a small count, use a loose phrase like "a handful" instead of the exact number.
- Wrap the approximate figure in **double asterisks** so the interface can highlight it. Bold only the figures, never whole phrases or the softening word.
Timing: today is {{ today }}. These stats cover {{ period_label }} ({{ period_start }} to {{ period_end }}). You may lightly reference the month, the season, or how far into the period things stand when it genuinely fits, but never invent events or facts.
The stats for this period:
<stats>
- Conversations handled: {{ conversations_handled }}. Distinct conversations {{ assistant_name }} replied in at least once. Raw volume and adoption, not a measure of quality.
- Hours saved: {{ hours_saved }} hours. A rough, directional estimate of agent time saved. A feel-good figure, not exact measured labor.
- Auto-resolution rate: {{ auto_resolution_rate }}% ({{ auto_resolution_trend }} points vs previous period). Of the conversations it handled, the share {{ assistant_name }} resolved on its own with no human reply. The core performance signal; higher is better.
- Handoff rate: {{ handoff_rate }}% ({{ handoff_trend }} points vs previous period). Of the conversations it handled, the share it escalated to a human agent. The inverse of deflection; lower is better.
- Reopen-after-resolve rate: {{ reopen_rate }}% ({{ reopen_trend }} points vs previous period). Of the conversations it auto-resolved, the share later reopened. A quality signal; lower is better, and a high value means it closed conversations the customer was not actually done with.
- Knowledge base: {{ knowledge_approved }} approved FAQ answers, {{ knowledge_documents }} documents, {{ knowledge_coverage }}% coverage (the share of FAQ answers the team has approved). This is setup the team controls, not something {{ assistant_name }} earned. It is a leading indicator: low coverage tends to cause low auto-resolution.
</stats>
Only the auto-resolution, handoff, and reopen rates reflect how {{ assistant_name }} actually performed, and they are the only things worth crediting it for. Conversations handled and hours saved are context. The knowledge base is an input, never a win to praise.
How to judge the numbers (rough bands, do not quote them in the summary):
- Auto-resolution rate: below 30% is low and early-stage, 30 to 50% is decent, above 50% is genuinely strong.
- Handoff rate: above 60% is high, 30 to 60% is moderate, below 30% is strong.
- Reopen-after-resolve rate: below 5% is healthy, 5 to 15% is worth watching, above 15% is a real problem.
- Knowledge coverage: only worth mentioning when below 85% (below 60% is seriously thin), as a likely cause of weak auto-resolution. At 85% or above it is just the healthy baseline, so do not mention or praise it.
- When the conversation volume is small (roughly under 30), rates are noisy, so describe them tentatively and do not over-interpret a perfect or terrible looking percentage.
Writing the summary:
- Cold start: if conversations handled is 0, there is no performance to report. Skip the auto-resolution, handoff, reopen, and hours-saved figures entirely. Instead note the knowledge base and say {{ assistant_name }} is set up and ready to start handling support (or ready to start once some knowledge is added, if the base is empty). Ignore the rest of these points in this case.
- Be honest and proportionate. Do not call a result impressive, strong, excellent, solid, flawless, or perfect unless it clears the "strong" band above. State a low or middling number plainly or as room to grow, never dressed up. A modest summary is fine and often correct.
- Lead with the genuinely strong results if there are any. If nothing clears the strong band, open plainly with the volume of work handled, without overselling it.
- Mention a trend only when it is meaningful, and judge it against the bands rather than the direction alone (a rate that rose but is still in the low band is not yet a win).
- Surface at most one proactive concern when a stat warrants it (a high handoff rate, a low auto-resolution rate, a rising reopen rate, or thin coverage). Skip it entirely when everything looks healthy. Keep it a calm observation about the data, not an alarm.
@@ -0,0 +1,203 @@
# frozen_string_literal: true
require 'faker'
require 'active_support/testing/time_helpers'
# Seeds Captain assistant activity for the reports/overview test data.
#
# Produces a variety of assistant-handled conversations in a single web inbox so
# every Captain assistant overview metric (handled, auto-resolution, handoff,
# hours saved, reopen rate, conversation depth) has realistic data:
# - :resolved_by_assistant assistant answers and Captain auto-resolves
# - :handled_by_both assistant answers, a human also replies and resolves
# - :handed_off assistant answers, then hands off to a human
# - :resolved_and_reopened Captain resolves, then the conversation reopens
#
# Reporting events are fired through ReportingEventListener directly (mirroring
# ConversationCreator) so the same rows the builder reads from get populated.
class Seeders::Reports::AssistantConversationCreator
include ActiveSupport::Testing::TimeHelpers
OUTCOMES = %i[resolved_by_assistant handled_by_both handed_off resolved_and_reopened].freeze
def initialize(account:, assistant:, inbox:, resources:)
@account = account
@assistant = assistant
@inbox = inbox
@contacts = resources[:contacts]
@agents = inbox.members.to_a.presence || resources[:agents]
end
def create_conversation(created_at:, outcome:)
conversation = nil
travel_to(created_at) do
conversation = build_conversation
conversation.save!
seed_dialogue(conversation, outcome)
end
travel_back
apply_outcome(conversation, created_at, outcome)
conversation
end
private
def build_conversation
contact = @contacts.sample
contact_inbox = @inbox.contact_inboxes.find_or_create_by!(contact: contact, source_id: SecureRandom.hex)
contact_inbox.conversations.create!(
account: @account,
inbox: @inbox,
contact: contact,
priority: [nil, 'high', 'medium', 'low'].sample
)
end
# Builds the message exchange for the conversation while time is frozen at its
# creation moment. Every outcome starts with a customer question and at least
# one public assistant reply so the conversation lands in the assistant's
# handled set; some outcomes add a human reply or a handoff.
def seed_dialogue(conversation, outcome)
customer_message = incoming_message(conversation)
travel(rand((20.seconds)..(5.minutes)))
assistant_reply(conversation, waiting_since: customer_message.created_at)
case outcome
when :handed_off then seed_handoff(conversation)
when :handled_by_both then seed_human_turn(conversation)
else seed_assistant_follow_up(conversation)
end
end
def seed_handoff(conversation)
travel(rand((1.minute)..(10.minutes)))
handoff_to_human(conversation)
travel(rand((1.minute)..(15.minutes)))
human_reply(conversation)
end
def seed_human_turn(conversation)
travel(rand((1.minute)..(15.minutes)))
human_reply(conversation)
end
# Pure assistant threads occasionally take a second turn, giving depth > 1.
def seed_assistant_follow_up(conversation)
return unless rand < 0.6
travel(rand((1.minute)..(10.minutes)))
incoming_message(conversation)
travel(rand((20.seconds)..(5.minutes)))
assistant_reply(conversation, waiting_since: Time.current)
end
def apply_outcome(conversation, created_at, outcome)
resolved_at = created_at + rand((30.minutes)..(8.hours))
case outcome
when :resolved_by_assistant
resolve_by_captain(conversation, resolved_at)
when :handled_by_both
resolve_by_human(conversation, resolved_at)
when :handed_off
resolve_by_human(conversation, resolved_at) if rand < 0.6
when :resolved_and_reopened
resolve_by_captain(conversation, resolved_at)
reopen(conversation, resolved_at + rand((1.hour)..(24.hours)))
end
end
def incoming_message(conversation)
conversation.messages.create!(
account: @account,
inbox: @inbox,
message_type: :incoming,
content: Faker::Lorem.paragraph(sentence_count: rand(1..3)),
sender: conversation.contact
)
end
def assistant_reply(conversation, waiting_since:)
message = conversation.messages.create!(
account: @account,
inbox: @inbox,
message_type: :outgoing,
private: false,
content: Faker::Lorem.paragraph(sentence_count: rand(1..4)),
sender: @assistant
)
trigger_reply_time(message, waiting_since)
message
end
def human_reply(conversation)
agent = @agents.sample
conversation.update_column(:assignee_id, agent.id) if conversation.assignee_id.nil? # rubocop:disable Rails/SkipsModelValidations
conversation.messages.create!(
account: @account,
inbox: @inbox,
message_type: :outgoing,
private: false,
content: Faker::Lorem.paragraph(sentence_count: rand(1..4)),
sender: agent
)
end
def resolve_by_captain(conversation, resolved_at)
mark_resolved(conversation, resolved_at)
travel_to(resolved_at) do
trigger_event('conversation_resolved', conversation)
trigger_event('conversation_captain_inference_resolved', conversation)
end
travel_back
end
def resolve_by_human(conversation, resolved_at)
mark_resolved(conversation, resolved_at)
travel_to(resolved_at) do
trigger_event('conversation_resolved', conversation)
end
travel_back
end
def reopen(conversation, reopened_at)
# rubocop:disable Rails/SkipsModelValidations
conversation.update_column(:status, :open)
conversation.update_column(:updated_at, reopened_at)
# rubocop:enable Rails/SkipsModelValidations
travel_to(reopened_at) do
trigger_event('conversation_opened', conversation)
end
travel_back
end
def handoff_to_human(conversation)
trigger_event('conversation_captain_inference_handoff', conversation)
end
def mark_resolved(conversation, resolved_at)
# rubocop:disable Rails/SkipsModelValidations
conversation.update_column(:status, :resolved)
conversation.update_column(:updated_at, resolved_at)
# rubocop:enable Rails/SkipsModelValidations
end
def trigger_event(name, conversation)
ReportingEventListener.instance.public_send(
name, Events::Base.new(name, Time.current, { conversation: conversation })
)
end
def trigger_reply_time(message, waiting_since)
ReportingEventListener.instance.reply_created(
Events::Base.new('reply_created', Time.current,
{ message: message, conversation: message.conversation, waiting_since: waiting_since })
)
end
end
+100 -2
View File
@@ -17,6 +17,9 @@
# - 5 teams with realistic distribution
# - 30 labels with random assignments
# - 3 inboxes with agent assignments
# - 1 Captain assistant bound to a single web inbox, with knowledge (FAQs + documents)
# and a variety of assistant-handled conversations (auto-resolved, handed off,
# handled with a human, resolved-then-reopened) for the assistant overview page
# - Realistic reporting events with historical timestamps
#
# Note: This seeder clears existing data for the account before seeding.
@@ -24,8 +27,9 @@
require 'faker'
require_relative 'conversation_creator'
require_relative 'message_creator'
require_relative 'assistant_conversation_creator'
# rubocop:disable Rails/Output
# rubocop:disable Rails/Output, Metrics/ClassLength
class Seeders::Reports::ReportDataSeeder
include ActiveSupport::Testing::TimeHelpers
@@ -36,6 +40,11 @@ class Seeders::Reports::ReportDataSeeder
TOTAL_LABELS = 30
TOTAL_INBOXES = 3
MESSAGES_PER_CONVERSATION = 5
# Captain assistant conversations, split across the outcomes the overview page reports on.
TOTAL_ASSISTANT_CONVERSATIONS = 120
ASSISTANT_KNOWLEDGE_APPROVED = 14
ASSISTANT_KNOWLEDGE_PENDING = 6
ASSISTANT_DOCUMENTS = 4
START_DATE = 3.months.ago # rubocop:disable Rails/RelativeDateConstant
END_DATE = Time.current
@@ -48,6 +57,8 @@ class Seeders::Reports::ReportDataSeeder
@labels = []
@inboxes = []
@contacts = []
@assistant = nil
@assistant_inbox = nil
end
def perform!
@@ -61,8 +72,10 @@ class Seeders::Reports::ReportDataSeeder
create_labels
create_inboxes
create_contacts
create_assistant
create_conversations
create_assistant_conversations
puts "Completed reports data seeding for account: #{@account.name}"
end
@@ -71,6 +84,7 @@ class Seeders::Reports::ReportDataSeeder
def clear_existing_data
puts "Clearing existing data for account: #{@account.id}"
clear_assistant_data
@account.teams.destroy_all
@account.conversations.destroy_all
@account.labels.destroy_all
@@ -80,6 +94,16 @@ class Seeders::Reports::ReportDataSeeder
@account.reporting_events.destroy_all
end
# Delete Captain records directly (assistant associations are destroy_async, which
# would leave rows around mid-reseed); order respects foreign keys.
def clear_assistant_data
assistant_ids = Captain::Assistant.for_account(@account.id).select(:id)
Captain::AssistantResponse.by_account(@account.id).delete_all
Captain::Document.for_account(@account.id).delete_all
CaptainInbox.where(captain_assistant_id: assistant_ids).delete_all
Captain::Assistant.for_account(@account.id).delete_all
end
def create_teams
TOTAL_TEAMS.times do |i|
team = @account.teams.create!(
@@ -208,6 +232,80 @@ class Seeders::Reports::ReportDataSeeder
print "\n"
end
# One assistant, bound to a single web inbox (the first one), as the overview page expects.
def create_assistant
@account.enable_features!('captain_integration', 'captain_integration_v2')
@assistant_inbox = @inboxes.first
@assistant = Captain::Assistant.create!(
account: @account,
name: "#{Faker::Company.name} Copilot",
description: 'Captain assistant handling website support conversations.',
config: { feature_faq: true, feature_memory: true, product_name: @account.name }
)
CaptainInbox.create!(captain_assistant: @assistant, inbox: @assistant_inbox)
create_assistant_knowledge
puts "Created assistant '#{@assistant.name}' for inbox '#{@assistant_inbox.name}'"
end
def create_assistant_knowledge
ASSISTANT_KNOWLEDGE_APPROVED.times { create_assistant_response(:approved) }
ASSISTANT_KNOWLEDGE_PENDING.times { create_assistant_response(:pending) }
ASSISTANT_DOCUMENTS.times do
Captain::Document.create!(
account: @account,
assistant: @assistant,
name: Faker::Company.catch_phrase,
external_link: "https://#{Faker::Internet.domain_name}/#{Faker::Internet.slug}",
content: Faker::Lorem.paragraphs(number: rand(2..4)).join("\n\n"),
status: :available,
sync_status: :synced
)
end
end
def create_assistant_response(status)
Captain::AssistantResponse.create!(
account: @account,
assistant: @assistant,
question: "#{Faker::Lorem.sentence(word_count: rand(4..8)).chomp('.')}?",
answer: Faker::Lorem.paragraph(sentence_count: rand(2..4)),
status: status
)
end
def create_assistant_conversations
creator = Seeders::Reports::AssistantConversationCreator.new(
account: @account,
assistant: @assistant,
inbox: @assistant_inbox,
resources: { contacts: @contacts, agents: @agents }
)
outcomes = assistant_outcome_distribution
outcomes.each_with_index do |outcome, i|
created_at = Faker::Time.between(from: 65.days.ago, to: END_DATE)
creator.create_conversation(created_at: created_at, outcome: outcome)
print "\rCreating assistant conversations: #{i + 1}/#{outcomes.size}"
end
print "\n"
end
# Weighted mix of outcomes so every overview metric has meaningful numbers, shuffled
# so they interleave across the time span rather than clustering by type.
def assistant_outcome_distribution
counts = {
resolved_by_assistant: (TOTAL_ASSISTANT_CONVERSATIONS * 0.4).round,
handled_by_both: (TOTAL_ASSISTANT_CONVERSATIONS * 0.25).round,
handed_off: (TOTAL_ASSISTANT_CONVERSATIONS * 0.2).round,
resolved_and_reopened: (TOTAL_ASSISTANT_CONVERSATIONS * 0.15).round
}
counts.flat_map { |outcome, count| [outcome] * count }.shuffle
end
def create_conversations
conversation_creator = Seeders::Reports::ConversationCreator.new(
account: @account,
@@ -231,4 +329,4 @@ class Seeders::Reports::ReportDataSeeder
print "\n"
end
end
# rubocop:enable Rails/Output
# rubocop:enable Rails/Output, Metrics/ClassLength