From a5fcecb3f672851fc383cbb5b1995d0e325cc7b0 Mon Sep 17 00:00:00 2001 From: Shivam Mishra Date: Tue, 7 Jul 2026 17:54:13 +0530 Subject: [PATCH] feat: assistant overview page [CW-7408] (#14889) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR adds a Captain Assistant **Overview** page to show some KPI metrics (conversations handled, auto-resolution, handoff, hours saved, reopen-after-resolve, conversation depth) with trend deltas vs the previous window, a real knowledge card, and a lazily-loaded, cached LLM welcome summary. ### Highlights - **Two contextual banners** on the overview: - **Inbox banner** — prompts the user to connect an inbox when the assistant has none, so it can actually do work. - **Coverage banner** — warns when FAQ coverage is below 85% with more than 100 responses pending review, linking straight to the pending queue. Dismissal persists per-assistant for 24h via localStorage. - **Batched stats builder** (`Captain::AssistantStatsBuilder`) computes both windows in single FILTER-aggregated scans to cut round trips, behind new `stats`/`summary` endpoints. - **Cards included but intentionally left dummy / not rendered yet:** `ResponseQualityCard` (flagged responses) and `CreditUsageCard` (credit usage + daily chart). Credits are an account-wide counter with no per-assistant or daily history, so there is no real data to back them yet; they ship in the codebase but are not wired into the page. ### Index migration - Replaces `index_messages_on_sender_type_and_sender_id` with `index_messages_on_sender_and_created` `(sender_type, sender_id, created_at)`. - **Why it helps:** the per-assistant windowed lookups filter `sender_*` *and* a `created_at` range. The old 2-column index matched every lifetime row for the assistant and filtered the time slice at the heap (~89% of rows discarded); adding `created_at` as a range column lets Postgres scan only the window, and fixes the row-count estimate so the planner picks a hash join over a nested loop on `reporting_events`. - **Why dropping the old index is safe:** the new index is a left-prefix superset `(sender_type, sender_id, ...)`, so every query the old one served is still served. No code references it by name, and dropping it keeps write amplification on `messages` neutral. Built/dropped with `CONCURRENTLY` and `if_not_exists`/`if_exists` guards. ## Preview CleanShot 2026-06-29 at 22 38
51@2x ## Banners #### Inbox connect alert CleanShot 2026-06-30 at 14 26 55@2x #### Coverage alert CleanShot 2026-06-30 at 14 25 41@2x --------- Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com> --- .../dashboard/api/captain/assistant.js | 16 ++ .../components-next/captain/PageLayout.vue | 3 +- .../overview/CoverageBanner.vue | 96 +++++++ .../pageComponents/overview/InboxBanner.vue | 73 +++++ .../pageComponents/overview/KnowledgeCard.vue | 87 ++++++ .../pageComponents/overview/MetricCard.vue | 40 +++ .../pageComponents/overview/QuickLinks.vue | 81 ++++++ .../pageComponents/overview/RangeSelector.vue | 78 +++++ .../pageComponents/overview/WelcomeCard.vue | 75 +++++ .../switcher/AssistantSwitcher.vue | 2 +- .../components-next/sidebar/Sidebar.vue | 8 + .../dashboard/helper/featureHelper.js | 1 + .../i18n/locale/en/integrations.json | 87 ++++++ .../dashboard/i18n/locale/en/settings.json | 1 + .../captain/assistants/overview/Index.vue | 151 ++++++++++ .../dashboard/captain/captain.routes.js | 9 +- .../captain/pages/AssistantsIndexPage.vue | 3 +- app/models/message.rb | 2 +- config/routes.rb | 3 + ...00_add_sender_created_index_to_messages.rb | 21 ++ db/schema.rb | 3 +- .../captain/assistant_drilldown_builder.rb | 162 +++++++++++ .../captain/assistant_stats_builder.rb | 224 +++++++++++++++ .../captain/assistant_stats_window.rb | 78 +++++ .../accounts/captain/assistants_controller.rb | 47 ++- .../app/policies/captain/assistant_policy.rb | 8 + lib/captain/overview_summary_service.rb | 83 ++++++ .../captain_overview_summary.liquid | 38 +++ .../reports/assistant_conversation_creator.rb | 203 +++++++++++++ lib/seeders/reports/report_data_seeder.rb | 102 ++++++- .../captain/assistant_stats_builder_spec.rb | 271 ++++++++++++++++++ .../captain/assistants_controller_spec.rb | 42 +++ .../policies/captain/assistant_policy_spec.rb | 2 +- 33 files changed, 2090 insertions(+), 10 deletions(-) create mode 100644 app/javascript/dashboard/components-next/captain/pageComponents/overview/CoverageBanner.vue create mode 100644 app/javascript/dashboard/components-next/captain/pageComponents/overview/InboxBanner.vue create mode 100644 app/javascript/dashboard/components-next/captain/pageComponents/overview/KnowledgeCard.vue create mode 100644 app/javascript/dashboard/components-next/captain/pageComponents/overview/MetricCard.vue create mode 100644 app/javascript/dashboard/components-next/captain/pageComponents/overview/QuickLinks.vue create mode 100644 app/javascript/dashboard/components-next/captain/pageComponents/overview/RangeSelector.vue create mode 100644 app/javascript/dashboard/components-next/captain/pageComponents/overview/WelcomeCard.vue create mode 100644 app/javascript/dashboard/routes/dashboard/captain/assistants/overview/Index.vue create mode 100644 db/migrate/20260630000000_add_sender_created_index_to_messages.rb create mode 100644 enterprise/app/builders/captain/assistant_drilldown_builder.rb create mode 100644 enterprise/app/builders/captain/assistant_stats_builder.rb create mode 100644 enterprise/app/builders/captain/assistant_stats_window.rb create mode 100644 lib/captain/overview_summary_service.rb create mode 100644 lib/integrations/openai/openai_prompts/captain_overview_summary.liquid create mode 100644 lib/seeders/reports/assistant_conversation_creator.rb create mode 100644 spec/enterprise/builders/captain/assistant_stats_builder_spec.rb diff --git a/app/javascript/dashboard/api/captain/assistant.js b/app/javascript/dashboard/api/captain/assistant.js index 157eba74e..dcd92f735 100644 --- a/app/javascript/dashboard/api/captain/assistant.js +++ b/app/javascript/dashboard/api/captain/assistant.js @@ -1,6 +1,10 @@ /* global axios */ import ApiClient from '../ApiClient'; +// Viewer's UTC offset in hours, matching the reports API convention so the +// backend can anchor calendar ranges to the viewer's day. +const getTimezoneOffset = () => -new Date().getTimezoneOffset() / 60; + class CaptainAssistant extends ApiClient { constructor() { super('captain/assistants', { accountScoped: true }); @@ -21,6 +25,18 @@ class CaptainAssistant extends ApiClient { message_history: messageHistory, }); } + + getStats({ assistantId, range }) { + return axios.get(`${this.url}/${assistantId}/stats`, { + params: { range, timezone_offset: getTimezoneOffset() }, + }); + } + + getSummary({ assistantId, range }) { + return axios.get(`${this.url}/${assistantId}/summary`, { + params: { range, timezone_offset: getTimezoneOffset() }, + }); + } } export default new CaptainAssistant(); diff --git a/app/javascript/dashboard/components-next/captain/PageLayout.vue b/app/javascript/dashboard/components-next/captain/PageLayout.vue index 817608f62..6c0d37d15 100644 --- a/app/javascript/dashboard/components-next/captain/PageLayout.vue +++ b/app/javascript/dashboard/components-next/captain/PageLayout.vue @@ -182,7 +182,8 @@ const handleCreateAssistant = () => { -
+
+
+import { computed, ref, watch } from 'vue'; +import { useRoute, useRouter } from 'vue-router'; +import { LocalStorage } from 'shared/helpers/localStorage'; + +const props = defineProps({ + knowledge: { + type: Object, + default: () => ({ approved: 0, pending: 0, documents: 0, coverage: 0 }), + }, +}); + +const route = useRoute(); +const router = useRouter(); + +// Dismissal is remembered per assistant for 24 hours (setFlag's default expiry). +const DISMISS_STORE = 'captain_overview_coverage_banner'; + +const accountId = computed(() => route.params.accountId); +const assistantId = computed(() => route.params.assistantId); + +// Re-read the stored flag whenever the assistant changes, otherwise the banner +// would keep the first assistant's dismissed state after switching. +const dismissed = ref(false); + +watch( + [accountId, assistantId], + ([account, assistant]) => { + dismissed.value = LocalStorage.getFlag(DISMISS_STORE, account, assistant); + }, + { immediate: true } +); + +// Thin coverage paired with a large review backlog: approving the pending FAQs +// is the quickest lever to lift auto-resolution, so nudge the team to act. +const COVERAGE_THRESHOLD = 85; +const PENDING_THRESHOLD = 100; + +const showBanner = computed( + () => + !dismissed.value && + (props.knowledge?.coverage ?? 0) < COVERAGE_THRESHOLD && + (props.knowledge?.pending ?? 0) > PENDING_THRESHOLD +); + +const dismiss = () => { + LocalStorage.setFlag(DISMISS_STORE, accountId.value, assistantId.value); + dismissed.value = true; +}; + +const goToPending = () => { + router.push({ + name: 'captain_assistants_responses_pending', + params: { + accountId: route.params.accountId, + assistantId: route.params.assistantId, + }, + }); +}; + + + diff --git a/app/javascript/dashboard/components-next/captain/pageComponents/overview/InboxBanner.vue b/app/javascript/dashboard/components-next/captain/pageComponents/overview/InboxBanner.vue new file mode 100644 index 000000000..45952fa11 --- /dev/null +++ b/app/javascript/dashboard/components-next/captain/pageComponents/overview/InboxBanner.vue @@ -0,0 +1,73 @@ + + + diff --git a/app/javascript/dashboard/components-next/captain/pageComponents/overview/KnowledgeCard.vue b/app/javascript/dashboard/components-next/captain/pageComponents/overview/KnowledgeCard.vue new file mode 100644 index 000000000..80c7ae69d --- /dev/null +++ b/app/javascript/dashboard/components-next/captain/pageComponents/overview/KnowledgeCard.vue @@ -0,0 +1,87 @@ + + + diff --git a/app/javascript/dashboard/components-next/captain/pageComponents/overview/MetricCard.vue b/app/javascript/dashboard/components-next/captain/pageComponents/overview/MetricCard.vue new file mode 100644 index 000000000..9f8a76f43 --- /dev/null +++ b/app/javascript/dashboard/components-next/captain/pageComponents/overview/MetricCard.vue @@ -0,0 +1,40 @@ + + + diff --git a/app/javascript/dashboard/components-next/captain/pageComponents/overview/QuickLinks.vue b/app/javascript/dashboard/components-next/captain/pageComponents/overview/QuickLinks.vue new file mode 100644 index 000000000..e9fb45a07 --- /dev/null +++ b/app/javascript/dashboard/components-next/captain/pageComponents/overview/QuickLinks.vue @@ -0,0 +1,81 @@ + + + diff --git a/app/javascript/dashboard/components-next/captain/pageComponents/overview/RangeSelector.vue b/app/javascript/dashboard/components-next/captain/pageComponents/overview/RangeSelector.vue new file mode 100644 index 000000000..d003480dd --- /dev/null +++ b/app/javascript/dashboard/components-next/captain/pageComponents/overview/RangeSelector.vue @@ -0,0 +1,78 @@ + + + diff --git a/app/javascript/dashboard/components-next/captain/pageComponents/overview/WelcomeCard.vue b/app/javascript/dashboard/components-next/captain/pageComponents/overview/WelcomeCard.vue new file mode 100644 index 000000000..df336a7e4 --- /dev/null +++ b/app/javascript/dashboard/components-next/captain/pageComponents/overview/WelcomeCard.vue @@ -0,0 +1,75 @@ + + + + diff --git a/app/javascript/dashboard/components-next/captain/pageComponents/switcher/AssistantSwitcher.vue b/app/javascript/dashboard/components-next/captain/pageComponents/switcher/AssistantSwitcher.vue index a67d685f5..c78842241 100644 --- a/app/javascript/dashboard/components-next/captain/pageComponents/switcher/AssistantSwitcher.vue +++ b/app/javascript/dashboard/components-next/captain/pageComponents/switcher/AssistantSwitcher.vue @@ -65,7 +65,7 @@ const handleAssistantChange = async assistant => { const currentRouteName = route.name; const targetRouteName = - currentRouteName || 'captain_assistants_responses_index'; + currentRouteName || 'captain_assistants_overview_index'; await fetchDataForRoute(targetRouteName, assistant.id); diff --git a/app/javascript/dashboard/components-next/sidebar/Sidebar.vue b/app/javascript/dashboard/components-next/sidebar/Sidebar.vue index 15a007a74..0e6b3468b 100644 --- a/app/javascript/dashboard/components-next/sidebar/Sidebar.vue +++ b/app/javascript/dashboard/components-next/sidebar/Sidebar.vue @@ -440,6 +440,14 @@ const menuItems = computed(() => { label: t('SIDEBAR.CAPTAIN'), activeOn: ['captain_assistants_create_index'], children: [ + { + name: 'Overview', + label: t('SIDEBAR.CAPTAIN_OVERVIEW'), + activeOn: ['captain_assistants_overview_index'], + to: accountScopedRoute('captain_assistants_index', { + navigationPath: 'captain_assistants_overview_index', + }), + }, { name: 'FAQs', label: t('SIDEBAR.CAPTAIN_RESPONSES'), diff --git a/app/javascript/dashboard/helper/featureHelper.js b/app/javascript/dashboard/helper/featureHelper.js index c90ec15db..a8e5fd2b7 100644 --- a/app/javascript/dashboard/helper/featureHelper.js +++ b/app/javascript/dashboard/helper/featureHelper.js @@ -19,6 +19,7 @@ const FEATURE_HELP_URLS = { webhook: 'https://chwt.app/hc/webhooks', billing: 'https://chwt.app/pricing', saml: 'https://chwt.app/hc/saml', + captain: 'https://chwt.app/captain-docs', captain_billing: 'https://chwt.app/hc/captain_billing', }; diff --git a/app/javascript/dashboard/i18n/locale/en/integrations.json b/app/javascript/dashboard/i18n/locale/en/integrations.json index 3c13f0d51..a39165268 100644 --- a/app/javascript/dashboard/i18n/locale/en/integrations.json +++ b/app/javascript/dashboard/i18n/locale/en/integrations.json @@ -392,6 +392,93 @@ "CAPTAIN": { "NAME": "Captain", "HEADER_KNOW_MORE": "Know more", + "OVERVIEW": { + "HEADER": "Overview", + "WELCOME": { + "LABEL": "Captain summary", + "LOADING": "Generating summary…" + }, + "INBOX_BANNER": { + "TEXT": "This assistant isn't connected to any inbox yet, so it won't respond to conversations.", + "ACTION": "Connect inbox", + "DISMISS": "Dismiss" + }, + "COVERAGE_BANNER": { + "TEXT": "{count} FAQs are pending review, keeping coverage at {coverage}%. Approve them so your assistant can resolve more on its own.", + "ACTION": "Review FAQs", + "DISMISS": "Dismiss" + }, + "RANGES": { + "LAST_DAYS": "Last {count} days", + "THIS_MONTH": "This month", + "LAST_MONTH": "Last month" + }, + "METRICS": { + "HANDLED": { + "LABEL": "Conversations handled", + "HINT": "Distinct conversations this assistant replied in." + }, + "AUTO_RESOLUTION": { + "LABEL": "Auto-resolution rate", + "HINT": "Share of handled conversations closed without a human reply." + }, + "HANDOFF": { + "LABEL": "Handoff rate", + "HINT": "Share of handled conversations escalated to a human agent." + }, + "HOURS_SAVED": { + "LABEL": "Hours saved", + "HINT": "Estimate: Captain replies times the team's average response time. Directional, not measured labor." + }, + "REOPEN": { + "LABEL": "Reopen rate", + "HINT": "Auto-resolved conversations that were reopened afterwards." + }, + "DEPTH": { + "LABEL": "Messages / conversation", + "HINT": "Average replies the assistant sends per conversation." + } + }, + "KNOWLEDGE": { + "TITLE": "Knowledge coverage", + "COVERAGE": "{pct}% approved", + "APPROVED": "Approved FAQs", + "PENDING": "Pending FAQs", + "DOCUMENTS": "Documents" + }, + "FLAGGED": { + "TITLE": "Response quality", + "TOTAL": "{count} flagged · {rate}" + }, + "CREDITS": { + "TITLE": "Credit usage", + "UNIT": "credits", + "LEGEND": "Daily credits used", + "AXIS_START": "{count}d ago", + "AXIS_END": "Today" + }, + "LINKS": { + "DOCS": { + "TITLE": "Captain docs", + "DESCRIPTION": "Guides and how-tos for Captain" + }, + "PLAYGROUND": { + "TITLE": "Playground", + "DESCRIPTION": "Test this assistant's replies" + }, + "BILLING": { + "TITLE": "Billing", + "DESCRIPTION": "Manage credits and plan" + } + }, + "FLAG_REASONS": { + "INCORRECT": "Incorrect info", + "INCOMPLETE": "Incomplete", + "OUTDATED": "Outdated", + "INAPPROPRIATE": "Inappropriate", + "OTHER": "Other" + } + }, "ASSISTANT_SWITCHER": { "ASSISTANTS": "Assistants", "SWITCH_ASSISTANT": "Switch between assistants", diff --git a/app/javascript/dashboard/i18n/locale/en/settings.json b/app/javascript/dashboard/i18n/locale/en/settings.json index 4caca05fb..23058d739 100644 --- a/app/javascript/dashboard/i18n/locale/en/settings.json +++ b/app/javascript/dashboard/i18n/locale/en/settings.json @@ -324,6 +324,7 @@ "ALL_COMPANIES": "All Companies", "CAPTAIN": "Captain", "CAPTAIN_ASSISTANTS": "Assistants", + "CAPTAIN_OVERVIEW": "Overview", "CAPTAIN_DOCUMENTS": "Documents", "CAPTAIN_RESPONSES": "FAQs", "CAPTAIN_TOOLS": "Tools", diff --git a/app/javascript/dashboard/routes/dashboard/captain/assistants/overview/Index.vue b/app/javascript/dashboard/routes/dashboard/captain/assistants/overview/Index.vue new file mode 100644 index 000000000..687e97286 --- /dev/null +++ b/app/javascript/dashboard/routes/dashboard/captain/assistants/overview/Index.vue @@ -0,0 +1,151 @@ + + + diff --git a/app/javascript/dashboard/routes/dashboard/captain/captain.routes.js b/app/javascript/dashboard/routes/dashboard/captain/captain.routes.js index 1ab4fa501..8448f32cf 100644 --- a/app/javascript/dashboard/routes/dashboard/captain/captain.routes.js +++ b/app/javascript/dashboard/routes/dashboard/captain/captain.routes.js @@ -6,6 +6,7 @@ import CaptainPageRouteView from './pages/CaptainPageRouteView.vue'; import AssistantsIndexPage from './pages/AssistantsIndexPage.vue'; import AssistantEmptyStateIndex from './assistants/Index.vue'; +import AssistantOverviewIndex from './assistants/overview/Index.vue'; import AssistantSettingsIndex from './assistants/settings/Settings.vue'; import AssistantInboxesIndex from './assistants/inboxes/Index.vue'; import AssistantPlaygroundIndex from './assistants/playground/Index.vue'; @@ -36,6 +37,12 @@ const metaV2 = { }; const assistantRoutes = [ + { + path: frontendURL('accounts/:accountId/captain/:assistantId/overview'), + component: AssistantOverviewIndex, + name: 'captain_assistants_overview_index', + meta, + }, { path: frontendURL('accounts/:accountId/captain/:assistantId/faqs'), component: ResponsesIndex, @@ -129,7 +136,7 @@ export const routes = [ return { name: 'captain_assistants_index', params: { - navigationPath: 'captain_assistants_responses_index', + navigationPath: 'captain_assistants_overview_index', ...to.params, }, }; diff --git a/app/javascript/dashboard/routes/dashboard/captain/pages/AssistantsIndexPage.vue b/app/javascript/dashboard/routes/dashboard/captain/pages/AssistantsIndexPage.vue index 01ec64618..d366d4254 100644 --- a/app/javascript/dashboard/routes/dashboard/captain/pages/AssistantsIndexPage.vue +++ b/app/javascript/dashboard/routes/dashboard/captain/pages/AssistantsIndexPage.vue @@ -53,6 +53,7 @@ const routeToLastActiveAssistant = () => { const { navigationPath } = route.params; const isAValidRoute = [ + 'captain_assistants_overview_index', // Overview page 'captain_assistants_responses_index', // Faq page 'captain_assistants_documents_index', // Document page 'captain_assistants_scenarios_index', // Scenario page @@ -64,7 +65,7 @@ const routeToLastActiveAssistant = () => { const navigateTo = isAValidRoute ? navigationPath - : 'captain_assistants_responses_index'; + : 'captain_assistants_overview_index'; return routeToView(navigateTo, { accountId: route.params.accountId, diff --git a/app/models/message.rb b/app/models/message.rb index f25d2e112..220bdd549 100644 --- a/app/models/message.rb +++ b/app/models/message.rb @@ -34,7 +34,7 @@ # index_messages_on_conversation_id (conversation_id) # index_messages_on_created_at (created_at) # index_messages_on_inbox_id (inbox_id) -# index_messages_on_sender_type_and_sender_id (sender_type,sender_id) +# index_messages_on_sender_and_created (sender_type,sender_id,created_at) # index_messages_on_source_id (source_id) # diff --git a/config/routes.rb b/config/routes.rb index e0a1a8e50..c5e7ded2f 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -66,6 +66,9 @@ Rails.application.routes.draw do resources :assistants do member do post :playground + get :stats + get :summary + get :drilldown end collection do get :tools diff --git a/db/migrate/20260630000000_add_sender_created_index_to_messages.rb b/db/migrate/20260630000000_add_sender_created_index_to_messages.rb new file mode 100644 index 000000000..3424ead71 --- /dev/null +++ b/db/migrate/20260630000000_add_sender_created_index_to_messages.rb @@ -0,0 +1,21 @@ +class AddSenderCreatedIndexToMessages < ActiveRecord::Migration[7.1] + disable_ddl_transaction! + + # Adds created_at to the (sender_type, sender_id) index so per-assistant + # windowed lookups (Captain Overview stats) can range-scan the time slice + # instead of reading every lifetime row and filtering at the heap. The new + # index is a left-prefix superset of the old one. + # + # TODO: drop the now-redundant index_messages_on_sender_type_and_sender_id + # once this index has been running in production long enough to confirm it + # fully replaces the old one. + def up + add_index :messages, [:sender_type, :sender_id, :created_at], + name: 'index_messages_on_sender_and_created', algorithm: :concurrently, if_not_exists: true + end + + def down + remove_index :messages, name: 'index_messages_on_sender_and_created', + algorithm: :concurrently, if_exists: true + end +end diff --git a/db/schema.rb b/db/schema.rb index 05afea9a1..1a676dd59 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[7.1].define(version: 2026_06_20_000000) do +ActiveRecord::Schema[7.1].define(version: 2026_06_30_000000) do # These extensions should be enabled to support this database enable_extension "pg_stat_statements" enable_extension "pg_trgm" @@ -1034,6 +1034,7 @@ ActiveRecord::Schema[7.1].define(version: 2026_06_20_000000) do t.index ["conversation_id"], name: "index_messages_on_conversation_id" t.index ["created_at"], name: "index_messages_on_created_at" t.index ["inbox_id"], name: "index_messages_on_inbox_id" + t.index ["sender_type", "sender_id", "created_at"], name: "index_messages_on_sender_and_created" t.index ["sender_type", "sender_id"], name: "index_messages_on_sender_type_and_sender_id" t.index ["source_id"], name: "index_messages_on_source_id" end diff --git a/enterprise/app/builders/captain/assistant_drilldown_builder.rb b/enterprise/app/builders/captain/assistant_drilldown_builder.rb new file mode 100644 index 000000000..b98aa7620 --- /dev/null +++ b/enterprise/app/builders/captain/assistant_drilldown_builder.rb @@ -0,0 +1,162 @@ +# Lists the underlying records behind a single Captain assistant stat card, so a +# viewer can drill from an aggregate (e.g. "auto-resolution 42%") into the exact +# conversations or messages that produced it. +# +# The window is resolved by Captain::AssistantStatsWindow from the same `range` +# and `timezone_offset` the stat card used, so the drilldown covers precisely the +# rows the card counted. Records are serialized with the shared reports drilldown +# serializer, so the existing frontend drilldown drawer/card can render them. +class Captain::AssistantDrilldownBuilder + ASSISTANT_SENDER_TYPE = 'Captain::Assistant'.freeze + RESOLVED_EVENT_NAMES = Captain::AssistantStatsBuilder::RESOLVED_EVENT_NAMES + HANDOFF_EVENT_NAMES = Captain::AssistantStatsBuilder::HANDOFF_EVENT_NAMES + + # Metrics whose records are individual messages rather than conversations. + MESSAGE_METRICS = %w[hours_saved].freeze + SUPPORTED_METRICS = %w[ + conversations_handled auto_resolution_rate handoff_rate hours_saved reopen_rate conversation_depth + ].freeze + + DEFAULT_PAGE = 1 + DEFAULT_PER_PAGE = 25 + MAX_PER_PAGE = 100 + + pattr_initialize :assistant, :params + + def self.supported_metric?(metric) = SUPPORTED_METRICS.include?(metric.to_s) + + def build + records = paginated_records.to_a + { meta: meta, payload: records.map { |record| record_serializer(records).serialize(record) } } + end + + private + + def account = assistant.account + + def window + @window ||= Captain::AssistantStatsWindow.new(params[:range], params[:timezone_offset]) + end + + def range = window.current + + def meta + { + metric: metric, + record_type: record_type, + current_page: current_page, + per_page: per_page, + total_count: paginated_records.total_count, + conversation_count: conversation_count, + range: { since: range.first.to_i, until: range.last.to_i } + } + end + + def conversation_count + return paginated_records.total_count unless message_metric? + + drilldown_scope.except(:includes).reorder(nil).distinct.count(:conversation_id) + end + + def paginated_records + @paginated_records ||= drilldown_scope.page(current_page).per(per_page) + end + + def drilldown_scope + case metric + when 'conversations_handled' then handled_conversations + when 'auto_resolution_rate' then conversations_for(resolved_events.select(:conversation_id)) + when 'handoff_rate' then event_conversations(HANDOFF_EVENT_NAMES) + when 'hours_saved' then public_reply_messages + when 'reopen_rate' then reopened_conversations + when 'conversation_depth' then depth_conversations + else + raise ArgumentError, "Unsupported assistant drilldown metric: #{metric}" + end + end + + # Messages the assistant authored in the window; the cohort every metric derives from. + def handled_messages + account.messages.where(sender_type: ASSISTANT_SENDER_TYPE, sender_id: assistant.id, created_at: range) + end + + def handled_conversation_ids + handled_messages.select(:conversation_id) + end + + def handled_conversations + conversations_for(handled_conversation_ids) + end + + # Public agent-facing replies the assistant sent; the rows behind hours_saved. + def public_reply_messages + handled_messages.where(message_type: :outgoing, private: false) + .includes(:sender, conversation: [:assignee, :contact, :inbox]) + .reorder(created_at: :desc) + end + + # Conversations in the handled cohort that recorded one of the given reporting + # events in the window (resolved or handed-off). + def event_conversations(event_names) + ids = account.reporting_events + .where(name: event_names, created_at: range, conversation_id: handled_conversation_ids) + .select(:conversation_id) + conversations_for(ids) + end + + # Captain resolves in the window, excluding bot-resolved rows whose conversation + # was also handed off, mirroring AssistantStatsBuilder#resolved_clause so the + # drilldown lists exactly the conversations the auto-resolution card counted. + def resolved_events + handoff_ids = account.reporting_events.where(name: HANDOFF_EVENT_NAMES, created_at: range).select(:conversation_id) + account.reporting_events + .where(name: RESOLVED_EVENT_NAMES, created_at: range, conversation_id: handled_conversation_ids) + .where("NOT (name = ? AND conversation_id IN (#{handoff_ids.to_sql}))", + Captain::AssistantStatsBuilder::BOT_RESOLVED_EVENT_NAME) + end + + # Auto-resolved conversations that reopened at/after their Captain resolve, + # mirroring AssistantStatsBuilder#reopen_rate's numerator cohort. + def reopened_conversations + ids = account.reporting_events + .where(name: 'conversation_opened') + .where('reporting_events.value > 0') + .where('reporting_events.event_end_time <= ?', range.last) + .joins("INNER JOIN (#{resolved_events.to_sql}) resolves " \ + 'ON resolves.conversation_id = reporting_events.conversation_id ' \ + 'AND reporting_events.event_end_time >= resolves.event_end_time') + .select('reporting_events.conversation_id') + conversations_for(ids) + end + + # Conversations the assistant sent at least one public reply in; the denominator behind conversation_depth. + def depth_conversations + conversations_for(handled_messages.where(message_type: :outgoing, private: false).select(:conversation_id)) + end + + def conversations_for(conversation_ids) + account.conversations + .where(id: conversation_ids) + .includes(:assignee, :contact, :inbox) + .order(created_at: :desc) + end + + def record_serializer(records) + @record_serializer ||= V2::Reports::DrilldownRecordSerializer.new(account, metric, false, records) + end + + def metric = params[:metric].to_s + + def message_metric? = MESSAGE_METRICS.include?(metric) + + def record_type = message_metric? ? 'message' : 'conversation' + + def current_page = [params[:page].to_i, DEFAULT_PAGE].max + + def per_page + requested_per_page = params[:per_page].to_i + requested_per_page = DEFAULT_PER_PAGE if requested_per_page <= 0 + + [requested_per_page, MAX_PER_PAGE].min + end +end diff --git a/enterprise/app/builders/captain/assistant_stats_builder.rb b/enterprise/app/builders/captain/assistant_stats_builder.rb new file mode 100644 index 000000000..7198c39e1 --- /dev/null +++ b/enterprise/app/builders/captain/assistant_stats_builder.rb @@ -0,0 +1,224 @@ +# Computes per-assistant overview metrics for the Captain Overview page. +# Each metric is returned for the current window and the previous equal-length +# window, plus a derived trend. +# +# Queries are batched to cut round trips: the message-derived counts (handled, +# public replies, depth) and the average reply time are each computed for both +# windows in a single scan via conditional FILTER aggregation. +class Captain::AssistantStatsBuilder + RESOLVED_EVENT_NAMES = %w[conversation_captain_inference_resolved conversation_bot_resolved].freeze + HANDOFF_EVENT_NAMES = %w[conversation_captain_inference_handoff conversation_bot_handoff].freeze + BOT_RESOLVED_EVENT_NAME = 'conversation_bot_resolved'.freeze + + attr_reader :assistant, :account + + delegate :range, :period, to: :window + + # `range` is either a day count ('7', '30', '90') or a named period + # ('this_month', 'last_month'). `timezone_offset` is the viewer's UTC offset in + # hours (as the reports API sends it), so month/day boundaries anchor to the + # viewer's day rather than UTC. Both windows are resolved by AssistantStatsWindow. + def initialize(assistant, range = Captain::AssistantStatsWindow::DEFAULT_RANGE, timezone_offset = nil) + @assistant = assistant + @account = assistant.account + @window = Captain::AssistantStatsWindow.new(range, timezone_offset) + end + + def metrics + messages = message_window_metrics + reply_times = avg_reply_times + current = window_metrics(current_range, messages[:current], reply_times[:current]) + previous = window_metrics(previous_range, messages[:previous], reply_times[:previous]) + + build_metrics(current, previous) + end + + private + + attr_reader :window + + def current_range + window.current + end + + def previous_range + window.previous + end + + def build_metrics(current, previous) + { + conversations_handled: pack(current[:handled], previous[:handled], :percent), + auto_resolution_rate: pack(current[:auto_resolution], previous[:auto_resolution], :point), + handoff_rate: pack(current[:handoff], previous[:handoff], :point), + hours_saved: pack(current[:hours_saved], previous[:hours_saved], :percent), + reopen_rate: pack(current[:reopen], previous[:reopen], :point), + conversation_depth: pack(current[:depth], previous[:depth], :absolute), + knowledge: knowledge + } + end + + # Combines the per-window message counts and reply time with the reporting-event metrics for one window. + def window_metrics(range, message_counts, avg_reply) + handled = message_counts[:handled] + public_count = message_counts[:public_count] + depth_conversations = message_counts[:depth_conversations] + resolution = resolution_counts(range) + + { + handled: handled, + auto_resolution: rate(resolution[:resolved], handled), + handoff: rate(resolution[:handoff], handled), + hours_saved: (public_count * avg_reply / 3600.0).round, + reopen: reopen_rate(range), + depth: depth_conversations.zero? ? 0 : (public_count.to_f / depth_conversations).round(1) + } + end + + # One scan over the assistant's messages computes handled, public-reply count, + # and depth-conversation count for both windows via conditional aggregation. + def message_window_metrics + public_clause = "message_type = #{Message.message_types[:outgoing]} AND private = false" + cur = window_clause(current_range) + prev = window_clause(previous_range) + + row = handled_scope(full_span).reorder(nil).pick( + Arel.sql("COUNT(DISTINCT conversation_id) FILTER (WHERE #{cur})"), + Arel.sql("COUNT(DISTINCT conversation_id) FILTER (WHERE #{prev})"), + Arel.sql("COUNT(*) FILTER (WHERE #{cur} AND #{public_clause})"), + Arel.sql("COUNT(*) FILTER (WHERE #{prev} AND #{public_clause})"), + Arel.sql("COUNT(DISTINCT conversation_id) FILTER (WHERE #{cur} AND #{public_clause})"), + Arel.sql("COUNT(DISTINCT conversation_id) FILTER (WHERE #{prev} AND #{public_clause})") + ) + + { + current: { handled: row[0], public_count: row[2], depth_conversations: row[4] }, + previous: { handled: row[1], public_count: row[3], depth_conversations: row[5] } + } + end + + # Average reply time (seconds) for both windows in one scan. + def avg_reply_times + row = account.reporting_events.where(name: 'reply_time', created_at: full_span).reorder(nil).pick( + Arel.sql("AVG(value) FILTER (WHERE #{window_clause(current_range)})"), + Arel.sql("AVG(value) FILTER (WHERE #{window_clause(previous_range)})") + ) + { current: row[0].to_f, previous: row[1].to_f } + end + + # Resolved and handed-off conversation counts for one window, in a single scan + # of the handled set's reporting events. + def resolution_counts(range) + row = account.reporting_events + .where(name: RESOLVED_EVENT_NAMES + HANDOFF_EVENT_NAMES, + created_at: range, + conversation_id: handled_scope(range).select(:conversation_id)) + .reorder(nil) + .pick( + Arel.sql("COUNT(DISTINCT conversation_id) FILTER (WHERE #{resolved_clause(range)})"), + Arel.sql("COUNT(DISTINCT conversation_id) FILTER (WHERE name IN (#{quoted(HANDOFF_EVENT_NAMES)}))") + ) + { resolved: row[0], handoff: row[1] } + end + + # A countable resolve is any inference resolve, or a bot resolve on a conversation + # with no handoff in the window. conversation_bot_resolved fires on any resolve + # without an agent message (reporting_event_listener), so a handed-off conversation + # that goes quiet and gets closed would otherwise count as an auto-resolution too; + # the reports bot_resolutions metric applies the same exclusion (:exclude_bot_handoffs). + def resolved_clause(range) + "name IN (#{quoted(RESOLVED_EVENT_NAMES)}) AND #{bot_resolve_handoff_exclusion(range)}" + end + + def bot_resolve_handoff_exclusion(range) + "NOT (name = #{quote(BOT_RESOLVED_EVENT_NAME)} AND conversation_id IN (#{handoff_conversation_ids(range).to_sql}))" + end + + def handoff_conversation_ids(range) + account.reporting_events.where(name: HANDOFF_EVENT_NAMES, created_at: range).select(:conversation_id) + end + + # Conversations the assistant participated in (authored any message). + def handled_scope(range) + account.messages.where(sender_type: 'Captain::Assistant', sender_id: assistant.id, created_at: range) + end + + # Span covering both windows so a single scan can split them with FILTER. + def full_span + [current_range.first, previous_range.first].min..current_range.last + end + + def window_clause(range) + "created_at >= #{quote(range.first)} AND created_at <= #{quote(range.last)}" + end + + def quote(value) + account.class.connection.quote(value) + end + + def quoted(values) + values.map { |value| quote(value) }.join(', ') + end + + # Of the conversations Captain auto-resolved, the share reopened afterwards. The cohort is + # derived from the assistant's handled conversations (not current inbox membership) so a later + # inbox reassignment doesn't drop historical resolves, and covers both the evaluated (inference) + # and time-based (bot) resolve paths so the denominator matches auto_resolution_rate. + def reopen_rate(range) + resolved_scope = account.reporting_events + .where(name: RESOLVED_EVENT_NAMES, created_at: range, + conversation_id: handled_scope(range).select(:conversation_id)) + .where(bot_resolve_handoff_exclusion(range)) + # event_end_time on a reopen is when it actually reopened. Join it to the conversation's own + # Captain resolves and keep only reopens at/after one of them, so a human resolve/reopen earlier + # in the same window isn't mistaken for a reopen-after-Captain-resolve. (Comparing the reopen's + # start time instead would misfire: the inference event is dispatched just after the generic + # conversation_resolved that seeds event_start_time, so it can land after the reopen's start.) + # The reopen itself must also fall inside the window, so a completed range (last_month, the + # previous window) doesn't count reopens that happened after it ended. + reopened = account.reporting_events + .where(name: 'conversation_opened') + .where('reporting_events.value > 0') + .where('reporting_events.event_end_time <= ?', range.last) + .joins("INNER JOIN (#{resolved_scope.to_sql}) resolves " \ + 'ON resolves.conversation_id = reporting_events.conversation_id ' \ + 'AND reporting_events.event_end_time >= resolves.event_end_time') + .distinct.count('reporting_events.conversation_id') + rate(reopened, resolved_scope.distinct.count(:conversation_id)) + end + + # Approved/pending FAQ counts and the document total in a single round trip. + def knowledge + approved, pending, documents = Captain::AssistantResponse.by_assistant(assistant.id).reorder(nil).pick( + Arel.sql("COUNT(*) FILTER (WHERE status = #{Captain::AssistantResponse.statuses['approved']})"), + Arel.sql("COUNT(*) FILTER (WHERE status = #{Captain::AssistantResponse.statuses['pending']})"), + Arel.sql("(SELECT COUNT(*) FROM captain_documents WHERE assistant_id = #{assistant.id.to_i})") + ) + total = approved + pending + + { + approved: approved, + pending: pending, + documents: documents, + coverage: total.zero? ? 0 : (approved.to_f / total * 100).round + } + end + + def rate(numerator, denominator) + return 0 if denominator.zero? + + (numerator.to_f / denominator * 100).round(1) + end + + def pack(current, previous, mode) + { current: current, previous: previous, trend: trend(current, previous, mode) } + end + + def trend(current, previous, mode) + case mode + when :percent + previous.zero? ? 0 : ((current - previous).to_f / previous * 100).round(1) + else # :point and :absolute are both current - previous + (current - previous).round(1) + end + end +end diff --git a/enterprise/app/builders/captain/assistant_stats_window.rb b/enterprise/app/builders/captain/assistant_stats_window.rb new file mode 100644 index 000000000..4495ef1e8 --- /dev/null +++ b/enterprise/app/builders/captain/assistant_stats_window.rb @@ -0,0 +1,78 @@ +# Resolves the current and previous comparison windows for Captain assistant +# stats. `range` is either a day count ('7', '30', '90') or a named period +# ('this_month', 'last_month'). The previous window mirrors the current one: the +# preceding N days for day ranges, or the preceding month for month ranges. +# `timezone_offset` is the viewer's UTC offset in hours (as the reports API sends +# it), so month/day boundaries anchor to the viewer's day rather than UTC. +# +# Shared by Captain::AssistantStatsBuilder (which needs both windows) and +# Captain::AssistantDrilldownBuilder (which drills into the current window), so a +# drilldown always covers exactly the rows its stat card counted. +class Captain::AssistantStatsWindow + include TimezoneHelper + + DEFAULT_RANGE = '30'.freeze + ALLOWED_RANGES = %w[7 30 90 this_month last_month].freeze + + attr_reader :range + + def initialize(range = DEFAULT_RANGE, timezone_offset = nil) + @range = ALLOWED_RANGES.include?(range.to_s) ? range.to_s : DEFAULT_RANGE + @timezone = timezone_name_from_offset(timezone_offset) || Time.zone + end + + def current + resolved_ranges[:current] + end + + def previous + resolved_ranges[:previous] + end + + # Human-readable description of the period the current window covers, for + # grounding the LLM summary in real dates. + def period + { label: period_label, starts_on: current.first.to_date, ends_on: current.last.to_date } + end + + private + + def resolved_ranges + @resolved_ranges ||= case range + when 'this_month' then this_month_ranges + when 'last_month' then last_month_ranges + else day_ranges + end + end + + # Current time anchored to the viewer's timezone, so calendar boundaries land on + # the viewer's day instead of UTC's. + def now + @now ||= Time.current.in_time_zone(@timezone) + end + + def this_month_ranges + start = now.beginning_of_month + elapsed = now - start + previous_start = start - 1.month + # Clamp to the previous month's end so a longer current month can't pull the + # comparison window into the current month and double-count its rows. + previous_end = [previous_start + elapsed, previous_start.end_of_month].min + { current: start..now, previous: previous_start..previous_end } + end + + def last_month_ranges + start = (now - 1.month).beginning_of_month + previous_start = start - 1.month + { current: start..start.end_of_month, previous: previous_start..previous_start.end_of_month } + end + + def day_ranges + days = range.to_i + { current: (now - days.days)..now, previous: (now - (2 * days).days)..(now - days.days) } + end + + def period_label + { 'this_month' => 'this month', 'last_month' => 'last month' }[range] || "the last #{range.to_i} days" + end +end diff --git a/enterprise/app/controllers/api/v1/accounts/captain/assistants_controller.rb b/enterprise/app/controllers/api/v1/accounts/captain/assistants_controller.rb index df9dfe5cc..4fbb93d20 100644 --- a/enterprise/app/controllers/api/v1/accounts/captain/assistants_controller.rb +++ b/enterprise/app/controllers/api/v1/accounts/captain/assistants_controller.rb @@ -2,7 +2,7 @@ class Api::V1::Accounts::Captain::AssistantsController < Api::V1::Accounts::Base before_action :current_account before_action -> { check_authorization(Captain::Assistant) } - before_action :set_assistant, only: [:show, :update, :destroy, :playground] + before_action :set_assistant, only: [:show, :update, :destroy, :playground, :stats, :summary, :drilldown] def index @assistants = account_assistants.ordered @@ -43,8 +43,53 @@ class Api::V1::Accounts::Captain::AssistantsController < Api::V1::Accounts::Base @tools = assistant.available_agent_tools end + def stats + render json: Captain::AssistantStatsBuilder.new(@assistant, params[:range], params[:timezone_offset]).metrics + end + + def summary + result = cached_or_generated_summary(Captain::AssistantStatsBuilder.new(@assistant, params[:range], params[:timezone_offset])) + + if result[:error] + render json: { error: result[:error] }, status: :unprocessable_content + else + render json: { message: result[:message] } + end + end + + def drilldown + return head :unprocessable_entity unless Captain::AssistantDrilldownBuilder.supported_metric?(params[:metric]) + + render json: Captain::AssistantDrilldownBuilder.new(@assistant, drilldown_params).build + end + private + def drilldown_params + params.permit(:metric, :range, :timezone_offset, :page, :per_page) + end + + def cached_or_generated_summary(builder) + cache_key = summary_cache_key(builder.range) + cached = Rails.cache.read(cache_key) + return cached if cached + + result = Captain::OverviewSummaryService.new( + account: Current.account, + assistant: @assistant, + first_name: Current.user.name.to_s.split.first, + stats: builder.metrics, + period: builder.period + ).perform + # Don't cache transient LLM/config failures, otherwise every reload returns 422 for the next hour. + Rails.cache.write(cache_key, result, expires_in: 1.hour) unless result[:error] + result + end + + def summary_cache_key(range) + "captain_overview_summary/#{@assistant.id}/#{Current.user.id}/#{range}/#{Date.current}" + end + def set_assistant @assistant = account_assistants.find(params[:id]) end diff --git a/enterprise/app/policies/captain/assistant_policy.rb b/enterprise/app/policies/captain/assistant_policy.rb index bbde3ffb0..573c0400c 100644 --- a/enterprise/app/policies/captain/assistant_policy.rb +++ b/enterprise/app/policies/captain/assistant_policy.rb @@ -11,6 +11,14 @@ class Captain::AssistantPolicy < ApplicationPolicy true end + def summary? + true + end + + def drilldown? + @account_user.administrator? + end + def tools? @account_user.administrator? end diff --git a/lib/captain/overview_summary_service.rb b/lib/captain/overview_summary_service.rb new file mode 100644 index 000000000..e11132d42 --- /dev/null +++ b/lib/captain/overview_summary_service.rb @@ -0,0 +1,83 @@ +# Generates the LLM welcome summary for the Captain Overview page from the +# assistant's stats hash (see Captain::AssistantStatsBuilder). Renders the +# captain_overview_summary.liquid prompt and returns markdown. +class Captain::OverviewSummaryService < Captain::BaseTaskService + pattr_initialize [:account!, :assistant!, :first_name!, :stats!, :period!] + + def perform + api_response = make_api_call( + feature: 'editor', + messages: [ + { role: 'system', content: system_prompt }, + { role: 'user', content: 'Write the summary.' } + ] + ) + + return api_response if api_response[:error] + + { message: api_response[:message] } + end + + private + + def system_prompt + Liquid::Template.parse(prompt_from_file('captain_overview_summary')).render(prompt_variables) + end + + def prompt_variables + stat_variables.merge(period_variables) + end + + def stat_variables + { + 'first_name' => first_name.to_s, + 'assistant_name' => assistant.name.to_s, + 'conversations_handled' => current(:conversations_handled), + 'hours_saved' => current(:hours_saved), + 'auto_resolution_rate' => current(:auto_resolution_rate), + 'auto_resolution_trend' => trend(:auto_resolution_rate), + 'handoff_rate' => current(:handoff_rate), + 'handoff_trend' => trend(:handoff_rate), + 'reopen_rate' => current(:reopen_rate), + 'reopen_trend' => trend(:reopen_rate), + 'knowledge_coverage' => stats.dig(:knowledge, :coverage).to_s, + 'knowledge_approved' => stats.dig(:knowledge, :approved).to_s, + 'knowledge_documents' => stats.dig(:knowledge, :documents).to_s + } + end + + def period_variables + { + 'today' => formatted_date(Time.zone.today), + 'period_label' => period[:label].to_s, + 'period_start' => formatted_date(period[:starts_on]), + 'period_end' => formatted_date(period[:ends_on]) + } + end + + def formatted_date(date) + date.strftime('%B %-d, %Y') + end + + def current(key) + stats.dig(key, :current).to_s + end + + def trend(key) + stats.dig(key, :trend).to_s + end + + def event_name + 'captain_overview_summary' + end + + def use_account_openai_hook? + true + end + + # The overview summary is an internal analytics readout, not a customer-facing + # response, so it should not consume or be blocked by the captain_responses quota. + def counts_toward_usage? + false + end +end diff --git a/lib/integrations/openai/openai_prompts/captain_overview_summary.liquid b/lib/integrations/openai/openai_prompts/captain_overview_summary.liquid new file mode 100644 index 000000000..19c32ed7c --- /dev/null +++ b/lib/integrations/openai/openai_prompts/captain_overview_summary.liquid @@ -0,0 +1,38 @@ +You are writing a short, warm summary of how an AI support assistant named "{{ assistant_name }}" performed over a reporting period, for {{ first_name }}, the person who manages it. + +Voice and format: +- Address {{ first_name }} directly and open with "Hey {{ first_name }},". Be conversational, never robotic. +- Always call the assistant by its name, {{ assistant_name }}. Never call it "Captain", "the assistant", or "your assistant". +- This is a static, read-only poster on an analytics dashboard, not a chat. The reader cannot reply or ask you for anything. Never ask a question, invite a reply, offer further help, or say things like "let me know" or "I can dive in". +- Write 2 to 4 sentences in one short paragraph. Add a second short paragraph only for a genuinely useful heads-up. +- Output plain markdown only: no headings, lists, preamble, or sign-off. Do not use em dashes. +- Never state an exact figure. This summary is cached and the live numbers keep moving, so a precise value would quickly look wrong. Round every number down to a clean approximation and soften it with words like "around", "roughly", "about", "nearly", "just over", or "upwards of". For example, render **1,248** as "upwards of **1,200**", **63.2%** as "around **60%**", and **612** hours as "roughly **600** hours". For a small count, use a loose phrase like "a handful" instead of the exact number. +- Wrap the approximate figure in **double asterisks** so the interface can highlight it. Bold only the figures, never whole phrases or the softening word. + +Timing: today is {{ today }}. These stats cover {{ period_label }} ({{ period_start }} to {{ period_end }}). You may lightly reference the month, the season, or how far into the period things stand when it genuinely fits, but never invent events or facts. + +The stats for this period: + +- Conversations handled: {{ conversations_handled }}. Distinct conversations {{ assistant_name }} replied in at least once. Raw volume and adoption, not a measure of quality. +- Hours saved: {{ hours_saved }} hours. A rough, directional estimate of agent time saved. A feel-good figure, not exact measured labor. +- Auto-resolution rate: {{ auto_resolution_rate }}% ({{ auto_resolution_trend }} points vs previous period). Of the conversations it handled, the share {{ assistant_name }} resolved on its own with no human reply. The core performance signal; higher is better. +- Handoff rate: {{ handoff_rate }}% ({{ handoff_trend }} points vs previous period). Of the conversations it handled, the share it escalated to a human agent. The inverse of deflection; lower is better. +- Reopen-after-resolve rate: {{ reopen_rate }}% ({{ reopen_trend }} points vs previous period). Of the conversations it auto-resolved, the share later reopened. A quality signal; lower is better, and a high value means it closed conversations the customer was not actually done with. +- Knowledge base: {{ knowledge_approved }} approved FAQ answers, {{ knowledge_documents }} documents, {{ knowledge_coverage }}% coverage (the share of FAQ answers the team has approved). This is setup the team controls, not something {{ assistant_name }} earned. It is a leading indicator: low coverage tends to cause low auto-resolution. + + +Only the auto-resolution, handoff, and reopen rates reflect how {{ assistant_name }} actually performed, and they are the only things worth crediting it for. Conversations handled and hours saved are context. The knowledge base is an input, never a win to praise. + +How to judge the numbers (rough bands, do not quote them in the summary): +- Auto-resolution rate: below 30% is low and early-stage, 30 to 50% is decent, above 50% is genuinely strong. +- Handoff rate: above 60% is high, 30 to 60% is moderate, below 30% is strong. +- Reopen-after-resolve rate: below 5% is healthy, 5 to 15% is worth watching, above 15% is a real problem. +- Knowledge coverage: only worth mentioning when below 85% (below 60% is seriously thin), as a likely cause of weak auto-resolution. At 85% or above it is just the healthy baseline, so do not mention or praise it. +- When the conversation volume is small (roughly under 30), rates are noisy, so describe them tentatively and do not over-interpret a perfect or terrible looking percentage. + +Writing the summary: +- Cold start: if conversations handled is 0, there is no performance to report. Skip the auto-resolution, handoff, reopen, and hours-saved figures entirely. Instead note the knowledge base and say {{ assistant_name }} is set up and ready to start handling support (or ready to start once some knowledge is added, if the base is empty). Ignore the rest of these points in this case. +- Be honest and proportionate. Do not call a result impressive, strong, excellent, solid, flawless, or perfect unless it clears the "strong" band above. State a low or middling number plainly or as room to grow, never dressed up. A modest summary is fine and often correct. +- Lead with the genuinely strong results if there are any. If nothing clears the strong band, open plainly with the volume of work handled, without overselling it. +- Mention a trend only when it is meaningful, and judge it against the bands rather than the direction alone (a rate that rose but is still in the low band is not yet a win). +- Surface at most one proactive concern when a stat warrants it (a high handoff rate, a low auto-resolution rate, a rising reopen rate, or thin coverage). Skip it entirely when everything looks healthy. Keep it a calm observation about the data, not an alarm. diff --git a/lib/seeders/reports/assistant_conversation_creator.rb b/lib/seeders/reports/assistant_conversation_creator.rb new file mode 100644 index 000000000..542da1b42 --- /dev/null +++ b/lib/seeders/reports/assistant_conversation_creator.rb @@ -0,0 +1,203 @@ +# frozen_string_literal: true + +require 'faker' +require 'active_support/testing/time_helpers' + +# Seeds Captain assistant activity for the reports/overview test data. +# +# Produces a variety of assistant-handled conversations in a single web inbox so +# every Captain assistant overview metric (handled, auto-resolution, handoff, +# hours saved, reopen rate, conversation depth) has realistic data: +# - :resolved_by_assistant assistant answers and Captain auto-resolves +# - :handled_by_both assistant answers, a human also replies and resolves +# - :handed_off assistant answers, then hands off to a human +# - :resolved_and_reopened Captain resolves, then the conversation reopens +# +# Reporting events are fired through ReportingEventListener directly (mirroring +# ConversationCreator) so the same rows the builder reads from get populated. +class Seeders::Reports::AssistantConversationCreator + include ActiveSupport::Testing::TimeHelpers + + OUTCOMES = %i[resolved_by_assistant handled_by_both handed_off resolved_and_reopened].freeze + + def initialize(account:, assistant:, inbox:, resources:) + @account = account + @assistant = assistant + @inbox = inbox + @contacts = resources[:contacts] + @agents = inbox.members.to_a.presence || resources[:agents] + end + + def create_conversation(created_at:, outcome:) + conversation = nil + + travel_to(created_at) do + conversation = build_conversation + conversation.save! + seed_dialogue(conversation, outcome) + end + travel_back + + apply_outcome(conversation, created_at, outcome) + conversation + end + + private + + def build_conversation + contact = @contacts.sample + contact_inbox = @inbox.contact_inboxes.find_or_create_by!(contact: contact, source_id: SecureRandom.hex) + + contact_inbox.conversations.create!( + account: @account, + inbox: @inbox, + contact: contact, + priority: [nil, 'high', 'medium', 'low'].sample + ) + end + + # Builds the message exchange for the conversation while time is frozen at its + # creation moment. Every outcome starts with a customer question and at least + # one public assistant reply so the conversation lands in the assistant's + # handled set; some outcomes add a human reply or a handoff. + def seed_dialogue(conversation, outcome) + customer_message = incoming_message(conversation) + + travel(rand((20.seconds)..(5.minutes))) + assistant_reply(conversation, waiting_since: customer_message.created_at) + + case outcome + when :handed_off then seed_handoff(conversation) + when :handled_by_both then seed_human_turn(conversation) + else seed_assistant_follow_up(conversation) + end + end + + def seed_handoff(conversation) + travel(rand((1.minute)..(10.minutes))) + handoff_to_human(conversation) + travel(rand((1.minute)..(15.minutes))) + human_reply(conversation) + end + + def seed_human_turn(conversation) + travel(rand((1.minute)..(15.minutes))) + human_reply(conversation) + end + + # Pure assistant threads occasionally take a second turn, giving depth > 1. + def seed_assistant_follow_up(conversation) + return unless rand < 0.6 + + travel(rand((1.minute)..(10.minutes))) + incoming_message(conversation) + travel(rand((20.seconds)..(5.minutes))) + assistant_reply(conversation, waiting_since: Time.current) + end + + def apply_outcome(conversation, created_at, outcome) + resolved_at = created_at + rand((30.minutes)..(8.hours)) + + case outcome + when :resolved_by_assistant + resolve_by_captain(conversation, resolved_at) + when :handled_by_both + resolve_by_human(conversation, resolved_at) + when :handed_off + resolve_by_human(conversation, resolved_at) if rand < 0.6 + when :resolved_and_reopened + resolve_by_captain(conversation, resolved_at) + reopen(conversation, resolved_at + rand((1.hour)..(24.hours))) + end + end + + def incoming_message(conversation) + conversation.messages.create!( + account: @account, + inbox: @inbox, + message_type: :incoming, + content: Faker::Lorem.paragraph(sentence_count: rand(1..3)), + sender: conversation.contact + ) + end + + def assistant_reply(conversation, waiting_since:) + message = conversation.messages.create!( + account: @account, + inbox: @inbox, + message_type: :outgoing, + private: false, + content: Faker::Lorem.paragraph(sentence_count: rand(1..4)), + sender: @assistant + ) + trigger_reply_time(message, waiting_since) + message + end + + def human_reply(conversation) + agent = @agents.sample + conversation.update_column(:assignee_id, agent.id) if conversation.assignee_id.nil? # rubocop:disable Rails/SkipsModelValidations + + conversation.messages.create!( + account: @account, + inbox: @inbox, + message_type: :outgoing, + private: false, + content: Faker::Lorem.paragraph(sentence_count: rand(1..4)), + sender: agent + ) + end + + def resolve_by_captain(conversation, resolved_at) + mark_resolved(conversation, resolved_at) + travel_to(resolved_at) do + trigger_event('conversation_resolved', conversation) + trigger_event('conversation_captain_inference_resolved', conversation) + end + travel_back + end + + def resolve_by_human(conversation, resolved_at) + mark_resolved(conversation, resolved_at) + travel_to(resolved_at) do + trigger_event('conversation_resolved', conversation) + end + travel_back + end + + def reopen(conversation, reopened_at) + # rubocop:disable Rails/SkipsModelValidations + conversation.update_column(:status, :open) + conversation.update_column(:updated_at, reopened_at) + # rubocop:enable Rails/SkipsModelValidations + + travel_to(reopened_at) do + trigger_event('conversation_opened', conversation) + end + travel_back + end + + def handoff_to_human(conversation) + trigger_event('conversation_captain_inference_handoff', conversation) + end + + def mark_resolved(conversation, resolved_at) + # rubocop:disable Rails/SkipsModelValidations + conversation.update_column(:status, :resolved) + conversation.update_column(:updated_at, resolved_at) + # rubocop:enable Rails/SkipsModelValidations + end + + def trigger_event(name, conversation) + ReportingEventListener.instance.public_send( + name, Events::Base.new(name, Time.current, { conversation: conversation }) + ) + end + + def trigger_reply_time(message, waiting_since) + ReportingEventListener.instance.reply_created( + Events::Base.new('reply_created', Time.current, + { message: message, conversation: message.conversation, waiting_since: waiting_since }) + ) + end +end diff --git a/lib/seeders/reports/report_data_seeder.rb b/lib/seeders/reports/report_data_seeder.rb index 909818b72..fecb02deb 100644 --- a/lib/seeders/reports/report_data_seeder.rb +++ b/lib/seeders/reports/report_data_seeder.rb @@ -17,6 +17,9 @@ # - 5 teams with realistic distribution # - 30 labels with random assignments # - 3 inboxes with agent assignments +# - 1 Captain assistant bound to a single web inbox, with knowledge (FAQs + documents) +# and a variety of assistant-handled conversations (auto-resolved, handed off, +# handled with a human, resolved-then-reopened) for the assistant overview page # - Realistic reporting events with historical timestamps # # Note: This seeder clears existing data for the account before seeding. @@ -24,8 +27,9 @@ require 'faker' require_relative 'conversation_creator' require_relative 'message_creator' +require_relative 'assistant_conversation_creator' -# rubocop:disable Rails/Output +# rubocop:disable Rails/Output, Metrics/ClassLength class Seeders::Reports::ReportDataSeeder include ActiveSupport::Testing::TimeHelpers @@ -36,6 +40,11 @@ class Seeders::Reports::ReportDataSeeder TOTAL_LABELS = 30 TOTAL_INBOXES = 3 MESSAGES_PER_CONVERSATION = 5 + # Captain assistant conversations, split across the outcomes the overview page reports on. + TOTAL_ASSISTANT_CONVERSATIONS = 120 + ASSISTANT_KNOWLEDGE_APPROVED = 14 + ASSISTANT_KNOWLEDGE_PENDING = 6 + ASSISTANT_DOCUMENTS = 4 START_DATE = 3.months.ago # rubocop:disable Rails/RelativeDateConstant END_DATE = Time.current @@ -48,6 +57,8 @@ class Seeders::Reports::ReportDataSeeder @labels = [] @inboxes = [] @contacts = [] + @assistant = nil + @assistant_inbox = nil end def perform! @@ -61,8 +72,10 @@ class Seeders::Reports::ReportDataSeeder create_labels create_inboxes create_contacts + create_assistant create_conversations + create_assistant_conversations puts "Completed reports data seeding for account: #{@account.name}" end @@ -71,6 +84,7 @@ class Seeders::Reports::ReportDataSeeder def clear_existing_data puts "Clearing existing data for account: #{@account.id}" + clear_assistant_data @account.teams.destroy_all @account.conversations.destroy_all @account.labels.destroy_all @@ -80,6 +94,16 @@ class Seeders::Reports::ReportDataSeeder @account.reporting_events.destroy_all end + # Delete Captain records directly (assistant associations are destroy_async, which + # would leave rows around mid-reseed); order respects foreign keys. + def clear_assistant_data + assistant_ids = Captain::Assistant.for_account(@account.id).select(:id) + Captain::AssistantResponse.by_account(@account.id).delete_all + Captain::Document.for_account(@account.id).delete_all + CaptainInbox.where(captain_assistant_id: assistant_ids).delete_all + Captain::Assistant.for_account(@account.id).delete_all + end + def create_teams TOTAL_TEAMS.times do |i| team = @account.teams.create!( @@ -208,6 +232,80 @@ class Seeders::Reports::ReportDataSeeder print "\n" end + # One assistant, bound to a single web inbox (the first one), as the overview page expects. + def create_assistant + @account.enable_features!('captain_integration', 'captain_integration_v2') + @assistant_inbox = @inboxes.first + @assistant = Captain::Assistant.create!( + account: @account, + name: "#{Faker::Company.name} Copilot", + description: 'Captain assistant handling website support conversations.', + config: { feature_faq: true, feature_memory: true, product_name: @account.name } + ) + CaptainInbox.create!(captain_assistant: @assistant, inbox: @assistant_inbox) + create_assistant_knowledge + + puts "Created assistant '#{@assistant.name}' for inbox '#{@assistant_inbox.name}'" + end + + def create_assistant_knowledge + ASSISTANT_KNOWLEDGE_APPROVED.times { create_assistant_response(:approved) } + ASSISTANT_KNOWLEDGE_PENDING.times { create_assistant_response(:pending) } + + ASSISTANT_DOCUMENTS.times do + Captain::Document.create!( + account: @account, + assistant: @assistant, + name: Faker::Company.catch_phrase, + external_link: "https://#{Faker::Internet.domain_name}/#{Faker::Internet.slug}", + content: Faker::Lorem.paragraphs(number: rand(2..4)).join("\n\n"), + status: :available, + sync_status: :synced + ) + end + end + + def create_assistant_response(status) + Captain::AssistantResponse.create!( + account: @account, + assistant: @assistant, + question: "#{Faker::Lorem.sentence(word_count: rand(4..8)).chomp('.')}?", + answer: Faker::Lorem.paragraph(sentence_count: rand(2..4)), + status: status + ) + end + + def create_assistant_conversations + creator = Seeders::Reports::AssistantConversationCreator.new( + account: @account, + assistant: @assistant, + inbox: @assistant_inbox, + resources: { contacts: @contacts, agents: @agents } + ) + + outcomes = assistant_outcome_distribution + outcomes.each_with_index do |outcome, i| + created_at = Faker::Time.between(from: 65.days.ago, to: END_DATE) + creator.create_conversation(created_at: created_at, outcome: outcome) + + print "\rCreating assistant conversations: #{i + 1}/#{outcomes.size}" + end + + print "\n" + end + + # Weighted mix of outcomes so every overview metric has meaningful numbers, shuffled + # so they interleave across the time span rather than clustering by type. + def assistant_outcome_distribution + counts = { + resolved_by_assistant: (TOTAL_ASSISTANT_CONVERSATIONS * 0.4).round, + handled_by_both: (TOTAL_ASSISTANT_CONVERSATIONS * 0.25).round, + handed_off: (TOTAL_ASSISTANT_CONVERSATIONS * 0.2).round, + resolved_and_reopened: (TOTAL_ASSISTANT_CONVERSATIONS * 0.15).round + } + counts.flat_map { |outcome, count| [outcome] * count }.shuffle + end + def create_conversations conversation_creator = Seeders::Reports::ConversationCreator.new( account: @account, @@ -231,4 +329,4 @@ class Seeders::Reports::ReportDataSeeder print "\n" end end -# rubocop:enable Rails/Output +# rubocop:enable Rails/Output, Metrics/ClassLength diff --git a/spec/enterprise/builders/captain/assistant_stats_builder_spec.rb b/spec/enterprise/builders/captain/assistant_stats_builder_spec.rb new file mode 100644 index 000000000..6575c9856 --- /dev/null +++ b/spec/enterprise/builders/captain/assistant_stats_builder_spec.rb @@ -0,0 +1,271 @@ +require 'rails_helper' + +RSpec.describe Captain::AssistantStatsBuilder do + let(:account) { create(:account) } + let(:assistant) { create(:captain_assistant, account: account) } + let(:inbox) { create(:inbox, account: account) } + + before { create(:captain_inbox, captain_assistant: assistant, inbox: inbox) } + + describe '#metrics' do + # Two conversations handled in the current 30-day window, one in the previous. + let(:current_convo_a) { create(:conversation, account: account, inbox: inbox) } + let(:current_convo_b) { create(:conversation, account: account, inbox: inbox) } + let(:previous_convo) { create(:conversation, account: account, inbox: inbox) } + + before do + [current_convo_a, current_convo_b].each do |conversation| + create(:message, account: account, inbox: inbox, conversation: conversation, + sender: assistant, message_type: :outgoing, private: false, created_at: 5.days.ago) + end + create(:message, account: account, inbox: inbox, conversation: previous_convo, + sender: assistant, message_type: :outgoing, private: false, created_at: 45.days.ago) + end + + it 'returns every metric for the current and previous window' do + metrics = described_class.new(assistant, '30').metrics + + expect(metrics.keys).to contain_exactly( + :conversations_handled, :auto_resolution_rate, :handoff_rate, + :hours_saved, :reopen_rate, :conversation_depth, :knowledge + ) + expect(metrics[:conversations_handled]).to include(:current, :previous, :trend) + end + + it 'counts distinct handled conversations per window and the percent trend' do + handled = described_class.new(assistant, '30').metrics[:conversations_handled] + + expect(handled[:current]).to eq(2) + expect(handled[:previous]).to eq(1) + expect(handled[:trend]).to eq(100.0) + end + + it 'derives auto-resolution and handoff rates from reporting events on the handled set' do + create(:reporting_event, account: account, conversation: current_convo_a, + name: 'conversation_captain_inference_resolved') + create(:reporting_event, account: account, conversation: current_convo_b, + name: 'conversation_captain_inference_handoff') + + metrics = described_class.new(assistant, '30').metrics + + expect(metrics[:auto_resolution_rate][:current]).to eq(50.0) + expect(metrics[:handoff_rate][:current]).to eq(50.0) + end + + it 'does not count a bot resolve as an auto-resolution when the conversation was handed off' do + # convo_a: handoff, customer goes quiet, resolve lands without an agent message, so the + # listener still emits conversation_bot_resolved for the handed-off conversation. It must + # not count as an auto-resolution, but still counts as a handoff. + create(:reporting_event, account: account, conversation: current_convo_a, + name: 'conversation_bot_handoff') + create(:reporting_event, account: account, conversation: current_convo_a, + name: 'conversation_bot_resolved') + # convo_b: a clean bot resolve with no handoff still counts, so the exclusion is scoped + # to handed-off conversations and doesn't drop every bot resolve. + create(:reporting_event, account: account, conversation: current_convo_b, + name: 'conversation_bot_resolved') + + metrics = described_class.new(assistant, '30').metrics + + expect(metrics[:auto_resolution_rate][:current]).to eq(50.0) + expect(metrics[:handoff_rate][:current]).to eq(50.0) + end + + it 'still counts an inference resolve when the conversation was also handed off' do + create(:reporting_event, account: account, conversation: current_convo_a, + name: 'conversation_captain_inference_handoff') + create(:reporting_event, account: account, conversation: current_convo_a, + name: 'conversation_captain_inference_resolved') + + metrics = described_class.new(assistant, '30').metrics + + expect(metrics[:auto_resolution_rate][:current]).to eq(50.0) + expect(metrics[:handoff_rate][:current]).to eq(50.0) + end + + it 'excludes resolution events that fall outside the current window' do + create(:reporting_event, account: account, conversation: current_convo_a, + name: 'conversation_captain_inference_resolved', created_at: 60.days.ago) + + metrics = described_class.new(assistant, '30').metrics + + expect(metrics[:auto_resolution_rate][:current]).to eq(0.0) + end + + it 'computes conversation depth as public replies per handled conversation' do + depth = described_class.new(assistant, '30').metrics[:conversation_depth] + + # 2 public outgoing replies across 2 distinct conversations in the current window. + expect(depth[:current]).to eq(1.0) + end + + it 'ignores private notes and incoming messages when counting public replies' do + create(:message, account: account, inbox: inbox, conversation: current_convo_a, + sender: assistant, message_type: :outgoing, private: true, created_at: 5.days.ago) + + depth = described_class.new(assistant, '30').metrics[:conversation_depth] + + expect(depth[:current]).to eq(1.0) + end + end + + describe 'range handling' do + it 'accepts the allowed day and named ranges' do + %w[7 30 90 this_month last_month].each do |allowed| + expect(described_class.new(assistant, allowed).range).to eq(allowed) + end + end + + it 'falls back to the default range for values outside the allowed set' do + expect(described_class.new(assistant, '365000').range).to eq('30') + expect(described_class.new(assistant, 'bogus').range).to eq('30') + expect(described_class.new(assistant, nil).range).to eq('30') + end + end + + describe '#metrics reopen_rate' do + # A conversation the assistant handled (messaged) inside the current 30-day window. + let(:conversation) { create(:conversation, account: account, inbox: inbox) } + + before do + create(:message, account: account, inbox: inbox, conversation: conversation, + sender: assistant, message_type: :outgoing, private: false, created_at: 8.days.ago) + end + + it 'counts a reopen that happened after the captain resolve' do + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_bot_resolved', event_start_time: 6.days.ago, event_end_time: 6.days.ago) + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_opened', value: 120, event_start_time: 6.days.ago, event_end_time: 4.days.ago) + + expect(described_class.new(assistant, '30').metrics[:reopen_rate][:current]).to eq(100.0) + end + + it 'ignores a human resolve/reopen that happened before the captain resolve' do + # Earlier resolve/reopen cycle, then Captain resolves later in the same window. + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_opened', value: 120, event_start_time: 20.days.ago, event_end_time: 18.days.ago) + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_bot_resolved', event_start_time: 5.days.ago, event_end_time: 5.days.ago) + + expect(described_class.new(assistant, '30').metrics[:reopen_rate][:current]).to eq(0.0) + end + + it 'counts an evaluated-path reopen when bot_resolved is skipped and the inference event is newer' do + # Prior human reply => create_bot_resolved_event skips conversation_bot_resolved, so the cohort + # only holds the inference event, which is dispatched a moment after the generic conversation_resolved + # that seeds the reopen's event_start_time. The match must use the reopen's actual reopen time. + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_captain_inference_resolved', + event_start_time: 6.days.ago, event_end_time: 6.days.ago + 1.second) + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_opened', value: 120, event_start_time: 6.days.ago, event_end_time: 3.days.ago) + + expect(described_class.new(assistant, '30').metrics[:reopen_rate][:current]).to eq(100.0) + end + + it 'counts both inference and time-based bot resolves in the denominator' do + # conversation: inference-resolved and reopened + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_captain_inference_resolved', event_start_time: 6.days.ago, event_end_time: 6.days.ago) + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_opened', value: 120, event_start_time: 6.days.ago, event_end_time: 4.days.ago) + # other: time-based bot-resolved, never reopened + other = create(:conversation, account: account, inbox: inbox) + create(:message, account: account, inbox: inbox, conversation: other, + sender: assistant, message_type: :outgoing, private: false, created_at: 8.days.ago) + create(:reporting_event, account: account, inbox: inbox, conversation: other, + name: 'conversation_bot_resolved', event_start_time: 6.days.ago, event_end_time: 6.days.ago) + + expect(described_class.new(assistant, '30').metrics[:reopen_rate][:current]).to eq(50.0) + end + + it 'ignores a reopen that landed after a completed window ended' do + travel_to(Time.utc(2026, 7, 15)) do + convo = create(:conversation, account: account, inbox: inbox) + create(:message, account: account, inbox: inbox, conversation: convo, + sender: assistant, message_type: :outgoing, private: false, created_at: Time.utc(2026, 6, 10)) + create(:reporting_event, account: account, inbox: inbox, conversation: convo, + name: 'conversation_bot_resolved', created_at: Time.utc(2026, 6, 12), + event_start_time: Time.utc(2026, 6, 12), event_end_time: Time.utc(2026, 6, 12)) + # Reopened on July 1, after the June window closed; June's rate must not count it. + create(:reporting_event, account: account, inbox: inbox, conversation: convo, + name: 'conversation_opened', value: 120, + event_start_time: Time.utc(2026, 6, 12), event_end_time: Time.utc(2026, 7, 1)) + + expect(described_class.new(assistant, 'last_month').metrics[:reopen_rate][:current]).to eq(0.0) + end + end + + it 'derives the cohort from handled conversations, not current inbox membership' do + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_captain_inference_resolved', event_start_time: 6.days.ago, event_end_time: 6.days.ago) + create(:reporting_event, account: account, inbox: inbox, conversation: conversation, + name: 'conversation_opened', value: 120, event_start_time: 6.days.ago, event_end_time: 4.days.ago) + # The assistant is later removed from the inbox; the cohort must still resolve via handled messages. + CaptainInbox.where(captain_assistant: assistant).delete_all + + expect(described_class.new(assistant, '30').metrics[:reopen_rate][:current]).to eq(100.0) + end + end + + describe 'timezone anchoring' do + # 2026-07-01 03:00 UTC is still 2026-06-30 in any timezone behind UTC by 4h+. + it 'anchors the this_month window to the supplied offset, not UTC' do + travel_to(Time.utc(2026, 7, 1, 3, 0, 0)) do + utc = described_class.new(assistant, 'this_month').period + la = described_class.new(assistant, 'this_month', -7).period + + expect(utc[:starts_on]).to eq(Date.new(2026, 7, 1)) + expect(la[:starts_on]).to eq(Date.new(2026, 6, 1)) + expect(la[:ends_on]).to eq(Date.new(2026, 6, 30)) + end + end + + it 'defaults to UTC when no offset is given' do + travel_to(Time.utc(2026, 7, 1, 3, 0, 0)) do + expect(described_class.new(assistant, 'this_month').period[:starts_on]).to eq(Date.new(2026, 7, 1)) + end + end + end + + describe '#metrics knowledge' do + before do + create_list(:captain_assistant_response, 3, assistant: assistant, account: account, status: :approved) + create(:captain_assistant_response, assistant: assistant, account: account, status: :pending) + create_list(:captain_document, 2, assistant: assistant, account: account) + end + + it 'returns approved, pending, document counts and coverage' do + knowledge = described_class.new(assistant, '30').metrics[:knowledge] + + expect(knowledge).to eq(approved: 3, pending: 1, documents: 2, coverage: 75) + end + + it 'reports zero coverage when there are no responses' do + Captain::AssistantResponse.where(assistant: assistant).delete_all + + knowledge = described_class.new(assistant, '30').metrics[:knowledge] + + expect(knowledge[:coverage]).to eq(0) + end + end + + describe '#period' do + it 'labels a day range and exposes its bounds' do + period = described_class.new(assistant, '30').period + + expect(period[:label]).to eq('the last 30 days') + expect(period[:starts_on]).to eq(30.days.ago.to_date) + expect(period[:ends_on]).to eq(Time.zone.today) + end + + it 'labels the this_month range' do + expect(described_class.new(assistant, 'this_month').period[:label]).to eq('this month') + end + + it 'labels the last_month range' do + expect(described_class.new(assistant, 'last_month').period[:label]).to eq('last month') + end + end +end diff --git a/spec/enterprise/controllers/api/v1/accounts/captain/assistants_controller_spec.rb b/spec/enterprise/controllers/api/v1/accounts/captain/assistants_controller_spec.rb index 4689defaf..afb6aa2de 100644 --- a/spec/enterprise/controllers/api/v1/accounts/captain/assistants_controller_spec.rb +++ b/spec/enterprise/controllers/api/v1/accounts/captain/assistants_controller_spec.rb @@ -252,6 +252,48 @@ RSpec.describe 'Api::V1::Accounts::Captain::Assistants', type: :request do end end + describe 'GET /api/v1/accounts/{account.id}/captain/assistants/{id}/summary' do + let(:assistant) { create(:captain_assistant, account: account) } + let(:alice) { create(:user, account: account, role: :administrator, name: 'Alice Adams') } + let(:bob) { create(:user, account: account, role: :administrator, name: 'Bob Brown') } + let(:summary_service) { instance_double(Captain::OverviewSummaryService) } + + def get_summary(user) + get "/api/v1/accounts/#{account.id}/captain/assistants/#{assistant.id}/summary", + params: { range: '30' }, + headers: user.create_new_auth_token, + as: :json + end + + before do + # Test env uses a null store; swap in a real store so caching behaviour is observable. + allow(Rails).to receive(:cache).and_return(ActiveSupport::Cache::MemoryStore.new) + allow(Captain::OverviewSummaryService).to receive(:new).and_return(summary_service) + end + + it 'caches the summary per viewer so one user never receives another user\'s greeting' do + allow(summary_service).to receive(:perform).and_return({ message: 'Hi Alice' }) + + get_summary(alice) + get_summary(alice) # served from Alice's cache, no regeneration + get_summary(bob) # distinct cache key, regenerated for Bob + + expect(response).to have_http_status(:success) + expect(Captain::OverviewSummaryService).to have_received(:new).twice + end + + it 'does not cache failures so a transient error is retried' do + allow(summary_service).to receive(:perform).and_return({ error: 'LLM unavailable' }) + + get_summary(alice) + get_summary(alice) + + expect(response).to have_http_status(:unprocessable_content) + expect(json_response[:error]).to eq('LLM unavailable') + expect(Captain::OverviewSummaryService).to have_received(:new).twice + end + end + describe 'POST /api/v1/accounts/{account.id}/captain/assistants/{id}/playground' do let(:assistant) { create(:captain_assistant, account: account) } let(:valid_params) do diff --git a/spec/enterprise/policies/captain/assistant_policy_spec.rb b/spec/enterprise/policies/captain/assistant_policy_spec.rb index e3c62846f..e04b680e4 100644 --- a/spec/enterprise/policies/captain/assistant_policy_spec.rb +++ b/spec/enterprise/policies/captain/assistant_policy_spec.rb @@ -22,7 +22,7 @@ RSpec.describe Captain::AssistantPolicy, type: :policy do end end - permissions :tools?, :create?, :update?, :destroy?, :sync? do + permissions :tools?, :create?, :update?, :destroy?, :sync?, :drilldown? do context 'when administrator' do it { expect(assistant_policy).to permit(administrator_context, assistant) } end