379e28df1f
The bot metrics dashboard can show `handoff_rate + resolution_rate >
100%`. A single conversation can accumulate both
`conversation_bot_handoff` and `conversation_bot_resolved` events, and
the rate queries count them independently against a shared denominator.
## How it happens
```
Customer messages bot inbox
│
▼
┌──────────┐
│ pending │ (bot handling)
└────┬─────┘
│ bot can't help
▼
┌──────────┐
│ open │ (handed off → conversation_bot_handoff event created)
└────┬─────┘
│ agent clicks "Resolve" WITHOUT sending a message
▼
┌──────────┐
│ resolved │ conversation_resolved fires
└──────────┘
│
▼
create_bot_resolved_event guard checks:
✅ inbox.active_bot?
✅ no outgoing messages with sender_type: 'User' ← agent never messaged!
│
▼
conversation_bot_resolved event ALSO created ← BUG
│
▼
Same conversation counted in BOTH rates → sum exceeds 100%
```
## Why fix at the read path, not the write path
An earlier attempt added guards in the listener to make the two events
mutually exclusive per conversation — deleting `bot_resolved` when a
handoff fires, suppressing resolutions when a handoff exists. This was
rejected because conversations can be reopened across multiple cycles
(bot resolves on day 1, customer returns on day 5, bot hands off).
Deleting the day-1 resolution corrupts historical reports, and the async
event dispatcher makes listener-level guards vulnerable to race
conditions.
## What this PR does
Within a reporting window, if a conversation has both events, **handoff
wins** — the conversation is excluded from the resolution count. This is
applied via SQL subquery across all three read paths:
```
┌─────────────────────────┐
│ Reporting Events DB │
│ │
│ conv_bot_handoff: [A,B] │
│ conv_bot_resolved: [A,C]│
└────────┬────────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
BotMetricsBuilder ReportHelper CountReportBuilder
(rate cards) (bot_summary) (timeseries charts)
│ │ │
▼ ▼ ▼
resolutions: resolutions: resolutions:
[A,C] minus [A,B] same logic same logic
= [C] only = [C] only = [C] only
Result: Conversation A → handoff only
Conversation B → handoff only
Conversation C → resolution only
```
For wide date ranges spanning multiple lifecycles, a conversation
bot-resolved in one cycle and handed off in a later cycle will only show
as a handoff. This is an acceptable tradeoff — the alternative (>100%
rates) is clearly worse, and narrow ranges handle this correctly since
the events fall into different windows. No reporting events are
modified, so historical data stays intact.
## Diagnostic tool
`rake bot_metrics:diagnose` — read-only task that prompts for account ID
and date range, shows a before/after rate comparison without modifying
data.
---------
Co-authored-by: aakashb95 <aakashbakhle@gmail.com>
Co-authored-by: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com>
84 lines
3.0 KiB
Ruby
84 lines
3.0 KiB
Ruby
require 'rails_helper'
|
|
|
|
RSpec.describe Reports::ReportMetricRegistry do
|
|
describe '.fetch' do
|
|
it 'returns the definition for raw-only count metrics' do
|
|
metric = described_class.fetch(:conversations_count)
|
|
|
|
expect(metric.name).to eq(:conversations_count)
|
|
expect(metric.count?).to be(true)
|
|
expect(metric.rollup_supported?).to be(false)
|
|
expect(metric.raw_event_name).to be_nil
|
|
end
|
|
|
|
it 'returns the definition for avg_resolution_time' do
|
|
metric = described_class.fetch(:avg_resolution_time)
|
|
|
|
expect(metric.name).to eq(:avg_resolution_time)
|
|
expect(metric.average?).to be(true)
|
|
expect(metric.raw_event_name).to eq(:conversation_resolved)
|
|
expect(metric.rollup_metric).to eq(:resolution_time)
|
|
expect(metric.summary_key).to eq(:avg_resolution_time)
|
|
end
|
|
|
|
it 'locks the distinct conversation strategy for bot_handoffs_count' do
|
|
metric = described_class.fetch(:bot_handoffs_count)
|
|
|
|
expect(metric.count?).to be(true)
|
|
expect(metric.raw_event_name).to eq(:conversation_bot_handoff)
|
|
expect(metric.rollup_metric).to eq(:bot_handoffs_count)
|
|
expect(metric.raw_count_strategy).to eq(:distinct_conversation)
|
|
end
|
|
|
|
it 'locks the handoff exclusion strategy for bot_resolutions_count' do
|
|
metric = described_class.fetch(:bot_resolutions_count)
|
|
|
|
expect(metric.count?).to be(true)
|
|
expect(metric.raw_event_name).to eq(:conversation_bot_resolved)
|
|
expect(metric.rollup_metric).to eq(:bot_resolutions_count)
|
|
expect(metric.raw_count_strategy).to eq(:exclude_bot_handoffs)
|
|
end
|
|
|
|
it 'returns nil for unsupported metrics' do
|
|
expect(described_class.fetch(:unknown_metric)).to be_nil
|
|
end
|
|
end
|
|
|
|
describe '.supported?' do
|
|
it 'returns true for supported raw-only metrics' do
|
|
expect(described_class.supported?(:conversations_count)).to be(true)
|
|
end
|
|
|
|
it 'returns false for unsupported metrics' do
|
|
expect(described_class.supported?(:unknown_metric)).to be(false)
|
|
end
|
|
end
|
|
|
|
describe '.rollup_supported?' do
|
|
it 'returns true for rollup-backed metrics' do
|
|
expect(described_class.rollup_supported?(:reply_time)).to be(true)
|
|
end
|
|
|
|
it 'returns false for raw-only metrics' do
|
|
expect(described_class.rollup_supported?(:conversations_count)).to be(false)
|
|
end
|
|
end
|
|
|
|
describe '.summary_metrics' do
|
|
it 'returns the summary metric definitions in registry order' do
|
|
expect(
|
|
described_class.summary_metrics.map do |metric|
|
|
[metric.name, metric.summary_key, metric.aggregate, metric.raw_event_name, metric.rollup_metric]
|
|
end
|
|
).to eq(
|
|
[
|
|
[:resolutions_count, :resolved_conversations_count, :count, :conversation_resolved, :resolutions_count],
|
|
[:avg_resolution_time, :avg_resolution_time, :average, :conversation_resolved, :resolution_time],
|
|
[:avg_first_response_time, :avg_first_response_time, :average, :first_response, :first_response],
|
|
[:reply_time, :avg_reply_time, :average, :reply_time, :reply_time]
|
|
]
|
|
)
|
|
end
|
|
end
|
|
end
|