Aakash BakhleandGitHub
d57354c8b5
Tightens the resolved-conversation FAQ generator so it only proposes durable, reusable FAQ candidates supported by human support-agent messages. The implementation now sends a conversation-FAQ-specific transcript to the LLM: customer messages plus real human support-agent messages only, excluding bot, private, activity, and template messages. ## Closes - https://linear.app/chatwoot/issue/CW-7494/tighten-conversation-faq-generation-prompt ## What changed - Added a human-only transcript builder in `ConversationFaqService` instead of using the generic `conversation.to_llm_text` output. - Excluded bot/agent-bot messages before the LLM call, which removes the main bot-line leakage class deterministically. - Preserved native-channel human replies where outgoing messages are stored as `external_echo` without a `User` sender. - Kept a prompt decision gate requiring each FAQ to be backed by a complete public human-agent answer. - Added generic no-FAQ classes for spam, wrong-service conversations, private account/payment/order/certificate/troubleshooting cases, support workflow mechanics, and direct-link/file/quote outputs. - Added a separate `conversation_faq_generation` model route defaulting to `gpt-5.2`, while keeping `document_faq_generation` on its existing `gpt-4.1-mini` default. Conversation FAQ generation passes that feature default ahead of the legacy global `CAPTAIN_OPEN_AI_MODEL` setting unless an account-level override is configured. - Kept the prompt domain-neutral so it can still generate reusable product, service, policy, setup, and process FAQs outside SaaS contexts. ## Sampling notes - Production Langfuse traces showed `llm.captain.conversation_faq` calls using `gpt-4.1` in the sampled account set. - Locally, `Llm::FeatureRouter.resolve(feature: 'conversation_faq_generation')` now resolves to `gpt-5.2`. - Reviewed recent production `llm.captain.conversation_faq` traces across 13+ accounts in compact form. - Replayed 20 full traces across 10 accounts/domains, including education, hosting, retail/auto, APIs, logistics, tax/fiscal workflows, and Chatwoot account 1. - Explicit `gpt-5.2` replay with human-only conversation history returned no FAQ for 15/20 traces. - A comparison replay with `gpt-4.1-mini` returned no FAQ for only 7/20 traces, bringing back several private/order/payment/support-workflow cases. - Remaining non-empty `gpt-5.2` outputs are now mostly borderline/possibly useful human-agent-derived FAQs rather than obvious bot-sourced answers. ## How to test - Resolve conversations where the answer came only from the bot; no pending FAQ should be generated. - Resolve spam, unrelated, wrong-service, or private payment/order/account conversations; no pending FAQ should be generated. - Resolve conversations that require account/order/payment/login/private verification or a human handoff; no pending FAQ should be generated. - Resolve a conversation where a human agent gives a stable, reusable help-center answer; the generated pending FAQ should be general and self-contained.
290 lines
11 KiB
Ruby
290 lines
11 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
require 'rails_helper'
|
|
|
|
RSpec.describe 'Api::V1::Accounts::Captain::Preferences', type: :request do
|
|
let(:account) { create(:account) }
|
|
let(:admin) { create(:user, account: account, role: :administrator) }
|
|
let(:agent) { create(:user, account: account, role: :agent) }
|
|
|
|
def json_response
|
|
JSON.parse(response.body, symbolize_names: true)
|
|
end
|
|
|
|
describe 'GET /api/v1/accounts/{account.id}/captain/preferences' do
|
|
context 'when it is an unauthenticated user' do
|
|
it 'returns unauthorized' do
|
|
get "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:unauthorized)
|
|
end
|
|
end
|
|
|
|
context 'when it is an agent' do
|
|
it 'returns captain config' do
|
|
get "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: agent.create_new_auth_token,
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response).to have_key(:providers)
|
|
expect(json_response).to have_key(:models)
|
|
expect(json_response).to have_key(:features)
|
|
end
|
|
end
|
|
|
|
context 'when it is an admin' do
|
|
it 'returns captain config' do
|
|
get "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response).to have_key(:providers)
|
|
expect(json_response).to have_key(:models)
|
|
expect(json_response).to have_key(:features)
|
|
end
|
|
|
|
it 'returns effective model provider and source for each feature' do
|
|
account.update!(captain_models: { 'editor' => 'gpt-4.1' })
|
|
|
|
get "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response.dig(:features, :editor)).to include(
|
|
model: 'gpt-4.1',
|
|
selected: 'gpt-4.1',
|
|
provider: 'openai',
|
|
source: 'account_override'
|
|
)
|
|
expect(json_response.dig(:features, :label_suggestion)).to include(
|
|
model: Llm::Models.default_model_for('label_suggestion'),
|
|
selected: Llm::Models.default_model_for('label_suggestion'),
|
|
provider: 'openai',
|
|
source: 'default'
|
|
)
|
|
end
|
|
|
|
it 'returns the assistant YAML default for V1 accounts' do
|
|
get "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response.dig(:features, :assistant)).to include(
|
|
default: Llm::Models.default_model_for('assistant'),
|
|
selected: Llm::Models.default_model_for('assistant'),
|
|
source: 'default'
|
|
)
|
|
end
|
|
|
|
it 'returns GPT-5.2 as the assistant default for V2 accounts' do
|
|
account.enable_features!('captain_integration_v2')
|
|
|
|
get "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response.dig(:features, :assistant)).to include(
|
|
default: Llm::FeatureRouter::CAPTAIN_V2_ASSISTANT_MODEL,
|
|
selected: Llm::FeatureRouter::CAPTAIN_V2_ASSISTANT_MODEL,
|
|
source: 'default'
|
|
)
|
|
end
|
|
|
|
it 'keeps the V2 assistant default when an account override is selected' do
|
|
account.enable_features!('captain_integration_v2')
|
|
account.update!(captain_models: { 'assistant' => 'gpt-5.1' })
|
|
|
|
get "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response.dig(:features, :assistant)).to include(
|
|
default: Llm::FeatureRouter::CAPTAIN_V2_ASSISTANT_MODEL,
|
|
selected: 'gpt-5.1',
|
|
source: 'account_override'
|
|
)
|
|
end
|
|
end
|
|
end
|
|
|
|
describe 'PUT /api/v1/accounts/{account.id}/captain/preferences' do
|
|
context 'when it is an unauthenticated user' do
|
|
it 'returns unauthorized' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
params: { captain_models: { editor: 'gpt-4.1-mini' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:unauthorized)
|
|
end
|
|
end
|
|
|
|
context 'when it is an agent' do
|
|
it 'returns forbidden' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: agent.create_new_auth_token,
|
|
params: { captain_models: { editor: 'gpt-4.1-mini' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:unauthorized)
|
|
end
|
|
end
|
|
|
|
context 'when it is an admin' do
|
|
it 'updates captain_models' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { editor: 'gpt-4.1-mini' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response).to have_key(:providers)
|
|
expect(json_response).to have_key(:models)
|
|
expect(json_response).to have_key(:features)
|
|
expect(account.reload.captain_models['editor']).to eq('gpt-4.1-mini')
|
|
end
|
|
|
|
it 'does not persist unknown captain model feature keys' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { editor: 'gpt-4.1-mini', unknown_feature: 'gpt-4.1' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(account.reload.captain_models).to eq('editor' => 'gpt-4.1-mini')
|
|
end
|
|
|
|
it 'rejects invalid captain model values for the feature' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { label_suggestion: 'gpt-5.1' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:unprocessable_entity)
|
|
expect(json_response[:message]).to include('not a valid model for label_suggestion')
|
|
expect(account.reload.captain_models).to be_nil
|
|
end
|
|
|
|
it 'removes blank captain model overrides' do
|
|
account.update!(captain_models: { 'editor' => 'gpt-4.1' })
|
|
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { editor: '' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(account.reload.captain_models).to be_nil
|
|
expect(json_response.dig(:features, :editor)).to include(
|
|
selected: Llm::Models.default_model_for('editor'),
|
|
source: 'default'
|
|
)
|
|
end
|
|
|
|
it 'updates captain_models for document FAQ generation' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { document_faq_generation: 'gpt-5.2' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response.dig(:features, :document_faq_generation, :selected)).to eq('gpt-5.2')
|
|
expect(account.reload.captain_models['document_faq_generation']).to eq('gpt-5.2')
|
|
end
|
|
|
|
it 'updates captain_models for conversation FAQ generation' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { conversation_faq_generation: 'gpt-4.1-mini' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response.dig(:features, :conversation_faq_generation, :selected)).to eq('gpt-4.1-mini')
|
|
expect(account.reload.captain_models['conversation_faq_generation']).to eq('gpt-4.1-mini')
|
|
end
|
|
|
|
it 'updates captain_models for PDF FAQ generation' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { pdf_faq_generation: 'gpt-5.2' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response.dig(:features, :pdf_faq_generation, :selected)).to eq('gpt-5.2')
|
|
expect(account.reload.captain_models['pdf_faq_generation']).to eq('gpt-5.2')
|
|
end
|
|
|
|
it 'updates captain_features' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_features: { editor: true } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response).to have_key(:providers)
|
|
expect(json_response).to have_key(:models)
|
|
expect(json_response).to have_key(:features)
|
|
expect(account.reload.captain_features['editor']).to be true
|
|
end
|
|
|
|
it 'merges with existing captain_models' do
|
|
account.update!(captain_models: { 'editor' => 'gpt-4.1-mini', 'assistant' => 'gpt-5.1' })
|
|
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_models: { editor: 'gpt-4.1' } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response).to have_key(:providers)
|
|
expect(json_response).to have_key(:models)
|
|
expect(json_response).to have_key(:features)
|
|
models = account.reload.captain_models
|
|
expect(models['editor']).to eq('gpt-4.1')
|
|
expect(models['assistant']).to eq('gpt-5.1') # Preserved
|
|
end
|
|
|
|
it 'merges with existing captain_features' do
|
|
account.update!(captain_features: { 'editor' => true, 'assistant' => false })
|
|
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: { captain_features: { editor: false } },
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response).to have_key(:providers)
|
|
expect(json_response).to have_key(:models)
|
|
expect(json_response).to have_key(:features)
|
|
features = account.reload.captain_features
|
|
expect(features['editor']).to be false
|
|
expect(features['assistant']).to be false # Preserved
|
|
end
|
|
|
|
it 'updates both models and features in single request' do
|
|
put "/api/v1/accounts/#{account.id}/captain/preferences",
|
|
headers: admin.create_new_auth_token,
|
|
params: {
|
|
captain_models: { editor: 'gpt-4.1-mini' },
|
|
captain_features: { editor: true }
|
|
},
|
|
as: :json
|
|
|
|
expect(response).to have_http_status(:success)
|
|
expect(json_response).to have_key(:providers)
|
|
expect(json_response).to have_key(:models)
|
|
expect(json_response).to have_key(:features)
|
|
account.reload
|
|
expect(account.captain_models['editor']).to eq('gpt-4.1-mini')
|
|
expect(account.captain_features['editor']).to be true
|
|
end
|
|
end
|
|
end
|
|
end
|