Aakash BakhleandGitHub
d57354c8b5
Tightens the resolved-conversation FAQ generator so it only proposes durable, reusable FAQ candidates supported by human support-agent messages. The implementation now sends a conversation-FAQ-specific transcript to the LLM: customer messages plus real human support-agent messages only, excluding bot, private, activity, and template messages. ## Closes - https://linear.app/chatwoot/issue/CW-7494/tighten-conversation-faq-generation-prompt ## What changed - Added a human-only transcript builder in `ConversationFaqService` instead of using the generic `conversation.to_llm_text` output. - Excluded bot/agent-bot messages before the LLM call, which removes the main bot-line leakage class deterministically. - Preserved native-channel human replies where outgoing messages are stored as `external_echo` without a `User` sender. - Kept a prompt decision gate requiring each FAQ to be backed by a complete public human-agent answer. - Added generic no-FAQ classes for spam, wrong-service conversations, private account/payment/order/certificate/troubleshooting cases, support workflow mechanics, and direct-link/file/quote outputs. - Added a separate `conversation_faq_generation` model route defaulting to `gpt-5.2`, while keeping `document_faq_generation` on its existing `gpt-4.1-mini` default. Conversation FAQ generation passes that feature default ahead of the legacy global `CAPTAIN_OPEN_AI_MODEL` setting unless an account-level override is configured. - Kept the prompt domain-neutral so it can still generate reusable product, service, policy, setup, and process FAQs outside SaaS contexts. ## Sampling notes - Production Langfuse traces showed `llm.captain.conversation_faq` calls using `gpt-4.1` in the sampled account set. - Locally, `Llm::FeatureRouter.resolve(feature: 'conversation_faq_generation')` now resolves to `gpt-5.2`. - Reviewed recent production `llm.captain.conversation_faq` traces across 13+ accounts in compact form. - Replayed 20 full traces across 10 accounts/domains, including education, hosting, retail/auto, APIs, logistics, tax/fiscal workflows, and Chatwoot account 1. - Explicit `gpt-5.2` replay with human-only conversation history returned no FAQ for 15/20 traces. - A comparison replay with `gpt-4.1-mini` returned no FAQ for only 7/20 traces, bringing back several private/order/payment/support-workflow cases. - Remaining non-empty `gpt-5.2` outputs are now mostly borderline/possibly useful human-agent-derived FAQs rather than obvious bot-sourced answers. ## How to test - Resolve conversations where the answer came only from the bot; no pending FAQ should be generated. - Resolve spam, unrelated, wrong-service, or private payment/order/account conversations; no pending FAQ should be generated. - Resolve conversations that require account/order/payment/login/private verification or a human handoff; no pending FAQ should be generated. - Resolve a conversation where a human agent gives a stable, reusable help-center answer; the generated pending FAQ should be general and self-contained.
62 lines
1.9 KiB
Ruby
62 lines
1.9 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
require 'rails_helper'
|
|
|
|
RSpec.describe Llm::Models do
|
|
describe '.providers' do
|
|
it 'loads provider metadata from the config' do
|
|
expect(described_class.providers).to include(
|
|
'openai' => include('display_name' => 'OpenAI')
|
|
)
|
|
end
|
|
end
|
|
|
|
describe '.features' do
|
|
it 'keeps every feature default in the allowed model list' do
|
|
described_class.features.each do |feature_key, config|
|
|
expect(config['models']).to include(config['default']), "#{feature_key} default model must be allowed"
|
|
end
|
|
end
|
|
|
|
it 'references existing models from every feature' do
|
|
described_class.features.each do |feature_key, config|
|
|
missing_models = config['models'].reject { |model_name| described_class.models.key?(model_name) }
|
|
|
|
expect(missing_models).to be_empty, "#{feature_key} references missing models: #{missing_models.join(', ')}"
|
|
end
|
|
end
|
|
|
|
it 'routes document and conversation FAQ generation independently' do
|
|
expect(described_class.default_model_for('document_faq_generation')).to eq('gpt-4.1-mini')
|
|
expect(described_class.default_model_for('conversation_faq_generation')).to eq('gpt-5.2')
|
|
end
|
|
end
|
|
|
|
describe '.models' do
|
|
it 'references existing providers from every model' do
|
|
missing_providers = described_class.models.filter_map do |model_name, config|
|
|
provider = config['provider']
|
|
next if described_class.providers.key?(provider)
|
|
|
|
"#{model_name}: #{provider}"
|
|
end
|
|
|
|
expect(missing_providers).to be_empty
|
|
end
|
|
end
|
|
|
|
describe '.feature_config' do
|
|
it 'returns model metadata for a feature' do
|
|
config = described_class.feature_config('editor')
|
|
|
|
expect(config[:default]).to eq('gpt-4.1-mini')
|
|
expect(config[:models].first).to include(
|
|
id: 'gpt-4.1-mini',
|
|
display_name: 'GPT-4.1 Mini',
|
|
provider: 'openai',
|
|
credit_multiplier: 1
|
|
)
|
|
end
|
|
end
|
|
end
|