develop
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aakash BakhleandGitHub
|
d57354c8b5 |
feat: tighten conversation FAQ generation prompt (#14957)
Tightens the resolved-conversation FAQ generator so it only proposes durable, reusable FAQ candidates supported by human support-agent messages. The implementation now sends a conversation-FAQ-specific transcript to the LLM: customer messages plus real human support-agent messages only, excluding bot, private, activity, and template messages. ## Closes - https://linear.app/chatwoot/issue/CW-7494/tighten-conversation-faq-generation-prompt ## What changed - Added a human-only transcript builder in `ConversationFaqService` instead of using the generic `conversation.to_llm_text` output. - Excluded bot/agent-bot messages before the LLM call, which removes the main bot-line leakage class deterministically. - Preserved native-channel human replies where outgoing messages are stored as `external_echo` without a `User` sender. - Kept a prompt decision gate requiring each FAQ to be backed by a complete public human-agent answer. - Added generic no-FAQ classes for spam, wrong-service conversations, private account/payment/order/certificate/troubleshooting cases, support workflow mechanics, and direct-link/file/quote outputs. - Added a separate `conversation_faq_generation` model route defaulting to `gpt-5.2`, while keeping `document_faq_generation` on its existing `gpt-4.1-mini` default. Conversation FAQ generation passes that feature default ahead of the legacy global `CAPTAIN_OPEN_AI_MODEL` setting unless an account-level override is configured. - Kept the prompt domain-neutral so it can still generate reusable product, service, policy, setup, and process FAQs outside SaaS contexts. ## Sampling notes - Production Langfuse traces showed `llm.captain.conversation_faq` calls using `gpt-4.1` in the sampled account set. - Locally, `Llm::FeatureRouter.resolve(feature: 'conversation_faq_generation')` now resolves to `gpt-5.2`. - Reviewed recent production `llm.captain.conversation_faq` traces across 13+ accounts in compact form. - Replayed 20 full traces across 10 accounts/domains, including education, hosting, retail/auto, APIs, logistics, tax/fiscal workflows, and Chatwoot account 1. - Explicit `gpt-5.2` replay with human-only conversation history returned no FAQ for 15/20 traces. - A comparison replay with `gpt-4.1-mini` returned no FAQ for only 7/20 traces, bringing back several private/order/payment/support-workflow cases. - Remaining non-empty `gpt-5.2` outputs are now mostly borderline/possibly useful human-agent-derived FAQs rather than obvious bot-sourced answers. ## How to test - Resolve conversations where the answer came only from the bot; no pending FAQ should be generated. - Resolve spam, unrelated, wrong-service, or private payment/order/account conversations; no pending FAQ should be generated. - Resolve conversations that require account/order/payment/login/private verification or a human handoff; no pending FAQ should be generated. - Resolve a conversation where a human agent gives a stable, reusable help-center answer; the generated pending FAQ should be general and self-contained. |
||
|
Aakash BakhleandGitHub
|
83dda621c5 |
feat: default new accounts to captain v2 (#14917)
# Pull Request Template ## Description defaults new accounts to captain v2 ## Type of change - [x] New feature (non-breaking change which adds functionality) ## How Has This Been Tested? Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration. locally and specs ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] Any dependent changes have been merged and published in downstream modules |
||
|
Sony MathewandGitHub
|
b8b62ad0f1 |
feat: Harden model override preferences (5/6) (#14846)
## Description Hardens the Captain model override preferences API so account-level overrides follow the same feature-router contract used by runtime LLM calls. The API now permits model and feature keys from `llm.yml`, removes blank model overrides, rejects invalid saved model combinations, and returns each feature's effective model, provider, and source for UI clients. Fixes https://linear.app/chatwoot/issue/CW-7425/test-new-models ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? Verified the account preferences API and account model validation behavior for valid overrides, invalid model values, unknown feature keys, blank override removal, and effective model/provider/source payload metadata. - `eval "$(rbenv init -)" && bundle exec rspec spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/models/account_spec.rb spec/models/concerns/captain_featurable_spec.rb spec/lib/llm/feature_router_spec.rb` - `eval "$(rbenv init -)" && bundle exec rubocop app/controllers/api/v1/accounts/captain/preferences_controller.rb app/models/concerns/account_settings_schema.rb app/models/concerns/captain_featurable.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/models/account_spec.rb spec/models/concerns/captain_featurable_spec.rb spec/lib/llm/feature_router_spec.rb` - `git diff --check` ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Sony MathewandGitHub
|
43e0f8a141 |
feat: Route Enterprise LLM services (3/6) (#14841)
# Pull Request Template ## Description Routes Enterprise assistant, copilot, FAQ, contact memory, action-classifier, and false-promise detector LLM paths through feature-specific model resolution. `Llm::BaseAiService` now accepts feature/account context and uses `Llm::FeatureRouter` when that context is present, while retaining the installation-model fallback for unmigrated callers. This also adds a `document_faq_generation` feature default for generative FAQ/document content. Linear: https://linear.app/chatwoot/issue/CW-7425/test-new-models Depends on #14840 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? - `bundle exec rspec spec/lib/llm/models_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/captain/copilot/chat_service_spec.rb spec/enterprise/services/captain/llm/assistant_chat_service_spec.rb spec/enterprise/services/captain/llm/faq_generator_service_spec.rb spec/enterprise/services/captain/llm/conversation_faq_service_spec.rb spec/enterprise/services/captain/llm/assistant_action_classifier_service_spec.rb spec/enterprise/services/captain/llm/assistant_false_promise_service_spec.rb spec/enterprise/jobs/captain/conversation/response_builder_job_spec.rb` passed with 112 examples, 0 failures. - `bundle exec rspec spec/models/concerns/captain_featurable_spec.rb spec/models/account_spec.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/lib/llm/feature_router_spec.rb` passed with 87 examples, 0 failures. - `bundle exec rubocop enterprise/app/services/llm/base_ai_service.rb enterprise/app/services/captain/copilot/chat_service.rb enterprise/app/services/captain/llm/assistant_chat_service.rb enterprise/app/services/captain/llm/faq_generator_service.rb enterprise/app/services/captain/llm/conversation_faq_service.rb enterprise/app/services/captain/llm/contact_notes_service.rb enterprise/app/services/captain/llm/contact_attributes_service.rb enterprise/app/services/captain/llm/assistant_action_classifier_service.rb enterprise/app/services/captain/llm/assistant_false_promise_service.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/captain/copilot/chat_service_spec.rb spec/enterprise/services/captain/llm/assistant_chat_service_spec.rb spec/enterprise/services/captain/llm/faq_generator_service_spec.rb spec/enterprise/services/captain/llm/conversation_faq_service_spec.rb spec/enterprise/services/captain/llm/assistant_action_classifier_service_spec.rb spec/enterprise/services/captain/llm/assistant_false_promise_service_spec.rb spec/enterprise/jobs/captain/conversation/response_builder_job_spec.rb` passed with no offenses. - `bundle exec ruby -e "require 'yaml'; config = YAML.load_file('config/llm.yml'); abort('missing document_faq_generation') unless config.dig('features', 'document_faq_generation'); abort('missing default') unless config.dig('features', 'document_faq_generation', 'default'); puts 'llm.yml ok'"` passed. - `git diff --check` passed. ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
|
34b42a1ce1 |
feat: add global config for captain settings (#13141)
Co-authored-by: aakashb95 <aakashbakhle@gmail.com> Co-authored-by: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com> |