develop
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aakash BakhleandGitHub
|
d57354c8b5 |
feat: tighten conversation FAQ generation prompt (#14957)
Tightens the resolved-conversation FAQ generator so it only proposes durable, reusable FAQ candidates supported by human support-agent messages. The implementation now sends a conversation-FAQ-specific transcript to the LLM: customer messages plus real human support-agent messages only, excluding bot, private, activity, and template messages. ## Closes - https://linear.app/chatwoot/issue/CW-7494/tighten-conversation-faq-generation-prompt ## What changed - Added a human-only transcript builder in `ConversationFaqService` instead of using the generic `conversation.to_llm_text` output. - Excluded bot/agent-bot messages before the LLM call, which removes the main bot-line leakage class deterministically. - Preserved native-channel human replies where outgoing messages are stored as `external_echo` without a `User` sender. - Kept a prompt decision gate requiring each FAQ to be backed by a complete public human-agent answer. - Added generic no-FAQ classes for spam, wrong-service conversations, private account/payment/order/certificate/troubleshooting cases, support workflow mechanics, and direct-link/file/quote outputs. - Added a separate `conversation_faq_generation` model route defaulting to `gpt-5.2`, while keeping `document_faq_generation` on its existing `gpt-4.1-mini` default. Conversation FAQ generation passes that feature default ahead of the legacy global `CAPTAIN_OPEN_AI_MODEL` setting unless an account-level override is configured. - Kept the prompt domain-neutral so it can still generate reusable product, service, policy, setup, and process FAQs outside SaaS contexts. ## Sampling notes - Production Langfuse traces showed `llm.captain.conversation_faq` calls using `gpt-4.1` in the sampled account set. - Locally, `Llm::FeatureRouter.resolve(feature: 'conversation_faq_generation')` now resolves to `gpt-5.2`. - Reviewed recent production `llm.captain.conversation_faq` traces across 13+ accounts in compact form. - Replayed 20 full traces across 10 accounts/domains, including education, hosting, retail/auto, APIs, logistics, tax/fiscal workflows, and Chatwoot account 1. - Explicit `gpt-5.2` replay with human-only conversation history returned no FAQ for 15/20 traces. - A comparison replay with `gpt-4.1-mini` returned no FAQ for only 7/20 traces, bringing back several private/order/payment/support-workflow cases. - Remaining non-empty `gpt-5.2` outputs are now mostly borderline/possibly useful human-agent-derived FAQs rather than obvious bot-sourced answers. ## How to test - Resolve conversations where the answer came only from the bot; no pending FAQ should be generated. - Resolve spam, unrelated, wrong-service, or private payment/order/account conversations; no pending FAQ should be generated. - Resolve conversations that require account/order/payment/login/private verification or a human handoff; no pending FAQ should be generated. - Resolve a conversation where a human agent gives a stable, reusable help-center answer; the generated pending FAQ should be general and self-contained. |
||
|
Sony MathewandGitHub
|
cf134deb37 |
fix: Preserve Captain LLM defaults (#14858)
# Pull Request Template ## Description Adjusts the Captain LLM feature defaults after feature routing so the defaults stay intentional and avoid unintended high-cost model upgrades. Assistant, copilot, and onboarding content generation now default to `gpt-4.1`; audio transcription keeps `gpt-4o-mini-transcribe` as the default while exposing `whisper-1` as an available account override option. Related: https://linear.app/chatwoot/issue/CW-7425/test-new-models ## Type of change - [x] Bug fix (non-breaking change which fixes an issue) ## How Has This Been Tested? - `ruby -ryaml -e 'yaml = YAML.load_file("config/llm.yml"); features = yaml.fetch("features"); models = yaml.fetch("models"); features.each { |name, cfg| missing = Array(cfg["models"]) - models.keys; raise "#{name}: missing #{missing.join(",")}" if missing.any?; raise "#{name}: default not in models" unless Array(cfg["models"]).include?(cfg["default"]) }'` - `node -e "JSON.parse(require('fs').readFileSync('app/javascript/dashboard/i18n/locale/en/settings.json', 'utf8'))"` - `pnpm exec prettier --check app/javascript/dashboard/i18n/locale/en/settings.json` - `bundle exec rspec spec/lib/llm/feature_router_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/models/concerns/agentable_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/models/concerns/captain_featurable_spec.rb spec/models/account_spec.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/controllers/super_admin/accounts_controller_spec.rb` - `bundle exec rspec spec/enterprise/services/captain/llm/article_translation_service_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb` - `RUBOCOP_CACHE_ROOT=/private/tmp/rubocop_cache bundle exec rubocop enterprise/app/services/captain/llm/article_translation_service.rb enterprise/app/services/messages/audio_transcription_service.rb spec/enterprise/services/captain/llm/article_translation_service_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb` - `git diff --check` ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Sony MathewandGitHub
|
4e26c5b4bb |
feat: Route system LLM jobs (4/6) (#14843)
## Description Routes the remaining system-only and legacy-sensitive LLM jobs through feature-level model configuration, while preserving system credential usage and usage-accounting behavior. This adds dedicated defaults for help center article generation, onboarding content generation, query translation, transcription, and search embeddings so these flows can be configured per account without falling back to installation-wide model settings. Fixes https://linear.app/chatwoot/issue/CW-7425/test-new-models ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? Verified the feature routing defaults and account overrides for the touched Captain/system LLM paths, including the legacy OpenAI transcription and paginated FAQ services. - `eval "$(rbenv init -)" && bundle exec rspec spec/lib/captain/base_task_service_spec.rb spec/lib/llm/models_spec.rb spec/models/concerns/captain_featurable_spec.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/enterprise/services/captain/llm/paginated_faq_generator_service_spec.rb spec/enterprise/services/captain/llm/pdf_processing_service_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/enterprise/services/onboarding/help_center_article_builder_spec.rb spec/enterprise/services/captain/onboarding/website_analyzer_service_spec.rb` - `eval "$(rbenv init -)" && bundle exec rubocop app/controllers/api/v1/accounts/captain/preferences_controller.rb app/models/concerns/account_settings_schema.rb lib/captain/base_task_service.rb enterprise/app/services/captain/llm/article_translation_service.rb enterprise/app/services/captain/llm/article_writer_service.rb enterprise/app/services/captain/llm/embedding_service.rb enterprise/app/services/captain/llm/help_center_curation_service.rb enterprise/app/services/captain/llm/paginated_faq_generator_service.rb enterprise/app/services/captain/llm/translate_query_service.rb enterprise/app/services/captain/llm/widget_tagline_service.rb enterprise/app/services/captain/onboarding/website_analyzer_service.rb enterprise/app/services/messages/audio_transcription_service.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/lib/captain/base_task_service_spec.rb` - `ruby -e "require 'yaml'; config = YAML.load_file('config/llm.yml'); %w[document_faq_generation help_center_article_generation onboarding_content_generation help_center_query_translation audio_transcription help_center_search].each { |feature| abort(%(missing #{feature})) unless config.dig('features', feature) }; abort('wrong article default') unless config.dig('features', 'help_center_article_generation', 'default') == 'gpt-5.2'; puts 'llm.yml ok'"` - `git diff --check` ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Sony MathewandGitHub
|
43e0f8a141 |
feat: Route Enterprise LLM services (3/6) (#14841)
# Pull Request Template ## Description Routes Enterprise assistant, copilot, FAQ, contact memory, action-classifier, and false-promise detector LLM paths through feature-specific model resolution. `Llm::BaseAiService` now accepts feature/account context and uses `Llm::FeatureRouter` when that context is present, while retaining the installation-model fallback for unmigrated callers. This also adds a `document_faq_generation` feature default for generative FAQ/document content. Linear: https://linear.app/chatwoot/issue/CW-7425/test-new-models Depends on #14840 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? - `bundle exec rspec spec/lib/llm/models_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/captain/copilot/chat_service_spec.rb spec/enterprise/services/captain/llm/assistant_chat_service_spec.rb spec/enterprise/services/captain/llm/faq_generator_service_spec.rb spec/enterprise/services/captain/llm/conversation_faq_service_spec.rb spec/enterprise/services/captain/llm/assistant_action_classifier_service_spec.rb spec/enterprise/services/captain/llm/assistant_false_promise_service_spec.rb spec/enterprise/jobs/captain/conversation/response_builder_job_spec.rb` passed with 112 examples, 0 failures. - `bundle exec rspec spec/models/concerns/captain_featurable_spec.rb spec/models/account_spec.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/lib/llm/feature_router_spec.rb` passed with 87 examples, 0 failures. - `bundle exec rubocop enterprise/app/services/llm/base_ai_service.rb enterprise/app/services/captain/copilot/chat_service.rb enterprise/app/services/captain/llm/assistant_chat_service.rb enterprise/app/services/captain/llm/faq_generator_service.rb enterprise/app/services/captain/llm/conversation_faq_service.rb enterprise/app/services/captain/llm/contact_notes_service.rb enterprise/app/services/captain/llm/contact_attributes_service.rb enterprise/app/services/captain/llm/assistant_action_classifier_service.rb enterprise/app/services/captain/llm/assistant_false_promise_service.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/captain/copilot/chat_service_spec.rb spec/enterprise/services/captain/llm/assistant_chat_service_spec.rb spec/enterprise/services/captain/llm/faq_generator_service_spec.rb spec/enterprise/services/captain/llm/conversation_faq_service_spec.rb spec/enterprise/services/captain/llm/assistant_action_classifier_service_spec.rb spec/enterprise/services/captain/llm/assistant_false_promise_service_spec.rb spec/enterprise/jobs/captain/conversation/response_builder_job_spec.rb` passed with no offenses. - `bundle exec ruby -e "require 'yaml'; config = YAML.load_file('config/llm.yml'); abort('missing document_faq_generation') unless config.dig('features', 'document_faq_generation'); abort('missing default') unless config.dig('features', 'document_faq_generation', 'default'); puts 'llm.yml ok'"` passed. - `git diff --check` passed. ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Sony MathewandGitHub
|
8b977b35a8 |
feat: Add LLM feature router (1/6) (#14839)
## Description Adds the foundation for feature-specific LLM model routing so Captain AI features can resolve their effective provider/model from code defaults and account-level overrides. This fixes the provider metadata key in `config/llm.yml`, adds `Llm::FeatureRouter`, and routes existing `CaptainFeaturable` model defaults through the shared resolver. Fixes https://linear.app/chatwoot/issue/CW-7425/test-new-models ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? - `bundle exec rspec spec/lib/llm/models_spec.rb spec/lib/llm/feature_router_spec.rb spec/models/concerns/captain_featurable_spec.rb` - 23 examples, 0 failures - `bundle exec rubocop lib/llm/models.rb lib/llm/feature_router.rb app/models/concerns/captain_featurable.rb spec/lib/llm/models_spec.rb spec/lib/llm/feature_router_spec.rb spec/models/concerns/captain_featurable_spec.rb` - no offenses - `bundle exec ruby -e "require 'yaml'; config = YAML.load_file('config/llm.yml'); abort('missing providers') unless config['providers']; abort('missing models') unless config['models']; abort('missing features') unless config['features']; puts 'llm.yml ok'"` - `git diff --check` ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [ ] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
|
34b42a1ce1 |
feat: add global config for captain settings (#13141)
Co-authored-by: aakashb95 <aakashbakhle@gmail.com> Co-authored-by: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com> |