# Pull Request Template
## Description
Better scheduling and queueing mechanics for document auto-sync
- add jitter plan wise for document sync
- move auto-sync documents to purgeable queue
## Type of change
Please delete options that are not relevant.
- [x] New feature (non-breaking change which adds functionality)
## How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.
locally tested and with specs
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream
modules
---------
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
Co-authored-by: iamsivin <iamsivin@gmail.com>
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
Co-authored-by: Sony Mathew <sony@chatwoot.com>
Co-authored-by: Vishnu Narayanan <iamwishnu@gmail.com>
# Pull Request Template
## Description
skip documents that fail with ActiveRecord errors possibly due to
stale/corrupt data and not crash scheduler
How did we find out about this error?
before October 28th, 2025, we did not have url normalisation.
so we had document rows as:
id: 123 `https://example.com` status: `in_progress` --> likely stuck
crawl
id 234: `https://example.com/` status: `available`
When the schedule sync job ran, it ran an `document.update!(sync_status:
:syncing, last_sync_attempted_at: Time.current)` on the 234 one since it
was `available`
now `update!` runs `before_validation :normalize_external_link`
so `https://example.com/` became `https://example.com`
which invalidated:
`validates :external_link, uniqueness: { scope: :assistant_id }`
so the scheduler crashed.
This PR logs the skipped ones with their errors and continues to pick
other documents to scheduler doesn't crash
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
## How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.
spec
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream
modules
# Pull Request Template
## Description
- Wires up Controllers to auto-sync job
- adds plan based sync schedule
- a scheduler that runs every hour to check syncable documents
- guards the whole feature behind feature flag by reclaiming
`twilio_content_templates`
- Adds a global and account level cap on how many documents to enqueue
to prevent sudden burst at first run
- some refactor to simplify code
- specs
Fixes # (issue)
## Type of change
- [x] New feature (non-breaking change which adds functionality)
## How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.
specs and locally
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream
modules
# Pull Request Template
## Description
- Fetch main content only from Firecrawl, exclude some tags to remove
boilerplate
- Prompt changes for FAQ generation
## Type of change
Please delete options that are not relevant.
- [x] Bug fix (non-breaking change which fixes an issue)
## How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.
tested locally
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream
modules
There were customer reported issues with FAQs which were generated in a
different langauge than what they were expecting. The reason behind this
was that the language of the account was not considered in the prompt
provided. If the language of the content was say Spanish, and the
account locale was english. The output was not predicable. The output
depends on the model and the execution time.
This PR would update the prompt to behave consistently with the account
locale. Even though the content provided is in a different language, it
would generate FAQs in the account locale.
Changes:
- Updated the prompt to include a detailed expectation of the FAQs
quality along with the language
- Added specs for the services where the prompt generator is called.
Tested the prompt using Phoenix playground across GPT 5, GPT 4.1, GPT
4.0. The reasoning setting for GPT 5 needs to be low so that it doesn't
generate random questions like "What was this updated?"
- Fixed Firecrawl webhook payloads to ensure proper data handling and
delivery.
- Removed unused Robin AI code to improve codebase cleanliness and
maintainability.
- Implement authentication for the Firecrawl endpoint to improve
security. A key is generated to secure the webhook URLs from FireCrawl.
---------
Co-authored-by: Pranav <pranavrajs@gmail.com>
Currently, it’s unclear whether an FAQ item is generated from a
document, derived from a conversation, or added manually.
This PR resolves the issue by providing visibility into the source of
each FAQ. Users can now see whether an FAQ was generated or manually
added and, if applicable, by whom.
- Move the document_id to a polymorphic relation (documentable).
- Updated the APIs to accommodate the change.
- Update the service to add corresponding references.
- Updated the specs.
<img width="1007" alt="Screenshot 2025-01-15 at 11 27 56 PM"
src="https://github.com/user-attachments/assets/7d58f798-19c0-4407-b3e2-748a919d14af"
/>
---------
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
Migration Guide: https://chwt.app/v4/migration
This PR imports all the work related to Captain into the EE codebase. Captain represents the AI-based features in Chatwoot and includes the following key components:
- Assistant: An assistant has a persona, the product it would be trained on. At the moment, the data at which it is trained is from websites. Future integrations on Notion documents, PDF etc. This PR enables connecting an assistant to an inbox. The assistant would run the conversation every time before transferring it to an agent.
- Copilot for Agents: When an agent is supporting a customer, we will be able to offer additional help to lookup some data or fetch information from integrations etc via copilot.
- Conversation FAQ generator: When a conversation is resolved, the Captain integration would identify questions which were not in the knowledge base.
- CRM memory: Learns from the conversations and identifies important information about the contact.
---------
Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
Co-authored-by: Sojan <sojan@pepalo.com>
Co-authored-by: iamsivin <iamsivin@gmail.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
