# Pull Request Template
## Description
Large emails (2MB+ with multiple attachments) were causing IMAP email
processing jobs to timeout silently, blocking all subsequent emails from
being processed. This created an infinite loop where:
- Problematic emails were repeatedly fetched but never successfully
processed
- Other emails in the queue were never processed as we iterated
sequentially
- silent failures
### Solution
Enhanced the FetchImapEmailsJob with individual email processing
isolation:
### Key Changes
1. Individual Email Processing: Changed from map to each for better
memory efficiency
2. Timeout Protection: Added configurable timeout per email (default: 60
seconds)
3. Failure Tracking: Track failed emails with 6-hour expiry for retry
opportunities
4. Skip Logic: Skip emails that have failed 3+ times to prevent infinite
loops
5. Error Isolation: Each email is processed in its own error boundary
### Configuration
- Timeout: Configurable via EMAIL_PROCESSING_TIMEOUT_SECONDS using
GlobalConfigService
- Default: 60 seconds per email
- Failure Limit: 3 attempts before skipping
- Retry Window: 6 hours so that emails get 8 more chances in the 2 day
window
### Benefits
- Prevents queue blocking: One problematic email cannot stop others
- Maintains email order: Older emails (customers waiting longer)
processed first
- Automatic recovery: Failed emails get retry opportunities
- Better monitoring: Clear logging when emails timeout or are skipped
- Configurable: Deployments can adjust the timeout based on their needs
This fix ensures email processing reliability while maintaining existing
functionality.
## Type of change
Please delete options that are not relevant.
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality not to work as expected)
- [ ] This change requires a documentation update
## How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.
## Checklist:
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
## Description
The IMAP email fetch job (Inboxes::FetchImapEmailsJob) crashes with an
unhandled IOError: closed stream when the mail server's SSL socket is
closed mid-write during Net::IMAP#fetch. This error was being reported
to Sentry because the rescue clause only caught EOFError, not its parent
class IOError.
Fixes
[CW-6689](https://linear.app/chatwoot/issue/CW-6689/ioerror-closed-stream-ioerror)
Widened the rescue in fetch_imap_emails_job.rb from EOFError to IOError.
In Ruby's exception hierarchy, EOFError is a subclass of IOError:
```
StandardError
└── IOError
└── EOFError
```
The Sentry stacktrace shows a plain IOError: closed stream raised from
OpenSSL::Buffering#do_write → Net::IMAP#put_string → Net::IMAP#fetch.
Since this is an IOError (not EOFError), it bypassed the existing rescue
and fell through to the StandardError catch-all, which reported it to
Sentry as an unhandled exception.
Rescuing IOError now catches both:
IOError: closed stream — the reported crash (parent class)
EOFError — the previously handled case (still caught as a subclass)
## Type of change
- [ ] Bug fix (non-breaking change which fixes an issue)
## Checklist:
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Notion document
https://www.notion.so/chatwoot/Email-IMAP-Issue-30aa5f274c928062aa6bddc2e5877a63?showMoveTo=true&saveParent=true
## Description
PLAIN IMAP channels (non-OAuth) were silently retrying failed
authentication every minute, forever. When credentials are
wrong/expired, Net::IMAP::NoResponseError was caught and logged but
channel.authorization_error! was never called — so the Redis error
counter never incremented, reauthorization_required? was never set, and
admins were never notified. OAuth channels already had this handled
correctly via the Reauthorizable concern.
Additionally, Net::IMAP::ResponseParseError (raised by non-RFC-compliant
IMAP servers) was falling through to the StandardError catch-all,
flooding
Estimated impact before fix: ~70–75 broken IMAP inboxes generating
~700k–750k wasted Sidekiq jobs/week.
## Type of change
- [ ] Bug fix (non-breaking change which fixes an issue)
## Checklist:
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
When an IMAP server returns malformed or partial protocol responses,
Inboxes::FetchImapEmailsJob can raise Net::IMAP::ResponseParseError,
Net::IMAP::ResponseReadError, or Net::IMAP::ResponseTooLargeError.
Previously these errors fell through to the generic StandardError
handler and were captured repeatedly in Sentry.
This PR updates app/jobs/inboxes/fetch_imap_emails_job.rb to treat those
parser/read exceptions as handled IMAP failures by adding them to the
existing IMAP/network rescue block, so they are logged and retried on
the next scheduled run without exception tracking noise.
fixes:
https://chatwoot-p3.sentry.io/issues/7132037793/events/104fb9b4d80a4fb6ba3861c44c6c9b83/
How to reproduce:
- Use an IMAP server/inbox that intermittently returns malformed or
truncated protocol responses during search/fetch.
- Run Inboxes::FetchImapEmailsJob for that channel and observe the
raised parser/read exception.
How this was tested:
- bundle exec ruby -c app/jobs/inboxes/fetch_imap_emails_job.rb
- bundle exec rubocop app/jobs/inboxes/fetch_imap_emails_job.rb
Currently, auto-assignment runs only during conversation creation or
update events. If no agents are online when new conversations arrive,
those conversations remain unassigned.
With this change, unassigned conversations will be automatically
assigned once agents become available. The job runs every 15 minutes and
uses a fair distribution threshold of 100 to prevent a large number of
conversations from being assigned to a single available agent. This will
be customizable later.
Fixes the follow cases
- The ensure block released the lock even on LockAcquisitionError
- Custom timeout was not allowed
This also refactored the with_lock method, now the key has to be constructed in the parent function itself
Co-authored-by: Sojan Jose <sojan@pepalo.com>
This allows a user to add/update a custom regex and a cue while defining custom attributes(Only applicable for type- text).
While adding/editing custom attributes, the values are validated against the attribute definition regex, and if it is incorrect, a cue message or default error message is shown and restricts invalid values from being saved.
Fixes: #6866
The Inboxes::FetchImapEmailsJob is designed to fetch the entire email object and check if its message id is already in the database. However, this process is resource-intensive and time-consuming, as fetching the full email object takes a significant amount of time.
On average, fetching 100 emails can take approximately 3-4 minutes to complete depending on the IMAP server. Since we are not using server flags to identify which emails have already been fetched (to avoid compatibility issues with flags in different email services), we have to fetch all previously available emails. Currently we fetch all the messages that were created from yesterday. This becomes problematic with services like Gmail, which throttle requests based on bandwidth usage.
To address this issue, I have updated the logic as follows:
Fetch the sequence IDs of all the mails in the "Inbox" that were created from yesterday.
Use the FETCH command to fetch only the message-ids using BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)] with the sequence IDs we got in the previous step. This is a faster operation with lower bandwidth usage, as it only returns the sequence ID and message ID.
Check if the message IDs are already present in the database for the selected inbox.
If not present, fetch the entire email object using the FETCH command and create the message.
If the message ID is already present, ignore it and move on to the next message-id.
I have also addressed the issue of duplicate emails appearing in conversations when two fetch email jobs occur simultaneously. I have added a lock for the channel to ensure that the job gracefully exits without waiting for the current job to complete.
Fixes#7247Fixes#6082Fixes#8314
Co-authored-by: Sojan <sojan@pepalo.com>
* fix: mail logger for source-id email-id and message-id
* added mail provided to the tracking
* change the logger verb from info to error
* app/jobs/inboxes/fetch_imap_emails_job.rb
* codeclimate fix
* remove extra filter
* fixes: specs
- Adds the backend APIs required for Microsoft Email Channels
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
Co-authored-by: Sojan <sojan@pepalo.com>
This change allows the user to configure both IMAP and SMTP for an email inbox. IMAP enables the user to see emails in Chatwoot. And user can use SMTP to reply to an email conversation.
Users can use the default settings to send and receive emails for email inboxes if both IMAP and SMTP are disabled.
Fixes#2520
