Commit Graph
29 Commits
Author SHA1 Message Date
Vishnu NarayananandGitHub 52da165cb7 feat: add timeout for imap email job and skip problematic emails (#11981)
# Pull Request Template

## Description

Large emails (2MB+ with multiple attachments) were causing IMAP email
processing jobs to timeout silently, blocking all subsequent emails from
being processed. This created an infinite loop where:
- Problematic emails were repeatedly fetched but never successfully
processed
- Other emails in the queue were never processed as we iterated
sequentially
  - silent failures


  ### Solution

Enhanced the FetchImapEmailsJob with individual email processing
isolation:

  ### Key Changes

1. Individual Email Processing: Changed from map to each for better
memory efficiency
2. Timeout Protection: Added configurable timeout per email (default: 60
seconds)
3. Failure Tracking: Track failed emails with 6-hour expiry for retry
opportunities
4. Skip Logic: Skip emails that have failed 3+ times to prevent infinite
loops
  5. Error Isolation: Each email is processed in its own error boundary

  ### Configuration

- Timeout: Configurable via EMAIL_PROCESSING_TIMEOUT_SECONDS using
GlobalConfigService
  - Default: 60 seconds per email
  - Failure Limit: 3 attempts before skipping
- Retry Window: 6 hours so that emails get 8 more chances in the 2 day
window

  ### Benefits

  - Prevents queue blocking: One problematic email cannot stop others
- Maintains email order: Older emails (customers waiting longer)
processed first
  - Automatic recovery: Failed emails get retry opportunities
  - Better monitoring: Clear logging when emails timeout or are skipped
- Configurable: Deployments can adjust the timeout based on their needs

This fix ensures email processing reliability while maintaining existing
functionality.

## Type of change

Please delete options that are not relevant.

- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality not to work as expected)
- [ ] This change requires a documentation update

## How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.


## Checklist:

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
2026-05-25 15:16:52 +05:30
Tanmay Deep SharmaandGitHub 1987ac3d97 fix: remove bulk_auto_assignment_job cron schedule (#13877) 2026-03-31 10:56:59 +05:30
Sojan JoseandGitHub d84ae196d5 fix: call authorization_error! on IMAP auth failures (#13560) (revert) (#13671)
This reverts commit 7acd239c70 to further
debug upstream issues.
2026-02-26 18:45:18 -08:00
Tanmay Deep SharmaandGitHub 7acd239c70 fix: call authorization_error! on IMAP auth failures (#13560)
## Notion document

https://www.notion.so/chatwoot/Email-IMAP-Issue-30aa5f274c928062aa6bddc2e5877a63?showMoveTo=true&saveParent=true

## Description

PLAIN IMAP channels (non-OAuth) were silently retrying failed
authentication every minute, forever. When credentials are
wrong/expired, Net::IMAP::NoResponseError was caught and logged but
channel.authorization_error! was never called — so the Redis error
counter never incremented, reauthorization_required? was never set, and
admins were never notified. OAuth channels already had this handled
correctly via the Reauthorizable concern.
Additionally, Net::IMAP::ResponseParseError (raised by non-RFC-compliant
IMAP servers) was falling through to the StandardError catch-all,
flooding
Estimated impact before fix: ~70–75 broken IMAP inboxes generating
~700k–750k wasted Sidekiq jobs/week.

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)

## Checklist:

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
2026-02-26 18:01:23 +05:30
PranavandGitHub f4643116df feat: Run assignment every 15 minutes (#12334)
Currently, auto-assignment runs only during conversation creation or
update events. If no agents are online when new conversations arrive,
those conversations remain unassigned.

With this change, unassigned conversations will be automatically
assigned once agents become available. The job runs every 15 minutes and
uses a fair distribution threshold of 100 to prevent a large number of
conversations from being assigned to a single available agent. This will
be customizable later.
2025-08-29 15:10:56 -07:00
PranavandGitHub 51b9fd8eca fix: Disable IMAP inboxes that requires authorization (#12092)
This PR disables queueing IMAP sync jobs for emails channels that 
- are in free plan if on Chatwoot cloud.
- requires authorization
2025-08-01 16:32:29 -07:00
PranavandGitHub d017156f32 fix: Disable syncing IMAP if the account is suspended (#11031)
This PR disables the IMAP syncing if the account is suspended.
2025-03-05 17:10:24 -08:00
8d8ec23aa1 chore: [Snyk] Security upgrade rspec-rails from 6.1.4 to 6.1.5 (#10066)
![snyk-top-banner](https://github.com/andygongea/OWASP-Benchmark/assets/818805/c518c423-16fe-447e-b67f-ad5a49b5d123)

### Snyk has created this PR to fix 1 vulnerabilities in the rubygems
dependencies of this project.

#### Snyk changed the following file(s):

- `Gemfile`



<details>
<summary>⚠️ <b>Warning</b></summary>

```
Failed to update the Gemfile.lock, please update manually before merging.
```

</details>



#### Vulnerabilities that will be fixed with an upgrade:

|  | Issue | Score | 

:-------------------------:|:-------------------------|:-------------------------
![medium
severity](https://res.cloudinary.com/snyk/image/upload/w_20,h_20/v1561977819/icon/m.png
'medium severity') | Web Cache Poisoning
<br/>[SNYK-RUBY-RACK-1061917](https://snyk.io/vuln/SNYK-RUBY-RACK-1061917)
| &nbsp;&nbsp;**616**&nbsp;&nbsp;




---

> [!IMPORTANT]
>
> - Check the changes in this PR to ensure they won't cause issues with
your project.
> - Max score is 1000. Note that the real score may have changed since
the PR was raised.
> - This PR was automatically created by Snyk using the credentials of a
real user.

---

**Note:** _You are seeing this because you or someone else with access
to this repository has authorized Snyk to open fix PRs._

For more information: <img
src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiJhMWE2MzkzZS03ODdhLTRmYWItOGY1MS0zZjdmN2YzNzVlZDYiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6ImExYTYzOTNlLTc4N2EtNGZhYi04ZjUxLTNmN2Y3ZjM3NWVkNiJ9fQ=="
width="0" height="0"/>
🧐 [View latest project
report](https://app.snyk.io/org/chatwoot/project/b7197bbd-6200-4f23-931d-c39928584360?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;fix-pr)
📜 [Customise PR
templates](https://docs.snyk.io/scan-using-snyk/pull-requests/snyk-fix-pull-or-merge-requests/customize-pr-templates)
🛠 [Adjust project
settings](https://app.snyk.io/org/chatwoot/project/b7197bbd-6200-4f23-931d-c39928584360?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;fix-pr/settings)
📚 [Read about Snyk's upgrade
logic](https://support.snyk.io/hc/en-us/articles/360003891078-Snyk-patches-to-fix-vulnerabilities)

---

**Learn how to fix vulnerabilities with free interactive lessons:**

🦉 [Learn about vulnerability in an interactive lesson of Snyk
Learn.](https://learn.snyk.io/?loc&#x3D;fix-pr)

[//]: #
'snyk:metadata:{"customTemplate":{"variablesUsed":[],"fieldsUsed":[]},"dependencies":[{"name":"rspec-rails","from":"6.1.4","to":"6.1.5"}],"env":"prod","issuesToFix":[{"exploit_maturity":"Proof
of
Concept","id":"SNYK-RUBY-RACK-1061917","priority_score":616,"priority_score_factors":[{"type":"exploit","label":"Proof
of
Concept","score":107},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"5.9","score":295},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Web
Cache Poisoning"},{"exploit_maturity":"Proof of
Concept","id":"SNYK-RUBY-RACK-1061917","priority_score":616,"priority_score_factors":[{"type":"exploit","label":"Proof
of
Concept","score":107},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"5.9","score":295},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Web
Cache Poisoning"},{"exploit_maturity":"Proof of
Concept","id":"SNYK-RUBY-RACK-1061917","priority_score":616,"priority_score_factors":[{"type":"exploit","label":"Proof
of
Concept","score":107},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"5.9","score":295},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Web
Cache Poisoning"},{"exploit_maturity":"Proof of
Concept","id":"SNYK-RUBY-RACK-1061917","priority_score":616,"priority_score_factors":[{"type":"exploit","label":"Proof
of
Concept","score":107},{"type":"fixability","label":true,"score":214},{"type":"cvssScore","label":"5.9","score":295},{"type":"scoreVersion","label":"v1","score":1}],"severity":"medium","title":"Web
Cache
Poisoning"}],"prId":"a1a6393e-787a-4fab-8f51-3f7f7f375ed6","prPublicId":"a1a6393e-787a-4fab-8f51-3f7f7f375ed6","packageManager":"rubygems","priorityScoreList":[616],"projectPublicId":"b7197bbd-6200-4f23-931d-c39928584360","projectUrl":"https://app.snyk.io/org/chatwoot/project/b7197bbd-6200-4f23-931d-c39928584360?utm_source=github&utm_medium=referral&page=fix-pr","prType":"fix","templateFieldSources":{"branchName":"default","commitMessage":"default","description":"default","title":"default"},"templateVariants":["updated-fix-title","pr-warning-shown","priorityScore"],"type":"auto","upgrade":["SNYK-RUBY-RACK-1061917"],"vulns":["SNYK-RUBY-RACK-1061917"],"patch":[],"isBreakingChange":false,"remediationStrategy":"vuln"}'

---------

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
2024-09-16 09:44:07 +05:30
de044e29f0 chore: Rescue Oauth2::Error and mark the channel as inactive (#9358)
Co-authored-by: Sojan <sojan@pepalo.com>
2024-05-07 12:03:38 -07:00
PranavandGitHub 2c3e7d6554 chore: Temporarily disable the disconnect logic for inboxes (#9321)
* chore: Temporarily disable the disconnect logic for inboxes

* Remove unused spec
2024-04-30 07:49:37 -07:00
PranavandGitHub 77db0d0701 feat: Add configurable interval for IMAP sync (#9302) 2024-04-25 18:58:20 -07:00
Pranav Raj SandGitHub eb379e1849 fix(refactor): Cleanup the specs and the logic for FetchIMAP job (#8766) 2024-02-10 14:03:50 -08:00
33e98bf61a fix: Use BODY.PEEK[HEADER] to avoid parsing issues with mail providers (#8833)
Co-authored-by: Sojan <sojan@pepalo.com>
2024-02-01 17:24:59 -08:00
Pranav Raj SandGitHub ee3f734b7b chore: Refactor RefreshOauthTokenService to improve readability (#8820)
- Added a trait called microsoft_email for the Channel::Email factory.
- Rewrote the logic to make it simple to understand
- Rewrote the specs for readability
2024-01-31 12:24:12 +04:00
Pranav Raj SandGitHub 905ca94f71 chore: Remove unused attribute imap_inbox_synced_at (#8822)
- imap_inbox_synced_at is no longer used; this PR removes all the references to the same.
2024-01-31 11:48:46 +04:00
Surabhi SumanandGitHub 4b40c61201 feat: Support Regex validation for custom attributes (#7856)
This allows a user to add/update a custom regex and a cue while defining custom attributes(Only applicable for type- text).
While adding/editing custom attributes, the values are validated against the attribute definition regex, and if it is incorrect, a cue message or default error message is shown and restricts invalid values from being saved.

Fixes: #6866
2024-01-23 18:01:57 +04:00
Pranav Raj SandGitHub 022f4f899f Revert "feat: Support Azure single-tenant application using the Graph… (#7436) 2023-06-29 16:50:18 -07:00
Tejaswini ChileandGitHub d05c953eef feat: Support Azure single-tenant application using the Graph API (#6728) (#6878) 2023-06-28 08:13:08 +05:30
Sojan JoseandGitHub 7ab7bac6bf chore: Enable the new Rubocop rules (#7122)
fixes: https://linear.app/chatwoot/issue/CW-1574/renable-the-disabled-rubocop-rules
2023-05-19 14:37:10 +05:30
Tejaswini ChileandGitHub b994706265 fix: Email date attribute not being set (#7081)
Fixes: https://linear.app/chatwoot/issue/CW-1738/typeerror-no-implicit-conversion-of-nil-into-string-typeerror
2023-05-15 20:19:03 +05:30
Tejaswini ChileandGitHub 354010a6e1 chore: fetch mails with multiple attachments (#7030) 2023-05-14 10:02:36 +05:30
Sojan JoseandGitHub 85e57c2e94 chore: Reorganize Sidekiq Queues (#6976)
- Rearrange and reprioritize current sidekiq queues
- Trim the unnecessary queues

ref: https://linear.app/chatwoot/issue/CW-1480/chore-run-all-sidekiq-jobs-async
2023-05-04 15:44:16 +05:30
Sojan JoseandGitHub cf91e9eb58 chore: Use find_each instead of .all.each (#6975)
- Enable the rubocop Rails/FindEach
- Replace the .all.each with .find_each

This should let us avoid potential memory usage.
Motivation from the speedshop newsletter by Nate Berkopec

ref: https://www.rubyinrails.com/2017/11/16/use-find-each-instead-of-all-each-in-rails/
ref: https://linear.app/chatwoot/issue/CW-1480/chore-run-all-sidekiq-jobs-async
2023-04-25 09:32:35 +05:30
00cbdaa8ca Feat: Support for Microsoft Oauth in Email Channel (#6227)
- Adds the backend APIs required for Microsoft Email Channels

Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
Co-authored-by: Sojan <sojan@pepalo.com>
2023-01-17 02:39:05 +05:30
Aswin Dev P.SandGitHub 13a4e0e6d9 chore: Email improvements. (#4901)
* Update email processing logic.
* Fix sentry issues
2022-07-08 04:13:24 -07:00
Muhsin KelothandGitHub a8c6cd729b chore: Sync pre-chat fields after custom attribute update (#4692) 2022-06-20 14:16:49 +05:30
Muhsin KelothandGitHub e010f0c6f0 chore: Sync pre-chat fields after custom attribute destroy (#4456) 2022-04-18 14:06:27 +05:30
Aswin Dev P.SandGitHub 31cdc63e18 fix: Remove IMAP and SMTP email validation (#4435)
* Remove IMAP and SMTP email validation
* Rename imap_email & smtp_email columns to imap_login & smtp_login respectively.
* Use channel email domain if inbound email domain not present
2022-04-11 07:07:20 -07:00
Aswin Dev P.SandGitHub 24e6a92297 feat: IMAP Email Channel (#3298)
This change allows the user to configure both IMAP and SMTP for an email inbox. IMAP enables the user to see emails in Chatwoot. And user can use SMTP to reply to an email conversation.

Users can use the default settings to send and receive emails for email inboxes if both IMAP and SMTP are disabled.

Fixes #2520
2021-11-19 11:52:27 +05:30