## Description
Refactored the `Message#search_data` method to prevent exceeding
OpenSearch's 1000 field limit during reindex operations.
**Problem:** The previous implementation serialized entire ActiveRecord
objects (Inbox, Sender, Conversation) with all their JSONB fields,
causing dynamic field explosion in OpenSearch. This resulted in
`Searchkick::ImportError` with "Limit of total fields [1000] has been
exceeded".
**Solution:** Whitelisted only necessary fields for search and
filtering, and flattened JSONB `custom_attributes` into key-value pair
arrays to prevent unbounded field creation.
Linked to: CW-5861
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality not to work as expected)
- [x] This change requires a documentation update
## How Has This Been Tested?
- Verified rubocop passes with no offenses
- Code review of search field usage from
`enterprise/app/services/enterprise/search_service.rb`
- Analyzed actual search queries to determine required indexed fields
**Still needed:**
- Full reindex test on staging/production environment
- Verify search functionality still works after reindex
- Confirm field count is under 1000 limit
## Changes Made
### Before
- Indexed 1000+ fields (entire AR objects with JSONB)
- `inbox` = full Inbox object (23+ fields + JSONB)
- `sender` = full Contact/User/AgentBot object (10+ fields + JSONB)
- `conversation` = full push_event_data
- Dynamic JSONB keys creating unlimited fields
### After
- ~35-40 controlled fields
- Whitelisted search fields: `content`, `attachment_transcribed_text`,
`email_subject`
- Filter fields: `account_id`, `inbox_id`, `conversation_id`,
`sender_id`, `sender_type`, etc.
- Flattened `custom_attributes`: `[{key, value, value_type}]` format
- Helper methods: `search_conversation_data`, `search_inbox_data`,
`search_sender_data`, `search_additional_data`
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
## Post-merge Steps
After merging, the following steps are required:
1. **Reindex all messages:**
```bash
bundle exec rails runner "Message.reindex"
```
2. **Verify field count:**
```bash
bundle exec rails runner "
client = Searchkick.client
index_name = Message.searchkick_index.name
mapping = client.indices.get_mapping(index: index_name)
fields = mapping.dig(index_name, 'mappings', 'properties')
puts 'Total fields: ' + fields.keys.count.to_s
"
```
3. **Test search functionality** to ensure queries still work as
expected
---------
Co-authored-by: Vishnu Narayanan <iamwishnu@gmail.com>
Co-authored-by: Pranav <pranav@chatwoot.com>
## Description
Implements real-time company auto-association for contacts based on
email domains. This is **Part 2** of the company model production
rollout (CW-5726).
**Task:**
- When a contact is created with a business email, automatically create
and associate a company from the email domain
- When a contact is updated with an email for the first time (email was
previously nil), associate with a company
- Preserve existing company associations when email changes to avoid
user confusion
- Skip free email providers and disposable domains
**Dependencies:**
⚠️ Requires PR #12657 (Part 1: Backfill migration) to be merged first
**Linear ticket:**
[CW-5726](https://linear.app/chatwoot/issue/CW-5726/company-model-setting-it-up-on-production)
## Type of change
- [x] New feature (non-breaking change which adds functionality)
## How Has This Been Tested?
- Service specs: Tests business email detection, company creation,
association logic, edge cases (existing companies, free emails, nil
emails)
- Integration specs: Tests full callback flow for contact create/update
scenarios
- All tests passing: 10 examples, 0 failures
- RuboCop: 0 offenses
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules (PR #12657 pending)
---------
Co-authored-by: Sojan Jose <sojan@pepalo.com>
## Description
Implements company backfill migration infrastructure for existing
contacts. This is **Part 1 of 2** for the company model production
rollout as described in
[CW-5726](https://linear.app/chatwoot/issue/CW-5726/company-model-setting-it-up-on-production).
Creates jobs and services to associate existing contacts with companies
based on their email domains, filtering out free email providers (gmail,
yahoo, etc.) and disposable addresses.
**What's included:**
- Business email detector service with ValidEmail2 (uses
`disposable_domain?` to avoid DNS lookups)
- Per-account batch job to process contacts for one account
- Orchestrator job to iterate all accounts
- Rake task: `bundle exec rake companies:backfill`
~~*NOTE*: I'm using a hard-coded approach to determine if something is a
"business" email by filtering out emails that are usually personal. I've
also added domains that are common to some of our customers' regions.
This should be simpler. I looked into `Valid_Email2` and I couldn't find
anything to dictate whether an email is a personal email or a business
one. I don't think the approach used in the frontend is valid here.~~
UPDATE: Using `email_provider_info` gem instead.
**Pending - Part 2 (separate PR):** Real-time company creation for new
contacts
## Type of change
- [x] New feature (non-breaking change which adds functionality)
## How Has This Been Tested?
```bash
# Run all new tests
bundle exec rspec spec/enterprise/services/companies/business_email_detector_service_spec.rb \\
spec/enterprise/jobs/migration/company_account_batch_job_spec.rb \\
spec/enterprise/jobs/migration/company_backfill_job_spec.rb
# Run RuboCop
bundle exec rubocop enterprise/app/services/companies/business_email_detector_service.rb \\
enterprise/app/jobs/migration/company_account_batch_job.rb \\
enterprise/app/jobs/migration/company_backfill_job.rb \\
lib/tasks/companies.rake
```
**Performance optimization:**
- Uses `disposable_domain?` instead of `disposable?` to avoid DNS MX
lookups (discovered via tcpdump analysis - `disposable?` was making
network calls for every email, causing 100x slowdown)
## Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
---------
Co-authored-by: Sojan Jose <sojan@pepalo.com>
Tanmay Deep Sharma