## Description Inbound email attachments are stored with `file_type: 'file'` regardless of their actual MIME type. As a result, image screenshots shared by customers via email are not exposed to Captain V2's multimodal pipeline — `Captain::OpenAiMessageBuilderService#attachment_parts` selects images via `attachments.where(file_type: :image)` and emits a placeholder `"User has shared an attachment"` text part instead of an `image_url` part. The model never gets the image, so Captain keeps asking the customer to retype information that is already visible in the screenshot. This PR makes the email mailbox derive `file_type` from the blob's `content_type` using the existing shared `FileTypeHelper`, matching how every other inbound channel (`twilio`, `sms`, `telegram`, `line`, `tiktok`, `twitter`, `messenger`) and `MessageBuilder` already classify attachments. Fixes #14448 ## Type of change - [x] Bug fix (non-breaking change which fixes an issue) ## How Has This Been Tested? Reproduced and verified on a self-hosted production instance: 1. Real customer reply via email with a PNG screenshot of an in-app error. Before: ```ruby a = Message.find(<id>).attachments.first a.file_type # => "file" a.file.blob.content_type # => "image/png" Captain::OpenAiMessageBuilderService.new(message: a.message).generate_content # => [{type: 'text', text: '...'}, # {type: 'text', text: 'User has shared an attachment'}] ❌ no image_url ``` Captain reply: "Please copy and paste the full error text…" (model never saw the image). 2. After the patch + force-recreate, same conversation: ```ruby a.file_type # => "image" Captain::OpenAiMessageBuilderService.new(message: a.message).generate_content # => [{type: 'text', text: '...'}, # {type: 'image_url', image_url: {url: 'https://.../<blob>.png'}}] ✅ ``` Captain reply now correctly references the on-screen error text from the screenshot via the multimodal vision path — no more deflection. 3. Regression sanity-check on non-image attachments (PDF / Office docs): `file_type` falls through to `:file`, behavior unchanged. ## Notes for self-hosted operators Existing email image attachments in the DB will still have `file_type: 'file'`. A one-shot backfill is straightforward and safe (no data loss, only metadata): ```ruby Attachment.joins(message: :conversation) .where(messages: { content_type: 'incoming_email' }) .where(file_type: 'file') .find_each do |a| next unless a.file.attached? ct = a.file.blob.content_type.to_s next unless ct.start_with?('image/', 'audio/', 'video/') new_type = ct.start_with?('image/') ? :image : (ct.start_with?('video/') ? :video : :audio) a.update_columns(file_type: Attachment.file_types[new_type]) end ``` ## Checklist - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — happy to add a `mailbox_helper_spec` example for `process_regular_attachments` if maintainers prefer; existing specs in that file focus on inline-image handling. --------- Co-authored-by: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com>
126 lines
4.2 KiB
Ruby
126 lines
4.2 KiB
Ruby
module MailboxHelper
|
|
include MailboxInlineAttachmentHelper
|
|
include ::FileTypeHelper
|
|
|
|
private
|
|
|
|
def create_message
|
|
Rails.logger.info "[MailboxHelper] Creating message #{processed_mail.message_id}"
|
|
return if @conversation.messages.find_by(source_id: processed_mail.message_id).present?
|
|
|
|
@message = @conversation.messages.create!(
|
|
account_id: @conversation.account_id,
|
|
sender: @conversation.contact,
|
|
content: mail_content&.truncate(150_000),
|
|
inbox_id: @conversation.inbox_id,
|
|
message_type: 'incoming',
|
|
content_type: 'incoming_email',
|
|
source_id: processed_mail.message_id,
|
|
content_attributes: {
|
|
email: processed_mail.serialized_data,
|
|
cc_email: processed_mail.cc,
|
|
bcc_email: processed_mail.bcc
|
|
}
|
|
)
|
|
end
|
|
|
|
def add_attachments_to_message
|
|
return if @message.blank?
|
|
|
|
# Load email content once for all attachment processing
|
|
load_email_content
|
|
|
|
# ensure we don't add more than the permitted number of attachments
|
|
all_attachments = processed_mail.attachments.last(Message::NUMBER_OF_PERMITTED_ATTACHMENTS)
|
|
grouped_attachments = group_attachments(all_attachments)
|
|
|
|
process_inline_attachments(grouped_attachments[:inline]) if grouped_attachments[:inline].present?
|
|
process_regular_attachments(grouped_attachments[:regular]) if grouped_attachments[:regular].present?
|
|
|
|
@message.save!
|
|
end
|
|
|
|
def group_attachments(attachments)
|
|
# If the email lacks a text body or if inline attachments aren't images,
|
|
# treat them as standard attachments for processing.
|
|
inline_attachments = attachments.select do |attachment|
|
|
inline_attachment?(attachment)
|
|
end
|
|
|
|
regular_attachments = attachments - inline_attachments
|
|
{ inline: inline_attachments, regular: regular_attachments }
|
|
end
|
|
|
|
def process_regular_attachments(attachments)
|
|
Rails.logger.info "[MailboxHelper] Processing regular attachments for message with ID: #{processed_mail.message_id}"
|
|
attachments.each do |mail_attachment|
|
|
blob = mail_attachment[:blob]
|
|
attachment = @message.attachments.new(
|
|
account_id: @conversation.account_id,
|
|
file_type: file_type(blob.content_type)
|
|
)
|
|
attachment.file.attach(blob)
|
|
end
|
|
end
|
|
|
|
def process_inline_attachments(attachments)
|
|
Rails.logger.info "[MailboxHelper] Processing inline attachments for message with ID: #{processed_mail.message_id}"
|
|
|
|
attachments.each do |mail_attachment|
|
|
embed_inline_image_source(mail_attachment)
|
|
end
|
|
|
|
# update the message content with the updated html and text content
|
|
@message.content_attributes[:email][:html_content][:full] = @html_content
|
|
@message.content_attributes[:email][:text_content][:full] = @text_content
|
|
end
|
|
|
|
def embed_inline_image_source(mail_attachment)
|
|
if @html_content.present?
|
|
upload_inline_image(mail_attachment)
|
|
elsif @text_content.present?
|
|
embed_plain_text_email_with_inline_image(mail_attachment)
|
|
end
|
|
end
|
|
|
|
def embed_plain_text_email_with_inline_image(mail_attachment)
|
|
attachment_name = mail_attachment[:original].filename
|
|
img_tag = "<img src=\"#{inline_image_url(mail_attachment[:blob])}\" alt=\"#{attachment_name}\">"
|
|
|
|
tag_to_replace = "[image: #{attachment_name}]"
|
|
|
|
if @text_content.include?(tag_to_replace)
|
|
@text_content = @text_content.gsub(tag_to_replace, img_tag)
|
|
else
|
|
@text_content += "\n\n#{img_tag}"
|
|
end
|
|
end
|
|
|
|
def inline_image_url(blob)
|
|
Rails.application.routes.url_helpers.url_for(blob)
|
|
end
|
|
|
|
def create_contact
|
|
@contact_inbox = ::ContactInboxWithContactBuilder.new(
|
|
source_id: processed_mail.original_sender,
|
|
inbox: @inbox,
|
|
contact_attributes: {
|
|
name: identify_contact_name,
|
|
email: processed_mail.original_sender,
|
|
additional_attributes: { source_id: "email:#{processed_mail.message_id}" }
|
|
}
|
|
).perform
|
|
|
|
@contact = @contact_inbox.contact
|
|
Rails.logger.info "[MailboxHelper] Contact created with ID: #{@contact.id} for inbox with ID: #{@inbox.id}"
|
|
end
|
|
|
|
def mail_content
|
|
if processed_mail.text_content.present?
|
|
processed_mail.text_content[:reply]
|
|
elsif processed_mail.html_content.present?
|
|
processed_mail.html_content[:reply]
|
|
end
|
|
end
|
|
end
|