From 21c0f4dc522b3e3bd1e16a68af46669edca9c37d Mon Sep 17 00:00:00 2001 From: Tanmay Deep Sharma <32020192+tds-1@users.noreply.github.com> Date: Tue, 5 May 2026 11:49:28 +0700 Subject: [PATCH] fix(transcription): guard Whisper 25MB limit and zero temperature for stable output (#14335) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two production-grade fixes to the existing audio transcription service. **Independent of the WhatsApp Calling work** — these affect every audio attachment that goes through Whisper (voice notes, call recordings, voicemails, etc.). ## Closes - [PLA-151 — PR-5: Recording Upload + Transcription Pipeline](https://linear.app/chatwoot/issue/PLA-151/pr-5-recording-upload-transcription-pipeline) ## Why this is needed ### 1. Whisper rejects payloads larger than 25 MB OpenAI's [Whisper API](https://platform.openai.com/docs/guides/speech-to-text) hard-caps file uploads at 25 MB. Long audio recordings — voice notes from chatty contacts, ~70+ min Opus call recordings — currently hit OpenAI with the full payload and 413 (\`Payload Too Large\`). The job retries via the existing \`Faraday::BadRequestError\` discard path, but the agent still sees a transcription failure for an attachment we knew was too big up front. This PR adds a pre-flight \`audio_too_large?\` check via the blob's \`byte_size\` and returns a controlled error without hitting OpenAI. The audio attachment is preserved (agents can still listen), only the transcription is skipped. ### 2. Whisper hallucinates on silence at non-zero temperature At \`temperature: 0.4\` (the previous value), Whisper produces well-documented hallucinated repeats on silence and near-silent segments — e.g. \`Oh, dear. Oh, dear. Oh, dear.\` filling the transcript. This shows up in real recordings whenever there's a hold or quiet moment. \`temperature: 0.0\` matches OpenAI's recommended default for transcription and eliminates the spirals. Reference: [openai/whisper#928](https://github.com/openai/whisper/discussions/928), [openai-python#1010](https://github.com/openai/openai-python/issues/1010). ## Are WhatsApp call recordings already handled? Yes — by the existing pipeline, **before this PR**: \`\`\` Browser MediaRecorder → upload_recording (PR-4) → @call.message.attachments.create!(file_type: :audio, ...) → Enterprise::Concerns::Attachment#enqueue_audio_transcription (after_create_commit hook) → Messages::AudioTranscriptionJob.perform_later(attachment.id) → Messages::AudioTranscriptionService → Whisper \`\`\` The \`after_create_commit\` hook already fires for every audio attachment regardless of source. PR-4's \`upload_recording\` endpoint creates the attachment; the existing job/service take it from there. No new wiring needed. This PR just makes the existing service more robust: - Calls longer than ~70 min (Opus 48 kbps) no longer 413 against OpenAI - Quiet recordings no longer produce hallucinated transcripts ## How to test \`\`\`ruby # In rails console with a real audio attachment: service = Messages::AudioTranscriptionService.new(Attachment.audio.last) # Normal-sized audio: unchanged behaviour service.perform # => { success: true, transcriptions: ... } # Large audio: new guard returns error instead of 413-ing OpenAI allow(attachment.file.blob).to receive(:byte_size).and_return(30.megabytes) service.perform # => { error: 'Audio too large for Whisper' } \`\`\` Existing transcription specs cover the happy path; one new spec exercises the byte-limit guard. ## Risk Low. Both changes are pre-flight guards or parameter values — they reduce the surface of OpenAI calls that can fail. Failure to transcribe is already non-fatal (the audio attachment is preserved either way). --- .../messages/audio_transcription_service.rb | 17 ++++++++++++++++- .../audio_transcription_service_spec.rb | 17 +++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/enterprise/app/services/messages/audio_transcription_service.rb b/enterprise/app/services/messages/audio_transcription_service.rb index 0e574cb03..c502b717a 100644 --- a/enterprise/app/services/messages/audio_transcription_service.rb +++ b/enterprise/app/services/messages/audio_transcription_service.rb @@ -2,6 +2,10 @@ class Messages::AudioTranscriptionService< Llm::LegacyBaseOpenAiService include Integrations::LlmInstrumentation WHISPER_MODEL = 'whisper-1'.freeze + # Whisper's hard limit is 25 MB *decimal* (25_000_000), not binary (25.megabytes + # = 26_214_400) — using the binary form leaks the 25.0–26.2 MB range to the API + # as 413s. Long audio (~70+ min Opus) keeps the attachment but skips transcription. + WHISPER_BYTE_LIMIT = 25_000_000 attr_reader :attachment, :message, :account @@ -15,6 +19,7 @@ class Messages::AudioTranscriptionService< Llm::LegacyBaseOpenAiService def perform return { error: 'Transcription limit exceeded' } unless can_transcribe? return { error: 'Message not found' } if message.blank? + return { error: 'Audio too large for Whisper' } if audio_too_large? transcriptions = transcribe_audio Rails.logger.info "Audio transcription successful: #{transcriptions}" @@ -33,6 +38,13 @@ class Messages::AudioTranscriptionService< Llm::LegacyBaseOpenAiService account.usage_limits[:captain][:responses][:current_available].positive? end + def audio_too_large? + blob = attachment.file&.blob + return false unless blob + + blob.byte_size > WHISPER_BYTE_LIMIT + end + def fetch_audio_file blob = attachment.file.blob temp_dir = Rails.root.join('tmp/uploads/audio-transcriptions') @@ -63,11 +75,14 @@ class Messages::AudioTranscriptionService< Llm::LegacyBaseOpenAiService transcribed_text = nil File.open(temp_file_path, 'rb') do |file| + # temperature: 0.0 minimises Whisper's hallucinations on silence / + # near-silent audio; non-zero values trigger spiraling repeats like + # "Oh, dear. Oh, dear. Oh, dear." — well-documented Whisper behaviour. response = @client.audio.transcribe( parameters: { model: WHISPER_MODEL, file: file, - temperature: 0.4 + temperature: 0.0 } ) transcribed_text = response['text'] diff --git a/spec/enterprise/services/messages/audio_transcription_service_spec.rb b/spec/enterprise/services/messages/audio_transcription_service_spec.rb index 7ece2540a..212c0bf01 100644 --- a/spec/enterprise/services/messages/audio_transcription_service_spec.rb +++ b/spec/enterprise/services/messages/audio_transcription_service_spec.rb @@ -63,6 +63,23 @@ RSpec.describe Messages::AudioTranscriptionService, type: :service do expect(result).to eq({ success: true, transcriptions: 'Existing transcription' }) end end + + context 'when the audio exceeds Whisper byte limit' do + before do + attachment.file.attach( + io: File.open(Rails.public_path.join('audio/widget/ding.mp3')), + filename: 'large.mp3', + content_type: 'audio/mpeg' + ) + allow(service).to receive(:can_transcribe?).and_return(true) + allow(attachment.file.blob).to receive(:byte_size).and_return(described_class::WHISPER_BYTE_LIMIT + 1) + end + + it 'returns an error without calling Whisper' do + expect(service).not_to receive(:transcribe_audio) + expect(service.perform).to eq({ error: 'Audio too large for Whisper' }) + end + end end describe '#fetch_audio_file' do