Instagram webhooks use a Redis mutex to protect the first-message path
for a contact/account pair. When multiple webhook events for the same
Instagram contact arrive at nearly the same time, they can all enter the
“find or create” flow together.
There are two pieces involved:
`ContactInbox` maps the external Instagram scoped user id to a Chatwoot
contact inside an inbox. That side already has a unique `(inbox_id,
source_id)` index and retry handling, so duplicate contact-inbox
creation is mostly protected at the database layer.
`Conversation` creation is more fragile. The message builder looks for
an existing active conversation for the contact/inbox and creates one
when none is found. If two first-message jobs run concurrently, both can
see “no active conversation yet” and both can create a conversation. The
mutex exists to bump those jobs apart long enough for the first one to
create the conversation, so the next one appends to it instead of
creating a duplicate.
In production, the old behavior could turn that race dampener into a
drop path. Once `Webhooks::InstagramEventsJob` exhausted its lock
retries, ActiveJob stopped retrying with
`MutexApplicationJob::LockAcquisitionError`. Since ActiveJob retries are
not FIFO, later jobs could still win the lock while older jobs kept
getting deferred until exhaustion.
## Mutex Application Job Changes
This adds `retry_on_lock_conflict` as a small wrapper around the
existing `retry_on LockAcquisitionError` pattern.
The new API keeps the default behavior intact, but lets jobs explicitly
define what should happen after lock retry exhaustion:
```ruby
retry_on_lock_conflict wait: 1.second,
attempts: 3,
on_exhaustion: :process_without_lock
```
The fallback receives the original job arguments, so job classes do not
need to reach into ActiveJob internals like `arguments.first`.
## Instagram Fallback
Instagram now processes the webhook payload even after the mutex retry
window is exhausted.
This matches what the mutex was meant to do in the first place: bump
concurrent events apart long enough to avoid duplicate conversation
creation, not permanently block or drop webhook events. If a job cannot
acquire the lock after a few retries, it continues through the normal
Instagram processing path without the mutex.
## Retry And Lock Tuning
The Instagram lock retry window now uses deterministic backoff:
```ruby
retry_on_lock_conflict wait: ->(executions) { executions.seconds },
attempts: 3,
on_exhaustion: :process_without_lock
```
The lock TTL is also set explicitly:
```ruby
with_lock(key, 3.seconds)
```
This matters because Rails treats `attempts` as total executions, not
retries after the first execution. With a fixed `wait: 1.second` and
`attempts: 3`, the exhaustion handler can run roughly two seconds after
the first lock conflict. That is earlier than a 3-second lock TTL, so
the fallback could process without the lock while the original mutex is
still valid. That would reopen the duplicate-conversation race the lock
is meant to dampen.
Using a proc wait makes the timing predictable and avoids Rails jitter
for this retry path. The first conflict waits about 1 second, the second
waits about 2 seconds, and the final attempt happens around the 3-second
mark. That lines the retry window up with the Redis lock TTL before
falling back to `process_without_lock`.
The protected race window is intentionally small. `ContactInbox`
creation already has database uniqueness protection, while conversation
creation is the softer `find active conversation || create` path. We
only need to give the first job enough time to create the conversation
state that later jobs should reuse.
The mutex still does not preserve message order, and ActiveJob retries
are not FIFO. A longer retry window would mostly add latency for hot
Instagram contacts without making ordering more correct. After the lock
TTL has elapsed, processing without the lock is the better tradeoff than
dead-lettering customer messages.
