akadmin/chatwoot
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

chore(automations): remove design docs from branch

Browse Source
This commit is contained in:
Tanmay Deep Sharma
2026-07-15 10:56:07 +05:30
parent 5d5fa0c21a
commit 5c0cbcbaaa
4 changed files with 0 additions and 1215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/delayed-automations-implementation-plan.md
-335
View File
@@ -1,335 +0,0 @@
# Delayed Automations — Implementation Plan (final)
The build source of truth for CW-7513, target **Jul 15**. Companions:
`docs/delayed-automations.md` (approved design — the what/why) ·
`docs/time-based-automations-phasing-and-rollout.md` (product phases + rollout stages —
this plan implements its Phase 0/1).
Supersedes earlier drafts: feature-flag storage was extended on develop
(`873d16f54c`, #14947 — second bitset column `accounts.feature_flags_ext_1`, 0/63 used), so
the flag is a plain features.yml append; no repurpose migration.
---
## Locked decisions
1. **Delay bounds**: `execution_delay` in minutes, `nil` or `10..43_200` (10 min – 30 days).
2. **Feature flag `delayed_automations`** — appended at the end of `config/features.yml`
with `column: feature_flags_ext_1` (mandatory: the legacy `feature_flags` column is
63/63 full; the header documents this), `enabled: false`, `chatwoot_internal: true`.
No DB migration for the flag itself — the ext column already exists; ConfigLoader
reconciles the new name (with its `column` metadata) into
`ACCOUNT_LEVEL_FEATURE_DEFAULTS` on `db:migrate`. The flag is the **per-account stop**,
enforced at three layers: params (Phase 5), arming (Phase 3), fire time (Phase 4).
3. **Instance kill switch `DISABLE_DELAYED_AUTOMATIONS`** — new entry in
`config/installation_config.yml` (`value: false`, `type: boolean`, `locked: false`),
checked in **both** sweep and per-row jobs (Phase 4). Flipping it takes effect within one
tick, no deploy (`InstallationConfig` `after_commit :clear_cache`).
4. **Episode anchors, split by what armed the rule** (this is the only genuinely new logic;
see design doc §5.3):
- *Conversation events*: `"status:{status_changed_at || created_at}"` — clock never
resets while the conversation stays in the state; leaving + re-entering = new episode.
- *`message_created`, triggering message outgoing* (Story 2 — customer went quiet):
`"reply_chase:{max incoming message id}"`; upsert **updates `due_at` + `message_id`**
so the clock tracks the latest agent reply; a customer reply changes the key →
fire-time skip.
- *`message_created`, triggering message incoming* (agent went quiet — the
unassign-on-no-response ask from the CW-7513 comments): `"awaiting_agent:{waiting_since}"`;
conflict = no-op (clock counts from the first unanswered customer message). Chatwoot
clears `waiting_since` on agent/bot reply, so the recomputed key changes and the row
skips — without this anchor, an "unassign after 30 min silence" rule would unassign
agents who replied promptly.
5. **`attribute_changed` conditions + delay are mutually exclusive** (model validation):
the re-check can't reconstruct `changed_attributes` at fire time.
6. **Anchor message persisted** (`message_id` column): `ConditionsFilterService` scopes
message conditions to the triggering message, so the fire-time re-check needs it.
7. **Retention**: `executed`/`skipped` rows kept 30 days (they are the audit trail and the
incident blast-radius query), pruned by the daily scheduled job.
8. **No fallback to immediate execution anywhere**: flag off / guard failure ⇒ skip. A
24h-delayed `send_message` silently becoming instant is worse than not firing.
## Phase 1 — Schema & config
### 1a. Migration: `add_execution_delay_to_automation_rules`
```ruby
add_column :automation_rules, :execution_delay, :integer # minutes, NULL = immediate
```
### 1b. Migration: `add_status_changed_at_to_conversations`
```ruby
add_column :conversations, :status_changed_at, :datetime # no default, no backfill
```
Plain nullable no-default add (the `sla_policy_id`/`assignee_agent_bot_id` precedent).
Readers use `status_changed_at.presence || created_at` — stable across a delay window,
which is all the episode key needs.
### 1c. Migration: `create_automation_rule_pending_executions`
```ruby
create_table :automation_rule_pending_executions do |t|
t.references :automation_rule, null: false
t.references :conversation, null: false
t.references :account, null: false
t.bigint :message_id # anchor message for message_created rules
t.datetime :due_at, null: false
t.string :episode_key, null: false
t.integer :status, null: false, default: 0 # pending: 0, processing: 1, executed: 2, skipped: 3
t.string :skip_reason # rule_inactive | flag_disabled | conversation_gone |
# episode_moved | conditions_changed | expired
t.timestamps
end
add_index :automation_rule_pending_executions, [:status, :due_at]
add_index :automation_rule_pending_executions,
[:automation_rule_id, :conversation_id, :episode_key],
unique: true, name: 'uniq_automation_pending_execution_episode'
```
`processing` is the claim state (double-fire guard); `skip_reason` answers "why didn't my
rule fire" for support and drives incident blast-radius queries.
### 1d. Config files (no migration)
- `config/features.yml`: append `delayed_automations` at the end —
`column: feature_flags_ext_1`, `enabled: false`, `chatwoot_internal: true`. Never reorder;
the entry's position within its column is permanent once merged.
- `config/installation_config.yml`: `DISABLE_DELAYED_AUTOMATIONS` (`value: false`,
`type: boolean`, `locked: false`).
## Phase 2 — Models
### `app/models/automation_rule_pending_execution.rb` (new, flat name per repo convention)
- `belongs_to :automation_rule, :conversation, :account`; optional `belongs_to :message`.
- `enum status: { pending: 0, processing: 1, executed: 2, skipped: 3 }`.
- `scope :sweepable` — due `pending` rows (`due_at <= now`) **or** `processing` rows whose
lock is older than `STALE_PROCESSING_TIMEOUT` (`.or` of two enum scopes). This single scope
replaces the old separate `due` window + `reclaim_stale!` bulk update; stale claims simply
become claimable again. Expiry (`due_at < 3.days.ago`) is decided per-row at fire time.
- `self.purge_terminal!` — `executed`/`skipped` rows past `RETENTION_WINDOW` (30 days) are
`delete_all`ed in batches of 1000 (keeps the table bounded; `delete_all` skips no needed
validation and is not a `SkipsModelValidations` concern).
- `#claim!` — atomic compare-and-set under `with_lock`: a row is claimable if `pending`, or
`processing` but stale. The claim happens **inside the per-row job**, not the sweep, so a
duplicate enqueue (overlapping sweep or reclaimed stale row) loses the claim and returns —
this is the double-fire guard. Refreshing `updated_at` on claim renews the lock.
- `self.schedule(rule:, conversation:, message: nil)` — computes `episode_key` + `due_at =
Time.current + rule.execution_delay.minutes`, then `create!` guarded by the unique episode
index. On `RecordNotUnique`: status / incoming-anchored episodes no-op (clock not reset);
outgoing-anchored (reply-chase) episodes update `due_at` + `message_id` when still `pending`
(a terminal row is never re-armed). Validation-running writes — no `insert`/`upsert`.
- `#episode_current?` — recomputes the episode key from live conversation state, compares.
### `app/models/automation_rule.rb`
- `has_many :pending_executions, class_name: 'AutomationRulePendingExecution',
dependent: :delete_all`.
- Validation: `execution_delay` nil or integer in `10..43_200`.
- Validation: no `execution_delay` together with an `attribute_changed` condition.
- Update schema annotation.
### `app/models/conversation.rb`
- `before_save -> { self.status_changed_at = Time.current if status_changed? }` — covers
create (status always "changes" on create) and every transition; runs before the existing
`notify_status_change` / `dispatch_conversation_updated_event` callbacks so listeners see
the fresh value.
- `has_many :automation_rule_pending_executions, dependent: :delete_all` (also on Account).
- Verify `status_changed_at` needs no entry in the conversation-updated watched-attributes
list — we don't want an extra `conversation_updated` storm; a plain column write inside
the same save is fine.
## Phase 3 — Write path (listener)
### `app/listeners/automation_rule_listener.rb`
Replace the two `ActionService.new(...).perform if conditions_match.present?` call sites
(the `message_created` and `process_conversation_event` loops) with a shared private method:
```ruby
def execute_rule(rule, account, conversation, message: nil)
if rule.execution_delay.present?
return unless account.feature_enabled?('delayed_automations')
AutomationRulePendingExecution.schedule(rule: rule, conversation: conversation, message: message)
else
AutomationRules::ActionService.new(rule, account, conversation).perform
end
end
```
Flag off ⇒ no arming and **no immediate fallback** (locked decision #8). Everything upstream
(loop guard `performed_by_automation?`, auto-reply skip, rule lookup,
`ConditionsFilterService`) is untouched.
## Phase 4 — Fire path (sweep)
### `app/jobs/trigger_scheduled_items_job.rb`
Add one line: `AutomationRules::TriggerPendingExecutionsJob.perform_later`. No change to
`config/schedule.yml` (rides the existing `*/5` cron).
### `app/jobs/automation_rules/trigger_pending_executions_job.rb` (new, queue `scheduled_jobs`)
```ruby
# instance kill switch — MUST use GlobalConfig.get (it applies the `type: boolean` cast;
# get_value returns the raw cached value, and super-admin edits store the STRING 'false',
# which is truthy — the ENABLE_*_CHANNEL_HUMAN_AGENT read pattern):
return if GlobalConfig.get('DISABLE_DELAYED_AUTOMATIONS')['DISABLE_DELAYED_AUTOMATIONS']
purged = AutomationRulePendingExecution.purge_terminal!
rows = AutomationRulePendingExecution.sweepable.order(:due_at).limit(sweep_limit).to_a
rows.each { |row| AutomationRules::ProcessPendingExecutionJob.perform_later(row) }
# The sweep only enqueues — the per-row job claims (double-fire guard), expires, and executes.
# end-of-run summary, JSON so New Relic ingests fields without a parsing rule:
# [AutomationRules::TriggerPendingExecutionsJob] {"event":"completed","enqueued":N,
# "capped":bool,"purged":N,"duration_ms":N}
```
`sweep_limit`: constant on the job (default 1000) with an InstallationConfig override
(Captain `ScheduleSyncsJob` pattern — a no-deploy tuning knob). Overflow rows stay `pending`
for the next tick; log `capped: true` + remaining count — no silent truncation.
Retention pruning lives in `Internal::TriggerDailyScheduledItemsJob`: batched
`where(status: %i[executed skipped]).where(updated_at: ...30.days.ago)` deletes.
Queue reality (`config/sidekiq.yml` is strict-priority): `scheduled_jobs` ranks below `low`,
so the sweep can be late — `due_at <= now` semantics already tolerate that; per-row jobs on
`medium` preempt `default` and below — the cap is what protects the instance. `WebhookJob`
is also on `medium`, so webhook-heavy delayed rules share that budget.
### `app/jobs/automation_rules/process_pending_execution_job.rb` (new, queue `medium`,
mirrors `Sla::ProcessAppliedSlaJob`)
`discard_on ActiveJob::DeserializationError` (a deleted row fails GlobalID lookup). Re-checks
the kill switch first (sweep-only checking would let a full sweep's already-enqueued rows keep
firing after the flip): if set, return without claiming — the row stays `pending` and replays
or expires via the window. Then `claim!` (bail if lost), then the expiry check
(`due_at < DUE_WINDOW.ago` → `expired`). Guard chain, each failure → `skipped!` with
`skip_reason`:
1. rule exists and `active?` → `rule_inactive`
2. account flag still enabled → `flag_disabled` (the per-account stop)
3. conversation exists → `conversation_gone`
4. `pending.episode_current?` → `episode_moved`
5. `ConditionsFilterService.new(rule, conversation, { message: pending.message }).perform` —
the re-check (anchor message passed so message-scoped conditions evaluate correctly;
`changed_attributes` intentionally absent, locked decision #5) → `conditions_changed`
6. all pass → `AutomationRules::ActionService.new(rule, account, conversation).perform`,
then `executed!`
Per-row rescue → `ChatwootExceptionTracker.new(e, account: account).capture_exception`; the
row stays `processing` on unexpected errors (the next sweep re-enqueues it once the lock goes
stale, and `claim!` lets it run again), it is NOT marked skipped.
Loop safety: `ActionService#initialize` already sets `Current.executed_by = rule` and resets
it in `ensure`, so events emitted by delayed actions carry `performed_by: rule` and are
ignored by the listener — identical to the immediate path.
## Phase 5 — API
### `app/controllers/api/v1/accounts/automation_rules_controller.rb`
Param-level flag gating (the enterprise contacts-controller precedent — `permitted_params`
merges `:company_id` only when `feature_enabled?('companies')`): include `:execution_delay`
in `automation_rules_permit` only when
`Current.account.feature_enabled?('delayed_automations')`; when the param is present with
the flag off, render 422 with a clear error (explicit beats silent stripping). `clone` works
unchanged (`dup` copies the column).
### `app/views/api/v1/accounts/automation_rules/partials/_automation_rule.json.jbuilder`
Expose `execution_delay`.
## Phase 6 — Frontend
All under `app/javascript/dashboard/`:
1. **`routes/dashboard/settings/automation/AutomationRuleForm.vue`** — below the Event
select: "Execute" radio group (Immediately / After a delay) + number input + unit select
(minutes / hours / days). Convert to minutes on save; hydrate back to the largest clean
unit on edit. Client-side validation mirrors the 10 min–30 days bounds.
2. **`helper/automationHelper.js`** — `generateAutomationPayload` passes `execution_delay`
through (null when "Immediately"). Add a `formatDelay` helper for the badge label.
3. **`composables/useAutomation.js` / `useEditableAutomation.js`** — default
`execution_delay: null` on create; carry the field when editing/cloning.
4. **Rule list badge** — "Runs after 4h" chip when `execution_delay` present.
5. **Flag gating** — add `delayed_automations` to `featureFlags.js`; render the Execute
control only when `isFeatureEnabledonAccount` says so. Rules that already have a delay
keep their badge when the flag is off, and the settings page shows a "delayed execution
is disabled for this account" banner (flag-off must never silently hide configured
behavior).
6. **i18n** — `i18n/locale/en/automation.json` only: radio labels, unit names, validation
message, badge text, disabled-flag banner.
7. Store (`store/modules/automations.js`) and API client (`api/automation.js`) are generic
passthroughs — no changes expected.
Tailwind-only styling; Composition API `<script setup>` (form already is).
## Phase 7 — Specs (in scope per design doc §7)
- `spec/models/automation_rule_spec.rb` — delay bounds; `attribute_changed` × delay rejection.
- `spec/models/automation_rule_pending_execution_spec.rb` — episode key derivation (all
three anchors); insert/upsert semantics (status + incoming-anchored: clock not reset;
outgoing-anchored: due_at tracks latest agent reply); `episode_current?` incl.
agent-reply-clears-`waiting_since` cancellation; `claim!` (once-only + stale reclaim);
`sweepable` selection; `purge_terminal!`.
- `spec/models/conversation_spec.rb` — `status_changed_at` set on create and every
transition, untouched on non-status saves.
- `spec/listeners/automation_rule_listener_spec.rb` — delayed rule records a pending
execution and does NOT run actions; nil delay unchanged; `performed_by` automation events
create no rows; flag off → no arming AND no immediate fallback.
- `spec/jobs/automation_rules/trigger_pending_executions_job_spec.rb` — due+pending and stale
`processing` enqueued (not future); kill switch no-ops the sweep; cap limits enqueues;
terminal rows past retention purged.
- `spec/jobs/automation_rules/process_pending_execution_job_spec.rb` — full guard chain
with `skip_reason` per branch; per-row `expired`; kill switch leaves row pending; duplicate
enqueue executes once (claim guard); happy path executes
actions exactly once.
- `spec/controllers/api/v1/accounts/automation_rules_controller_spec.rb` —
permits/persists/serializes `execution_delay` when flag on; 422 when submitted with flag
off; clone copies it.
- Story-level integration spec (cheap, high value): Story 1 and Story 2 tables from the
design doc as end-to-end listener→sweep specs with `travel_to`.
## Manual QA script (maps to design-doc stories + rollout Stage B)
1. Story 1: rule `conversation_updated` + `status = pending` + 10 min delay + add label →
flip to pending, touch the conversation twice (no clock reset), wait for sweep → label.
Repeat but resolve before due → no label, row `skipped`/`episode_moved` (resolving
changes `status_changed_at`, so the episode guard fires before the conditions re-check).
2. Story 2: rule `message_created` + `message_type = outgoing` + delay + send_message →
agent reply, then customer reply before due → cancelled (`episode_moved`). Without
customer reply → follow-up sent once; the follow-up itself doesn't re-arm the rule.
3. Agent-idle recipe (CW-7513 comments): rule `message_created` + `message_type = incoming`
+ 30 min delay + `assign_agent: None` → customer message, no agent reply → unassigned at
sweep. Repeat with agent replying before due → NOT unassigned (`waiting_since` episode
cancelled, `episode_moved`).
4. Story 4: pre-existing rule untouched → still fires instantly.
5. Story 5: disable rule with pending row → sweep marks `rule_inactive`; delete rule →
rows gone.
6. Flag: disable `delayed_automations` on the account → delay control hidden + banner
shown; armed rows skip with `flag_disabled`; no new rows armed. Set
`DISABLE_DELAYED_AUTOMATIONS` → sweep no-ops within one tick and queued per-row jobs
revert rows to pending.
7. Local tip: temporarily drop the cron to `*/1` or run
`AutomationRules::TriggerPendingExecutionsJob.perform_now` from console.
## Sequencing / PR split
Stacked PRs off `feature/cw-7513`:
1. **PR 1 — backend**: Phases 1–5 + specs. Reviewable standalone; feature is
**API-complete** — this is the Jul 15 degenerate path: if PR 2 slips, the canary
customer's rules can be created via console/API with only PR 1 shipped.
2. **PR 2 — frontend**: Phase 6 (delay control behind the flag, badge, banner, i18n).
Work order within PR 1: schema/config → conversation hook → pending-execution model
(episode semantics are the only genuinely new logic — do them test-first) → listener branch
→ jobs → controller/jbuilder.
## Risks / edge cases to keep in view
- **Upsert vs. validation races**: `insert`/`upsert` skip AR validations — fine, all values
are server-computed; the unique index is the real guard.
- **`conversation_updated` storm cost**: one indexed insert-conflict per matching event, no
row growth per episode (unique index). Watch the unique index's bloat after rollout.
- **Rule edited while pending**: conditions re-check uses current conditions (by design);
`due_at` keeps its original value (documented v1 behavior).
- **`send_email_to_team` / webhook actions at fire time**: work unchanged — rate limit
(`within_email_rate_limit?`) and `Current.executed_by` both live in ActionService; verify
in specs, don't reimplement.
- **Audit overlay**: `enterprise/app/models/enterprise/audit/automation_rule.rb` audits the
new column automatically — verify the audited payload includes `execution_delay`.
- **Super-admin flag UI**: `selected_feature_flags=` spans both bitset columns since
#14947 — the per-account toggle works with zero extra code; being `chatwoot_internal`,
the flag is visible only on Cloud super-admin until Stage E.
- **Retroactivity limitation** (documented, not a bug): delayed rules arm on events; a new
rule does nothing for conversations already sitting in the target state with no further
activity. Release notes + user guide + rule-form microcopy state this; closing it is a
Phase 2 decision (see rollout doc).
- **Deprecation note in `lib/events/types.rb`** (opened/resolved → status_changed): no
impact, we add no new events.
docs/delayed-automations-overview.md
-149
View File
@@ -1,149 +0,0 @@
# Delayed automations, in five minutes
**Team explainer · CW-7513** — automation rules can now wait. Set a delay on any rule and it
runs later, but only if what triggered it is still true.
> **TL;DR** — We never schedule an *action*. We schedule a *question* — "is this still
> true?" — and only act when the answer is yes.
Companion docs: [design](delayed-automations.md) ·
[implementation plan](delayed-automations-implementation-plan.md) ·
[phasing & rollout](time-based-automations-phasing-and-rollout.md)
---
## One new option on the rule form
Every automation rule gets an **Execute: immediately / after a delay** choice
(10 minutes to 30 days). Everything else — events, conditions, actions — is the automation
feature people already know. The three things customers asked for become plain rules:
| Scenario | Rule | If things change first |
|---|---|---|
| **Stuck in a status too long** | `Conversation Updated` + `status = pending` + **after 4h** → add label | Resolved before the 4 hours pass? Nothing happens. |
| **Customer went quiet** | `Message Created` + `type = outgoing` + **after 24h** → send message | "Just checking in…" — sent once, cancelled automatically if the customer replies first. |
| **Agent went quiet** | `Message Created` + `type = incoming` + **after 30m** → unassign agent | Frees the conversation for reassignment — cancelled the moment the agent actually replies. |
## The core idea: delay + re-check
"Has been in state X for N hours" is the same as "matched at some moment, and still matches
N hours later." So the rule is evaluated twice — once to start the timer, once before acting.
```mermaid
flowchart LR
E[Event fires<br/><small>message, status change…</small>] --> M[Rule matches<br/><small>conditions pass, as today</small>]
M --> T[Timer armed<br/><small>due = now + delay</small>]
T -. delay elapses .-> R{Still true?<br/><small>conditions re-checked</small>}
R -- yes --> A[Actions run]
R -- no --> C[Cancelled silently<br/><small>reason recorded</small>]
style T fill:#e9f2fe,stroke:#2781f6
style A fill:#e4f4ec,stroke:#248a5c
style C fill:#faf1dc,stroke:#a97413
```
The re-check is what makes the semantics right: a follow-up message cancels itself when the
customer replies; a "stuck in pending" label never lands on a conversation that got resolved.
## Under the hood: one table and the cron we already have
No new events, no future-scheduled Sidekiq jobs sitting in Redis for days. A delayed match
writes one row; the existing 5-minute scheduler sweeps whatever is due — the same pattern
snooze, campaigns, auto-resolve and SLA already use.
```mermaid
flowchart LR
E[Conversation event] --> L[Automation listener<br/><small>rule + conditions match</small>]
L -- no delay --> A[Actions run<br/><small>instantly, unchanged</small>]
L -- delay set --> P[(pending execution row<br/><small>who · which rule · due when</small>)]
P --> S[5-minute sweep<br/><small>picks up due rows</small>]
S --> G{Guards + re-check}
G -- still true --> A
G -- not anymore --> K[Skipped<br/><small>reason saved</small>]
style P fill:#e9f2fe,stroke:#2781f6
style A fill:#e4f4ec,stroke:#248a5c
style K fill:#faf1dc,stroke:#a97413
```
Accounts with no delayed rules create zero rows and zero cost. Repeated events for the same
wait collapse into the same row — the timer does not reset every time someone touches the
conversation.
## One conversation, two endings
The "stuck in pending" rule from above, on a real timeline:
**Stays pending → the rule fires**
| 10:00 | 11:30 | 14:00 |
|---|---|---|
| Moved to Pending — timer armed, due 14:00 | Agent adds a note — clock unchanged | Still pending → **label added** ✓ |
**Resolved in time → the rule cancels itself**
| 10:00 | 13:00 | 14:00 |
|---|---|---|
| Moved to Pending — timer armed, due 14:00 | Resolved — the wait it measured is over | **Nothing happens** — cancelled, nothing visible |
Leaving and re-entering Pending starts a fresh timer. A rule fires at most once per
"episode" — the platform owns the anti-loop guarantee; users never build "only once" hacks
into their conditions.
> **Good to know:** delayed rules apply to conversations with activity *after* the rule is
> created. A brand-new rule doesn't retroactively scan the existing backlog — closing that
> gap is on the roadmap (Phase 2).
## "Why didn't my rule fire?"
Every timer that doesn't run records exactly why. Six reasons cover every case:
| Reason | What it means |
|---|---|
| `conditions_changed` | The conversation no longer matched at fire time — the usual, healthy cancellation. |
| `episode_moved` | The wait it was measuring ended — customer replied, agent replied, or the status changed. |
| `rule_inactive` | The rule was switched off or deleted while the timer was running. |
| `flag_disabled` | Delayed automations were turned off for the account after the timer was armed. |
| `conversation_gone` | The conversation was deleted. |
| `expired` | The row was overdue by more than 3 days (e.g. long downtime) — we don't replay stale actions. |
Rows are kept for 30 days — support and engineering can answer "what fired, what didn't,
and why" for any conversation. An admin-facing history view ships in Phase 2.
## Three ways to stop it, three different sizes
| Scope | Lever | Behaviour |
|---|---|---|
| One rule | The existing on/off toggle | Pending timers for a disabled rule are skipped at fire time. |
| One account | `delayed_automations` feature flag | Gates the form control, the timer arming, and firing. Off means nothing arms and nothing fires — it never falls back to running instantly. |
| Whole instance | `DISABLE_DELAYED_AUTOMATIONS` config | Super-admin setting, takes effect within one 5-minute tick, no deploy. Checked by the sweep **and** every in-flight job. |
## Where we are
```mermaid
flowchart LR
A[✅ Built & tested<br/><small>backend + UI, specs green,<br/>verified end-to-end in the app</small>] --> B[Internal dogfood<br/><small>our own account</small>]
B --> C[Canary — Jul 15<br/><small>requesting customers</small>]
C --> D[Cloud GA<br/><small>waved, gated on metrics</small>]
D --> E[Self-hosted<br/><small>next tagged release</small>]
style A fill:#e4f4ec,stroke:#248a5c
style B fill:#e9f2fe,stroke:#2781f6
```
**Covered in v1**
- Delay on any rule, 10 min – 30 days
- All three customer scenarios, including unassign-on-silence
- Automatic cancellation on reply / status change
- Fires at most once per episode — no loops, by design
- Full audit trail with skip reasons
**Deliberately later**
- "Hours since X" conditions for already-quiet conversations — Phase 2
- Business-hours aware delays — Phase 2
- Admin-facing run history UI — Phase 2
- Reminder chains ("remind ×3, then resolve") — Phase 3
- Calendar/cron triggers — Phase 4
---
Code on `feature/cw-7513` · Tracking: CW-7513, parent CW-5790
docs/delayed-automations.md
-335
View File
@@ -1,335 +0,0 @@
# Delayed Automations — Time-Based Automation Rules
**Status:** Design approved, pending implementation
**Target:** July 15 release
**Owner:** Tanmay
**Key decisions**
- Merged into existing automation rules (same model, same UI, same API) — not a separate
feature or a special rule type. Delay is an optional property (`execution_delay`).
- No future-scheduled Sidekiq jobs — DB `due_at` + the existing 5-minute cron sweep
(`TriggerScheduledItemsJob`), the same entry point SLA uses.
- Fire-time **re-check** of the rule's conditions is what gives "stayed in state for N hours"
semantics; the episode guard (key recomputation) is what auto-cancels follow-ups when the
customer replies.
---
## 1. Problem
Chatwoot automation rules are purely **event-driven**. A rule fires at the instant one of five
events occurs (`conversation_created`, `conversation_updated`, `conversation_opened`,
`conversation_resolved`, `message_created`) and never again. There is no way to express
*duration*:
- "If a conversation stays in status X for more than N hours → add a label"
- "If the end user doesn't reply within N hours → send a follow-up message"
- "If a conversation is open for more than N hours → add a label"
Nothing in the system wakes up N hours later to check. That is the gap this feature fills.
## 2. Goals
- Automation rules can be configured with a **delay**: "execute this rule N hours/minutes after
the triggering event, if the conditions still hold."
- Covers the three customer asks above using **existing events, conditions, and actions**.
- Supported actions include at least **add label** and **send message** (both already exist).
- Fully backward compatible: existing rules keep behaving exactly as today.
- Safe at Chatwoot Cloud volume: load must scale with *pending delayed rules*, not raw event traffic.
## 3. Non-goals (v1)
- **Per-action delays / multi-step sequences** (wait 24h → message → wait 24h → label).
v1 is one delay per rule. Sequences are a natural follow-up.
- **Exact-second firing.** Delays are hours-scale; firing within ~5 minutes of the due time
is acceptable.
- **New duration condition attributes** (`hours_in_status`, etc.). The delay + re-check model
covers the v1 stories without them; richer "time since X" anchors are Phase 2 — see
`docs/time-based-automations-phasing-and-rollout.md`.
- **Recurring execution** (fire every N hours while the condition holds). v1 fires at most
once per conversation per qualifying episode.
## 4. Core model: delay + re-check
The design rests on one insight:
> **"Has been in state X for N hours" ≡ "matched the conditions at T₀, and still matches at T₀ + N."**
So instead of building a new duration-condition engine, we:
1. Evaluate the rule's conditions at event time, exactly as today.
2. If the rule has a delay, **record a pending execution** due at `now + delay` instead of
acting immediately.
3. When the due time arrives, **re-evaluate the same conditions** against the conversation's
current state.
- Still matches → run the actions.
- No longer matches → discard silently. This is what makes the semantics correct: a
follow-up message is automatically cancelled when the customer replies, a "stuck in
pending" label is skipped if the conversation moved on.
Everything downstream — `AutomationRules::ConditionsFilterService`,
`AutomationRules::ActionService`, all existing actions — is reused unchanged.
### Not a special type of rule
A delayed rule is **not** a new rule type — no STI, no new class, no new `event_name`. It stays
a plain `AutomationRule`; the only difference is whether `execution_delay` is set:
```ruby
rule.execution_delay # => nil → executes immediately (today's path)
rule.execution_delay # => 240 → records a pending execution due in 4 hours
```
Why a property beats a type:
- Any existing rule can become delayed (and back) by setting one field, without recreating it
as a different kind of object.
- The listener branch is a single `if` at the point where actions would run; everything
upstream (rule lookup by account+event, condition evaluation) is identical for both paths.
- The UI stays one form. A separate type would mean a type picker, separate list filters, and
separate API handling — ceremony with no behavioral payoff.
The only places it *acts* special are the rule-list badge ("runs after 4h") and the fire-time
guard chain — both driven off the field, not a type.
### Why not Sidekiq `perform_in`?
The obvious implementation (enqueue a future Sidekiq job per matching event) is wrong at our
volume:
- `conversation_updated` fires on nearly every touch. Each match would push a job into Redis'
scheduled set, where it sits for **hours to days**. Redis bloats and the scheduled-set poller
degrades. Work scales with raw event volume — the worst thing to couple to.
- Cancelling/deduping scheduled Sidekiq jobs is awkward (no efficient lookup by
rule+conversation).
Instead we use the pattern Chatwoot already uses for snooze, auto-resolve, campaigns, and SLA:
**store a `due_at` in Postgres and sweep due rows from the existing 5-minute cron**
(`TriggerScheduledItemsJob`). One indexed row per pending (rule, conversation) pair, a bounded
`WHERE due_at <= now` range scan, Redis untouched.
## 5. Architecture
### 5.1 Data model
**`automation_rules.execution_delay`** — new integer column, delay in **minutes**.
`NULL` (default) = execute immediately, i.e. today's behavior. No backfill needed.
**`automation_rule_pending_executions`** — new table (name TBD):
| Column | Type | Notes |
|---|---|---|
| `automation_rule_id` | bigint FK | rule to execute |
| `conversation_id` | bigint FK | target conversation |
| `account_id` | bigint FK | for scoping/cleanup |
| `due_at` | datetime | `event_time + rule.execution_delay` |
| `episode_key` | string | identifies the qualifying episode (see §5.3) |
| `status` | enum | `pending` / `processing` / `executed` / `skipped` |
| timestamps | | |
Final schema (incl. the `message_id` anchor, `skip_reason`, and the `processing` claim
state) is in the implementation plan, Phase 1c.
Indexes:
- `(status, due_at)` — the sweep query.
- Unique `(automation_rule_id, conversation_id, episode_key)` — dedup: repeated events for the
same episode collapse into one row. For status- and waiting_since-anchored episodes the
clock is **not reset**; outgoing-anchored (reply-chase) episodes update `due_at` to track
the latest agent reply (impl plan, locked decision #4).
**`conversations.status_changed_at`** — new datetime column, set on every status transition
(hooked into the existing status-change path on `Conversation`). Serves two purposes:
- It is the `episode_key` source for status-based rules, so "entered open → left → re-entered
open" counts as a *new* episode with a fresh timer.
- Lets the fire-time check confirm the status held continuously, not just coincidentally at
both endpoints.
### 5.2 Event-time flow (write path)
Inside the existing `AutomationRuleListener` path — no new events:
```
event fires
└─ for each active rule matching (account, event_name):
ConditionsFilterService.perform
└─ matched?
├─ rule.execution_delay.nil? → ActionService.perform (unchanged, today's path)
└─ rule.execution_delay set → upsert pending execution
due_at = now + delay
(unique index ⇒ one row per episode; clock NOT
reset — except outgoing-anchored reply-chase
episodes, which update due_at to track the
latest agent reply)
```
### 5.3 Episode semantics — "don't reset the clock"
`conversation_updated` fires on every touch. For a rule like *"status = pending for 4 hours"*:
- First event where conditions match → pending row created, `due_at = entered_pending_at + 4h`,
`episode_key = status_changed_at`.
- Every subsequent update while still pending → same episode key → upsert no-ops. The timer
keeps counting from when the conversation *entered* the state.
- Conversation leaves pending, later returns → new `status_changed_at` → new episode → fresh
timer. The old pending row is discarded at fire time by the re-check.
For message rules the anchor depends on the triggering message's direction (impl plan,
locked decision #4): *outgoing-anchored* (customer went quiet) keys on the latest incoming
message id, with `due_at` tracking the latest agent reply; *incoming-anchored* (agent went
quiet) keys on `waiting_since`, which Chatwoot clears on agent/bot reply — so the reply
cancels the pending action via the episode guard.
### 5.4 Fire-time flow (sweep)
Hooked into the existing **`TriggerScheduledItemsJob`** (every 5 minutes, `config/schedule.yml`
→ `trigger_scheduled_items_job`, cron `*/5`) — the same entry point SLA piggybacks on. Note:
SLA does *not* run every minute; it also rides this 5-minute job.
```
TriggerScheduledItemsJob (cron, */5)
└─ AutomationRules::TriggerPendingExecutionsJob
└─ PendingExecution.pending.where(due_at: ..Time.current)
.find_each → ProcessPendingExecutionJob (per record):
1. rule still exists & active? no → mark skipped
2. conversation still exists? no → mark skipped
3. episode still current? (anchor unchanged)
no → mark skipped
4. ConditionsFilterService still true? no → mark skipped ← the re-check
5. ActionService.perform → mark executed
```
**Same pattern as SLA, but flatter.** SLA needs a 3-level fan-out
(`TriggerSlasForAccountsJob` → per-account `ProcessAccountAppliedSlasJob` → per-record
`ProcessAppliedSlaJob`) because it must *recompute* deadlines from conversation timestamps on
every pass — it doesn't know in advance when a breach will happen. We do know: `due_at` is
precomputed at event time. So we skip the per-account fan-out entirely:
- One global indexed range query on `(status, due_at)`. The table only contains rows that are
actually waiting — accounts with no delayed rules contribute zero rows, zero cost.
- Per-record jobs for the actual execution (guards + re-check + `ActionService`), so one slow
or failing conversation doesn't block the batch — this part mirrors
`Sla::ProcessAppliedSlaJob` exactly.
**Why 5-minute cadence is enough.** Delays are hours-scale ("4 hours", "24 hours"); worst case
a rule fires 5 minutes late, which is invisible at that scale. A 1-minute cadence would buy
precision nobody asked for at 5× the cron load. If tighter precision is ever needed, the fix is
cadence-only — one line in `schedule.yml` — with no design change.
Notes:
- `Current.executed_by = rule` is set exactly as in the immediate path, so events emitted by the
delayed actions are recognized as automation-performed and don't retrigger rules (loop guard).
- Executed/skipped rows are kept 30 days (audit trail / incident blast-radius queries — impl
plan locked decision #7), pruned by the daily scheduled job.
### 5.5 Volume characteristics
| Concern | Answer |
|---|---|
| Redis scheduled-set growth | Zero — no future jobs enqueued |
| Write amplification from update storms | Collapsed by the unique episode index (1 row per rule×conversation×episode) |
| Sweep cost | Indexed range scan on `(status, due_at)`; scales with count of *due* rows, not traffic |
| Accounts without delayed rules | Zero cost — no pending rows are ever created |
| Firing precision | Within the 5-min cron window; fine for hours-scale delays |
### 5.6 Frontend
In the Add/Edit Automation Rule dialog, below Event (per the mockup):
```
Execute ( • ) Immediately
( ) After a delay: [ 4 ] [ hours ▾ ]
```
- Stored as minutes in `execution_delay`; UI offers minutes/hours/days units.
- Shown for all existing events — no new event types in the dropdown.
- Rule list row shows a badge like "runs after 4h" for delayed rules.
- Changes: `AutomationRuleForm` + `constants.js` untouched for conditions/actions;
`automation.json` i18n additions; API strong params + jbuilder gain `execution_delay`.
## 6. User stories & expected behavior
### Story 1 — Label conversations stuck in Pending
> *As an admin, I create: Event = **Conversation Updated**, Condition = **Status equals Pending**,
> Delay = **4 hours**, Action = **Add label `stale-pending`**.*
| What happens | Behavior |
|---|---|
| Conversation moves to Pending at 10:00 | Rule matches → pending execution created, due 14:00 |
| Agent adds a private note at 11:30 (another `conversation_updated`) | Same episode → no new row, **clock not reset**, still due 14:00 |
| Conversation still Pending at 14:00–14:05 sweep | Re-check passes → label `stale-pending` added |
| — or — conversation was resolved at 13:00 | Re-check fails at 14:00 → nothing happens, row marked skipped |
| Conversation reopens and goes Pending again next day | New `status_changed_at` → new episode → fresh 4-hour timer |
### Story 2 — Follow up when the customer goes quiet
> *As a support lead, I create: Event = **Message Created**, Condition = **Message Type is
> Outgoing**, Delay = **24 hours**, Action = **Send message** "Just checking in — did that solve
> it for you?"*
| What happens | Behavior |
|---|---|
| Agent replies to the customer Monday 15:00 | Pending execution created, due Tuesday 15:00 |
| Agent sends two more replies Monday 15:10 | Anchor updates → the follow-up tracks the latest agent reply |
| Customer replies Monday 18:00 | The reply changes the episode key; at fire time the episode guard sees it → **follow-up silently cancelled** (`episode_moved`) |
| Customer never replies | Tuesday ~15:00–15:05 → follow-up message sent to the end user, exactly once |
| The follow-up message itself is created | `Current.executed_by = rule` → does **not** re-arm the rule (no infinite follow-up loop) |
*(v1 note: whether the follow-up re-arms after each new agent reply, or fires once per
conversation, is decided by the episode key — default is once per "waiting" episode.)*
### Story 3 — Escalation label on long-open conversations
> *As an admin, I create: Event = **Conversation Created**, Condition = **Status equals Open**
> (optionally + Inbox = Support), Delay = **48 hours**, Action = **Add label `sla-risk`**.*
| What happens | Behavior |
|---|---|
| Conversation created Wednesday 09:00, still open Friday 09:00 | Label `sla-risk` added at the Friday ~09:00 sweep |
| Conversation resolved Thursday | Re-check fails Friday → no label |
| Conversation resolved Thursday, **reopened** Friday 08:00 | v1: original episode ended → no label from the old timer. (`conversation_opened` + delay can cover re-opens as a separate rule.) |
### Story 4 — Existing rules are untouched
> *As an existing customer, my current rules have no delay.*
| What happens | Behavior |
|---|---|
| Any existing rule fires | `execution_delay` is `NULL` → immediate path, byte-for-byte today's behavior |
| I edit an old rule and never touch the delay field | Still immediate |
### Story 5 — Rule lifecycle while a timer is pending
| What happens | Behavior |
|---|---|
| Admin **disables** the rule while executions are pending | Fire-time guard: rule inactive → all its pending rows skipped |
| Admin **deletes** the rule | Pending rows removed (FK/dependent destroy) |
| Admin **edits the delay** from 4h → 8h | Applies to new episodes; already-pending rows keep their original `due_at` (simple, predictable v1 rule) |
| Admin edits the **conditions** | Fire-time re-check uses the *current* conditions — the edited rule is what's enforced |
| Conversation deleted before due | Guard: skipped |
## 7. Scope for July 15
**In:**
1. Migration: `automation_rules.execution_delay`, `conversations.status_changed_at`,
pending-executions table.
2. Listener wiring: delayed rules record a pending execution instead of executing.
3. Sweep job hooked into `TriggerScheduledItemsJob` with the fire-time guard chain.
4. Frontend delay field in the rule form + list badge + i18n (`en.json` only).
5. Specs: model validations, listener branching, sweep guards/re-check, episode dedup.
**Out (follow-ups):** per-action wait steps, recurring fires, business-hours-aware delays
(SLA's `Sla::BusinessHoursService` is the ready-made building block when we want it),
exact-time firing.
## 8. Open questions
1. **Delay bounds** — enforce a min (≥ 5 min, below cron granularity is meaningless) and a max
(e.g. 30 days) at the model level.
2. **Follow-up re-arm policy** (Story 2): once per waiting-episode (default) vs. once per
conversation ever. Proposing per-episode.
3. **Enterprise overlay** — feature ships in OSS core (automation rules are OSS; the scan cost
is self-limiting since accounts without delayed rules create zero rows). Confirm no
enterprise gating is wanted.
4. Table/row retention: ~~delete executed/skipped rows immediately vs. keep N days~~ —
**resolved**: keep 30 days (audit/blast-radius), pruned by the daily scheduled job.
docs/time-based-automations-phasing-and-rollout.md
-396
View File
@@ -1,396 +0,0 @@
# Time-Based Automations — Product Phasing & Rollout Plan
**Status:** Draft for review (adversarially reviewed; implementation plan amended to match — see §5 note)
**Owner:** Tanmay
**Companions:** `docs/delayed-automations.md` (v1 design, approved) ·
`docs/delayed-automations-implementation-plan.md` (v1 build plan — amended by this doc's review)
**Tracking:** CW-7513 (v1, due Jul 15) · CW-5790 (parent tracker, "Needs Spec" — this doc is
the spec for everything past v1)
---
## 1. The end goal, defined
"Time-based automations" is not one feature — it is a ladder that every mature support
platform climbs in the same order (Zendesk, Freshdesk, Intercom, HubSpot, Help Scout, Front
all converge on it):
1. Delayed action with fire-time re-check ← **v1 (CW-7513)**
2. Run-once / anti-loop semantics owned by the platform, not the user
3. Rich "time since X" condition vocabulary (created / entered status / last customer reply /
last agent reply / assigned)
4. Business-hours-aware time accounting
5. SLA-integrated escalation (at-risk + breach stages with actions)
6. Multi-step sequences (follow-up → wait → follow-up → resolve, interruptible on reply)
7. Scheduled/calendar triggers and full workflow builder
**End goal for Chatwoot (12-month horizon): rungs 1–6.** Rung 7 (visual builder, cron
triggers) is a separate product investment and stays out of this plan except as a
compatibility constraint: nothing we ship may assume "one rule = one delayed action" so
deeply that sequences can't be layered on.
Three facts from the research shape everything below:
- **The v1 engine is already the end-state engine.** Per-conversation pending executions with
fire-time re-check is the architecture Front and Intercom use; Zendesk/Freshdesk's hourly
full-table sweeps are the legacy model whose user-visible warts (whole-hour granularity,
±1 h slop, "nullifying condition" hacks) we get to skip. Every later phase is additive
vocabulary and UI on the same table + sweep — no rewrite is on the ladder.
- **One capability the sweep model has that ours doesn't: eventless, retroactive
evaluation.** An event-armed engine does nothing for conversations already sitting in the
target state when a rule is created. This is a real, user-visible limitation ("why didn't
it label my existing backlog?") — v1 documents it plainly (§3, Phase 1) and Phase 2 carries
a named scope decision for closing it.
- **Community demand is mapped, with honest issue states.** Phases cite Chatwoot GitHub
issues as *evidence of demand*; several were bulk-closed as stale (marked below), so they
document the ask, not a ticket we get to close.
## 2. Product phases at a glance
| Phase | Ships | Evidenced demand | Target |
|---|---|---|---|
| **0. Engine (dark)** | Schema, pending-execution engine, sweep, guardrails, feature flag — no UI | — | with v1 PRs |
| **1. Delayed rules** | `execution_delay` + fire-time re-check; both reply directions via episode anchors; delay UI; badge | CW-7513 (all 3 examples + unassign-on-idle from comments); #1270 *(partial: single reminder)* | **Jul 15** |
| **2. Time vocabulary + business hours + run history** | "Time since last customer/agent reply, time in status, time since created" conditions; run inside/outside business hours; per-rule execution history UI | #6861 (open), #10455 (open), #12694 (open), #4542 *(closed-stale; documents the WhatsApp-24h ask)*; #6889 *(recipe becomes possible)* | ~6 weeks after Cloud GA |
| **3. Follow-up chains + SLA escalation** | Repeat-with-terminator ("remind ×N then resolve"); SLA at-risk/breach as automation events; snooze-duration action | #1270 (open, full ask), #6889 (open), #10118 *(closed-stale)* | Q4 2026 |
| **4. Scheduled triggers / builder** | Calendar/cron rule triggers, scheduled sends, visual sequences | #9484 (open) + scheduled-send cluster (#5554 closed-stale, #9771, #11486, #12146, #12823); CW-5790 original ask | 2027, own spec |
Each phase is independently shippable, independently valuable, and strictly additive on the
Phase 0 engine.
## 3. Phase detail
### Phase 0 — Engine, shipped dark
Everything in `docs/delayed-automations-implementation-plan.md` (migrations, pending-execution
model with claim states, listener branch, sweep jobs with caps/windows/kill switch, API
gating, feature flag) merged with no UI exposure. `execution_delay` stays `NULL` for every
rule, so the listener branch is dead code for 100% of traffic until the flag turns the UI on —
this is what makes the merge risk-free.
Exit criteria (measurable, and achievable while dark):
- Engine + guard specs green.
- Sweep running in production logging `"due":0,"enqueued":0` every 5 minutes.
- All three CW-7513 stories exercised **on staging** with compressed delays (10 min).
- §6 alerts created and verified end-to-end with a test trigger.
(Live dogfood on Chatwoot's own account requires the flag on — that is Stage B, Phase 1.)
### Phase 1 — Delayed rules (CW-7513, Jul 15)
The v1 surface: "Execute: immediately / after a delay" on the existing rule form, delay badge
in the rule list.
**What "time since last reply" means in v1 — both directions, precisely:**
- *Customer went quiet* (CW-7513 example 2, design-doc Story 2): rule anchored on an
**outgoing** message. Episode tracks the latest agent reply; a customer reply cancels the
pending follow-up at fire time.
- *Agent went quiet* (the unassign ask in the Linear comments; the #6861 phrasing): rule
anchored on an **incoming** message. Episode is keyed on `waiting_since`, which Chatwoot
clears when an agent or bot replies — so an agent reply cancels the pending action at fire
time. Without this anchor the recipe would unassign agents who *did* reply promptly; the
implementation plan's episode-key section now specifies both anchors (this was the single
most important correction from review).
With that, the unassign-on-no-response recipe from the Linear comments is buildable in the
UI on day one (the agent dropdown already offers "None", serializing to the `'nil'`
sentinel — `useAutomationValues.js:88`): *Message Created + Message Type is Incoming + delay
30 min + Assign agent: None*. Ship it as a documented recipe. The thread's follow-up ask
(agent online/offline as a condition) is not in v1 — logged as a Phase 3 open item.
**Known limitation, stated everywhere users will hit it** (release notes, user-guide,
rule-form microcopy): delayed rules apply to conversations that have activity *after* the
rule is created. A brand-new "stuck in Pending 4h" rule does not retroactively arm for
conversations already sitting in Pending. Closing this is a named Phase 2 decision.
**Run-once semantics** (ladder rung 2) are built in from day one via episode keys — the
lesson from Zendesk/Freshdesk, whose users must hand-build nullifying conditions. Never
expose a knob that lets a rule re-fire hourly.
**Observability in v1 is internal-only**: support/eng can answer "why didn't it fire" from
the pending-executions table (`skip_reason` column). Admin-facing run history ships in
Phase 2 — until then this is a known support load, priced in.
Exit criteria:
- Before Jul 15: the requesting customer confirms the two reply-direction readings above
match their expectation (don't discover a mismatch after release).
- Canary accounts using delayed rules for 1 week: `expired` = 0 sustained, fire-lag p95
< 10 min, zero new Sentry groups under the two job classes.
- Story 2's cancellation observed in production (a real customer reply cancelling a pending
follow-up), not just in specs.
- Week-1 executed:skipped mix recorded as the baseline band for Stage D gating.
### Phase 2 — Time vocabulary, business hours, run history
What users ask for the moment v1 lands (top evidenced asks):
1. **"Time since X" conditions** — new condition attributes evaluated by the same engine:
`hours_since_last_customer_message`, `hours_since_last_agent_message`,
`hours_in_current_status`, `hours_since_created`. Mechanically these are alternate
**episode anchors + due-at sources** for a pending execution — the rule form gains a
"when time elapses" trigger style that composes with existing conditions.
**Named scope decision — arming for quiet/pre-existing conversations:** event-armed
pending rows don't cover conversations with no future events. Options: (a) one-time
arming scan at rule save (bounded, predictable — recommended), (b) periodic eligibility
scan (reintroduces exactly the per-account sweep v1 avoided; needs its own cost model).
Decide at Phase 2 spec with Cloud row-count data from Phase 1.
2. **Business hours** — two separable features, in this order:
a. *Gate*: "only run this rule inside/outside inbox business hours" (a fire-time guard
against `working_hours`) — the #10455/#12694 ask.
b. *Accounting*: "4 business hours" delays that pause the clock outside schedules —
reuse `Sla::BusinessHoursService`. That service lives in `enterprise/`; either
accounting becomes a premium capability (matching SLA) or the calculator moves to OSS
core. Decide at Phase 2 spec; the schema needs nothing either way.
3. **Re-arm policy control** — expose the v1 episode semantics as an explicit choice
("once per conversation" / "every time it re-qualifies"), defaulting to current behavior.
4. **Per-rule execution history UI** — fired/skipped(+reason) list on the rule, from the
existing table + `skip_reason` (30-day retention shown as a visible constraint). This is
the Zendesk/Intercom-parity observability the feature's support load demands.
Exit criteria (self-contained, not tied to issue states):
- A rule "send a template when >23h since the last inbound customer message on a WhatsApp
inbox" works end-to-end (the WhatsApp 24h-session use case).
- A rule gated to business hours provably does not fire outside them; delay *accounting*
verified against SLA's calculator on shared fixtures.
- Each re-arm policy behaves per spec in the Story-2 integration test.
- An admin can answer "why didn't my rule fire on this conversation" from the rule's history
page without contacting support.
### Phase 3 — Follow-up chains + SLA escalation
1. **Repeat-with-terminator** (#1270's full ask): "send reminder every N hours, at most K
times, then resolve." Schema: `max_occurrences` + occurrence counter on the pending
execution — the episode model already prevents overlap. Sequences-*lite*: one repeated
action + one terminal action, deliberately short of a builder.
2. **SLA integration** (#6889): emit `sla_missed` / `sla_at_risk` as automation events
(enterprise overlay adds them to the event dropdown, the same way `sla_policy_id`
conditions are added today), so escalation rules compose from existing actions. SLA
already computes the timing; automations subscribe. (Note: #6889's literal "waited more
than X hours → escalate" becomes satisfiable with Phase 2 vocabulary — publish that
recipe with Phase 2; Phase 3 makes it SLA-native.)
3. **Snooze composition** (#10118's ask): `snooze_conversation` action gains a duration
param; snooze-expiry already rides the same 5-min sweep.
4. Evaluate the agent online/offline condition (Linear-thread follow-up ask) with
assignment-policy input.
Exit criteria: Intercom's canonical journey (reply-chase → reminder → auto-resolve)
buildable in two rules; SLA-escalation recipe documented; sweep p95 flat vs Phase 2 baseline.
### Phase 4 — Scheduled triggers / builder (out of this plan's scope)
Calendar/cron triggers ("every Monday 9am"), scheduled one-off sends, visual multi-step
builder. Needs its own spec (CW-5790 becomes that spec once Phases 1–3 close the
duration-based demand). The only obligation now: keep `event_name` + pending-executions
generic enough that a `schedule` pseudo-event can create pending executions without a
conversation event — the current schema (rule, conversation, due_at, episode) already
permits this.
## 4. Rollout plan (Phase 1 in full; later phases reuse the template)
### 4.1 Gating levers
| Lever | Choice for this feature |
|---|---|
| Per-account feature flag | `delayed_automations`, appended at the end of `config/features.yml` with **`column: feature_flags_ext_1`**. The bit-budget problem this plan originally flagged was solved on develop (`873d16f54c`, #14947): a second bitset column `accounts.feature_flags_ext_1` exists (0/63 used) and the features.yml header now mandates new flags use it — the legacy `feature_flags` column is 63/63 full. No repurpose migration; ConfigLoader reconciles the new name (with its `column` metadata) into `ACCOUNT_LEVEL_FEATURE_DEFAULTS` on migrate, and `selected_feature_flags=` / super-admin toggles span both columns since #14947. `enabled: false`, `chatwoot_internal: true` at introduction. Flag name and its position within the ext column are frozen forever once merged. |
| What the flag gates | **Three layers, one coherent semantics — the flag is the per-account stop.** (1) *Configuration*: the delay control in the rule form (`featureFlags.js` + `isFeatureEnabledonAccount`), and server-side, `execution_delay` is accepted only when the flag is on — param-level gating per the enterprise contacts-controller precedent (`permitted_params` includes `:company_id` only when `feature_enabled?('companies')`), returning 422 when a delay is submitted with the flag off (explicit beats silent stripping). (2) *Arming*: the listener does not create pending executions for flag-off accounts — and does **not** fall back to immediate execution (a 24h-delayed message silently becoming instant is worse than skipping). (3) *Fire time*: the guard chain checks the flag and marks rows `skipped` / `skip_reason: flag_disabled`. Rules keep their delay badge when the flag is off; the settings page shows a "delayed execution is disabled for this account" banner. |
| Instance kill switch | `DISABLE_DELAYED_AUTOMATIONS`, **declared in `config/installation_config.yml` with `type: boolean`** (so "false" means false — the bare `DISABLE_GRAVATAR` presence-check pattern would treat any non-blank string as disabled), checked in **both** the sweep job *and* the per-row job (sweep-only would let up to a full sweep's already-enqueued per-row jobs fire after the flip). Rows the per-row job refuses stay `pending` and replay or expire via the due-window. Takes effect within one tick, no deploy (`InstallationConfig` `after_commit :clear_cache`). |
| Rule-level off | Existing `active` toggle — already in the fire-time guard chain. |
| Premium? | **Ship non-premium** (see Decisions, §7). The `premium: true` + `ReconcilePlanFeaturesService` lever stays available with zero schema cost if the business later wants plan-gating. |
One honesty note: the wave-enablement rake in Stage D is **new work**, not existing
machinery. Its safety kit (dry-run default, `APPLY=true`, `LIMIT=n`, confirmation) is
specified here, modeled on `assignment_v2:migrate`'s interactive confirm + `ACCOUNT_ID`
scoping and `reporting_events_rollup.rake`'s dry-run prompt — no committed rake has the full
kit today.
### 4.2 Stages
**Stage A — Dark merge (deadline: Fri Jul 10 EOD; Sat Jul 11 is slack, not the plan).**
PR 1 (engine + flag, per the amended implementation plan) and PR 2 (UI behind flag) land on
`develop`. Named reviewers with a same-day review SLA are a §7 decision to lock **today**.
Confirm the weekend deploy policy: if Cloud doesn't deploy weekends, Friday's deploy is the
last train before dogfood. §5 marks which guardrails are Stage-A-blocking vs fast-follow so
a compressed schedule sheds the right load. Everyone is on the immediate path; the sweep
runs and logs `"due":0,"enqueued":0`.
**Stage B — Internal dogfood (Fri Jul 10 evening – Tue Jul 14).**
Enable the flag on Chatwoot's own support account + staging via super-admin
(`chatwoot_internal` keeps it invisible to self-hosted super-admins). The window spans a
weekend — compensate: staging rules at 10-minute delays to compress many cycles Sat–Sun,
plus one realistic 24h rule on the production support account; name who watches the first
sweep ticks Saturday morning. Go/no-go **Tue Jul 14** requires at least one business day
(Mon) of real traffic and: fire-lag p95 < 10 min, `expired` = 0, zero unexpected Sentry
groups, every skip row's `skip_reason` explainable, **and Story 2's cancellation path
observed against real traffic** (customer replied → follow-up cancelled).
**Stage C — Canary (Wed Jul 15 — the release commitment).**
Enable for a hand-picked cohort: the CW-7513 requesting customer, the accounts attached to
CW-5790's three customer conversations, ~10–20 design partners. **White-glove the first
one**: on Jul 15, set up the customer's three rules together with them, so the one-week
canary clock starts on day one instead of waiting for self-serve discovery. Mechanism:
super-admin toggle or `account.enable_features!('delayed_automations')`. CW-7513 is
"released" at this stage. Delayed `send_message` rules stay within dogfood + canary
accounts until Stage D (highest-blast-radius action; messages can't be unsent).
**Degenerate paths (decided now, not during the incident):**
- PR 2 (UI) slips → Jul 15 is still met: the feature is API-complete after PR 1; enable the
customer's flag and create their three rules via console/API.
- PR 1 slips past Mon Jul 13 → customer comms + Stage C moves to Jul 16–17. The commitment
is the capability in the customer's hands, not a specific artifact.
**Stage D — Cloud GA (earliest Jul 22; gated, not dated).**
Entry gate: ≥5 canary accounts each with ≥1 delayed rule that completed ≥1 full execution
cycle, and Phase 1 exit metrics holding. Waves ordered by *real* signal (an inert-flag wave
proves nothing — the flag only unlocks configuration):
1. **Wave 1**: accounts with ≥1 active automation rule, batched ascending by conversation
volume — the population that will actually create delayed rules, arriving gradually.
2. **Wave 2**: high-volume automation accounts, after Wave 1 metrics hold for 24–48h.
3. Zero-rule accounts get the flag via the new-account default flip below — bulk-enabling
them separately is ceremony.
Pause criterion between batches: any `expired > 0` sustained or new Sentry group within 24h
of a batch. New rake task carries the specified safety kit (dry-run default, `APPLY=true`,
`LIMIT=n`, interactive confirmation). Simultaneously flip the new-account default:
migration mutating `ACCOUNT_LEVEL_FEATURE_DEFAULTS['delayed_automations'].enabled = true` +
`GlobalConfig.clear_cache` (assignment_v2 `20260409091202` precedent).
**Stage E — Self-hosted GA (next tagged release after Stage D holds for 2 weeks).**
In one PR: `enabled: true` in `features.yml` (new installs), drop `chatwoot_internal`
(flag becomes visible in self-hosted super-admin), ship the Stage D config-flip + backfill
migrations (existing installs get them on `db:migrate` — ConfigLoader alone never flips an
existing install's defaults; the migration is mandatory, per the captain_tasks +
assignment_v2 precedent pair). **Exit criteria, not follow-ups:**
- Docs page: `DISABLE_DELAYED_AUTOMATIONS` (where to set, halts within ~5 min) and the
flag's exact three-layer semantics.
- Upgrade note: "if you run a custom Sidekiq queue list, `scheduled_jobs` **and** `medium`
must be processed or delayed rules never fire" (stock `sidekiq.yml` is titled a *sample*
configuration).
- Expectations paragraph: on strict-priority queues, low-concurrency installs will see
sweep lag; hours-scale delays are the design point.
- The run-once-semantics user-guide article.
**Stage F — Cleanup (2–3 weeks post-GA).**
Keep the controller flag guard (self-hosted admins legitimately toggle account flags).
Cleanup: retire the canary rake if one-off, close CW-7513, re-spec CW-5790 as Phase 2,
record learnings + the Phase 1 baseline bands for Phase 2 gating.
### 4.3 Rollback playbook
| Symptom | Response | Blast radius |
|---|---|---|
| Sweep melting `medium` queue / runaway executions | Set `DISABLE_DELAYED_AUTOMATIONS` → sweep no-ops next tick **and** already-enqueued per-row jobs refuse to execute (switch is checked in both). Pending rows accumulate; the 3-day due-window caps replay on re-enable | Instance-wide stop; immediate path untouched |
| One account's rules misbehaving | Disable the account's `delayed_automations` flag — per-account stop at all three layers (no new arming; armed rows skip with `flag_disabled`). Or deactivate the specific rule | Single account |
| **Re-check bug: rule fires where it shouldn't** | (1) Kill switch. (2) Blast radius = executed rows for that rule in the window (`rule_id, conversation_id, updated_at` — this audit trail is why `skip_reason`/rows are kept 30 days). (3) Reversible actions (labels, assignments): remediation rake driven off that row list. (4) `send_message`: **irreversible** — comms template + a named support owner; this is why delayed sends stay dogfood/canary-only until Stage D | Bounded by the executed-rows list |
| Bad migration / schema issue at Stage A | Standard revert; `execution_delay` is nullable with no readers when dark | Deploy-level |
An explicit non-rollback: **never delete the flag or reorder features.yml** — bit positions
are permanent (header contract in features.yml).
## 5. Engine guardrails (folded into the implementation plan)
> These originated as review amendments; the schema/scope/spec deltas are now **written into
> `docs/delayed-automations-implementation-plan.md`** so the PRs are built from one source of
> truth. This section keeps the rationale and the Stage-A-blocking split.
**Stage-A-blocking** (the sweep is unsafe without them):
- **Bounded due-window**: sweep selects `due_at: 3.days.ago..Time.current`
(campaign/snooze-reopen precedent); older rows → `skipped` / `skip_reason: expired`,
count logged — no silent truncation.
- **Per-sweep cap**: constant on the job class with an InstallationConfig override
(Captain ScheduleSyncsJob pattern — gives a no-deploy tuning knob), default 1000;
overflow logs `capped: true` + remaining count; due rows stay due for the next tick.
- **Claim transition**: the sweep only enqueues; the per-row job does the atomic
`pending → processing` claim under a row lock, so a row re-enqueued by an overlapping sweep
(or a reclaimed stale row) loses the claim and can't double-fire. Stale `processing` rows
older than 15 min become claimable again (mechanism per Captain's stale-claim recovery — its
`SYNC_STALE_TIMEOUT` is 2h; we choose 15 min to match the 5-min cadence).
- **Kill switch checked in both jobs** (§4.1).
- **Queue reality** (`config/sidekiq.yml` is strict-priority, no weights): per-row jobs on
`medium` (3rd) preempt `default` and below — the cap is what protects the instance;
`scheduled_jobs` (8th) sits *below* `low`, so the sweep itself can be late — `due_at <=
now` semantics already tolerate that. Note `WebhookJob` is **also** `queue_as :medium`,
so webhook-heavy delayed rules add to the same queue; covered by the cap.
- Per-row error isolation: `discard_on ActiveJob::DeserializationError`; rescue →
`ChatwootExceptionTracker.new(e, account:).capture_exception`, continue.
**Fast-follow tolerable** (days, not weeks; needed before Stage C):
- `skip_reason` terminal values on every skip path (`rule_inactive | conversation_gone |
flag_disabled | episode_moved | conditions_changed | expired`).
- Structured end-of-run summary log + dashboards (§6).
**Unchanged from the implementation plan** (called out because review probed them):
- Migration shapes: plain nullable no-default `add_column` on `conversations`; concurrent
indexes; retention pruning batched under the global 14s `statement_timeout`.
- Email actions keep `within_email_rate_limit?` via ActionService reuse — verify in specs,
don't reimplement.
## 6. Observability, metrics, alerts
**Structured summary log per sweep**, emitted as JSON so New Relic ingests fields without a
parsing rule:
`[AutomationRules::TriggerPendingExecutionsJob] {"event":"completed","enqueued":N,
"capped":false,"purged":N,"duration_ms":N}` — plus per-row terminal
`skip_reason` stored on the row (the support-facing answer to "why didn't my rule fire"
until Phase 2's history UI).
**Alerts are pre-Stage-A tasks with owners, or they don't exist.** Each needs its NRQL/Sentry
rule created, a notification channel, and a named acknowledger for Jul 10–22 (§7 decision):
| Alert | Mechanism | Owner / channel |
|---|---|---|
| Sweep summary absent > 15 min | NR loss-of-signal on `event:'completed'` from the job class | *fill at §7 sign-off* |
| `expired > 0` sustained (2+ ticks) | NRQL on the JSON field |〃 |
| Pending-row count growing across 6 ticks | NRQL on `due` | 〃 |
| New Sentry group under the two job classes | Sentry issue alert filtered by job-class tag | 〃 |
Stage A exit includes a live end-to-end test of each alert (test trigger → channel).
**Dashboards** (New Relic, existing log forwarding + newrelic-sidekiq-metrics):
fire lag (`executed_at - due_at` p50/p95; target p95 < 10 min) · outcome mix by
`skip_reason` (high `conditions_changed` is *healthy* — cancelled follow-ups working; any
`expired` means starvation) · pending count + oldest-pending age · `medium` and
`scheduled_jobs` queue latency.
**Numeric gates** (replacing fuzzy language): Stage B go/no-go and Stage D pause criteria in
§4.2; the week-1 canary executed:skipped mix is *recorded as the baseline band* — later
deviations from that band, not an arbitrary number, are the anomaly signal.
**Success metrics (product):** accounts with ≥1 delayed rule (adoption), delayed executions
per day, executed:skipped ratio vs baseline, and the CW-7513 customer's three scenarios
confirmed by them.
## 7. Decisions needed (owners; lock before Stage A — i.e., today/tomorrow)
1. **Flag name sign-off** (eng, Tanmay): `delayed_automations` on `feature_flags_ext_1` —
~~resolved~~: the ext column shipped on develop (#14947), so this is a plain features.yml
append; the name/position freezes on merge. Non-blocking beyond PR review.
2. **Premium or not** (product, Pranav/Sony): recommendation **non-premium** — base
`automations` is free, engine cost is self-limiting (no delayed rules → zero rows), and
Phases 2–3 carry natural premium hooks (business-hours accounting alongside SLA, SLA
escalation). Plan-gating later is one `ReconcilePlanFeaturesService` change.
3. **Named reviewers + review SLA for PR 1/2** (eng leads): Friday EOD merge deadline makes
this load-bearing.
4. **Alert ownership Jul 10–22** (eng/on-call): who receives and acks §6's four alerts.
5. **Weekend deploy policy** (infra): confirms whether Stage B starts Friday evening or
Monday.
6. **Delay bounds** (product): 10 min–30 days per the implementation plan — confirm.
7. **Canary cohort** (product/support): CW-7513 customer + CW-5790 attached accounts +
design partners; confirm the white-glove session on Jul 15.
8. **Phase 2 business-hours placement** (product, at Phase 2 spec): OSS gate + premium
accounting, or all-OSS. No schema impact either way.
## 8. Timeline
| Date | Milestone |
|---|---|
| Thu Jul 9 | This plan signed off; §7 decisions 1–5 locked; PR 1 (engine+flag, dark) in review |
| Fri Jul 10 | PR 2 (UI behind flag); **both merged EOD** (named-reviewer SLA); deploy; Stage B flag on for internal accounts |
| Sat–Sun Jul 11–12 | Slack for merge slip; staging compressed-delay cycles running; Saturday-morning sweep watch (named) |
| Mon Jul 13 | First full business day of dogfood signal |
| Tue Jul 14 | Go/no-go on §4.2 Stage B gates |
| **Wed Jul 15** | **Stage C canary — CW-7513 delivered** (white-glove setup with the requesting customer; degenerate path: console-created rules if PR 2 slipped) |
| Jul 22+ | Stage D Cloud GA — earliest date, entry-gated on canary evidence; waves per §4.2 + new-account default flip |
| Next tagged release | Stage E self-hosted GA (checklist in §4.2 are exit criteria) |
| +2–3 weeks | Stage F cleanup; Phase 2 spec kickoff (re-scope CW-5790; carries the arming-scan decision + business-hours placement) |
| ~Sep | Phase 2 ship (time vocabulary + business hours + run history) |
| Q4 2026 | Phase 3 (chains + SLA escalation) |
| 2027 | Phase 4 spec (scheduled triggers / builder) |