Help-center onboarding jobs were getting stuck in the `generating` state
indefinitely. `Onboarding::HelpCenterArticleWriterJob` only finalized
the generation counter for two exception types
(`Firecrawl::FirecrawlError`, `ArticleBuildFailed`). Any other error
exhausted Sidekiq's default retries and landed in the dead set without
ever bumping `finished`, leaving state at `total - 1` / `generating`
until the 7-day Redis TTL expired. Observed in production: 4 generations
wedged at exactly `finished = total - 1`, four days after the run, with
no further progress — the onboarding UI showed "generating" the whole
time.
This PR adds a `discard_on StandardError` catch-all to
`HelpCenterArticleWriterJob`, after the existing `retry_on
Firecrawl::FirecrawlError` and `discard_on ArticleBuildFailed` handlers.
ActiveJob matches in declaration order, first match wins, so existing
behavior is unchanged — the catch-all only absorbs errors that
previously fell through to unhandled retry-then-dead-set. It routes
through the same `on_writer_failure` → `finalize` path, so state always
progresses to `completed`.
<details><summary>
##### Script to remove dead jobs
</summary>
<p>
```rb
# One-off: unstick Onboarding::HelpCenterGenerationState keys that are wedged in
# "generating" because a writer job died on an unhandled exception (neither
# FirecrawlError nor ArticleBuildFailed) and exhausted Sidekiq retries without
# ever calling record_article_finished.
#
# These portals have real articles (finished is 1 short of total). Marking them
# "completed" is the honest terminal state: generation is done, one article failed.
#
# Run on a prod box (dry-run first, then REMOVE_DRY_RUN=1):
# RAILS_ENV=production bundle exec rails runner scripts/help_center_investigation/unstick_generating.rb
#
# To actually write, set REMOVE_DRY_RUN=1 in the environment.
pattern = format(Redis::Alfred::HELP_CENTER_GENERATION, id: '*')
dry_run = ENV['REMOVE_DRY_RUN'].blank?
stuck = []
Redis::Alfred.with do |conn|
conn.scan_each(match: pattern, count: 1000) do |key|
h = conn.hgetall(key)
next unless h['status'] == 'generating'
gen_id = key.sub('HELP_CENTER_GENERATION::', '')
stuck << { gen_id: gen_id, total: h['total'], finished: h['finished'], key: key }
unless dry_run
conn.hset(key, 'status', 'completed')
conn.expire(key, Onboarding::HelpCenterGenerationState::TTL)
end
end
end
puts "#{dry_run ? '[DRY RUN] ' : ''}Found #{stuck.size} stuck 'generating' states:"
stuck.each do |s|
puts " gen=#{s[:gen_id]} total=#{s[:total]} finished=#{s[:finished]} -> #{dry_run ? 'would mark completed' : 'marked completed'}"
end
puts
puts 'Re-run with REMOVE_DRY_RUN=1 to apply.' if dry_run
````
</p>
</details>