feat: broaden search scope for help center generation (#14880)

**Broaden the Firecrawl `map` search term list so help-center onboarding
doesn't skip sites with non-standard docs paths**

Help-center onboarding was skipping ~70% of new accounts, and ~60% of
those skips came from a single failure: `"map returned no links"`. The
root cause was an overly narrow hardcoded search query passed to
Firecrawl's `map` endpoint.

The curator (`Onboarding::HelpCenterCurator`) calls `Firecrawl.map(url,
search: MAP_SEARCH)` to discover candidate pages before the LLM curation
step. `MAP_SEARCH` was hardcoded to `"docs help support faq"` — a 4-term
list that only matched sites whose help content sat at `/docs`, `/help`,
`/support`, or `/faq`. Sites using `/resources`, `/guides`, `/kb`,
`/articles`, `/handbook`, `/learn`, `/how-to`, `/tutorial`,
`/troubleshooting`, or a docs subdomain found nothing, so the job raised
`CurationSkipped` and left the portal empty.

Firecrawl's `search` param is a grep-style substring filter across URL,
title, and description (not a semantic query), so the fix is to broaden
the term list rather than drop it. The LLM curator downstream
(`Captain::Llm::HelpCenterCurationService`) already filters returned
links by quality — it has the full URL-path-priority prompt and a
25-article hard ceiling — so a wider crawl net is safe and doesn't
change the final article quality bar.

**What changed**

- `enterprise/app/services/onboarding/help_center_curator.rb`:
`MAP_SEARCH` broadened from `"docs help support faq"` to a 13-term list
covering common help-content path hints. Comment added documenting why
the term list exists and that the LLM curator does the real filtering.
This commit is contained in:
Shivam Mishra
2026-06-30 14:37:15 +05:30
committed by GitHub
parent 2767bd434b
commit 8670f66155
@@ -1,6 +1,12 @@
class Onboarding::HelpCenterCurator
MAP_LIMIT = 500
MAP_SEARCH = 'docs help support faq'.freeze
# Firecrawl `map` `search` is a substring filter (grep-style) across URL,
# title, and description — not a semantic query. The original 4-term list
# (`docs help support faq`) missed sites whose help content lives at
# non-standard paths, producing ~60% of all onboarding skips via
# "map returned no links". Broaden the term list so more paths match; the
# LLM curator (HelpCenterCurationService) filters the results by quality.
MAP_SEARCH = 'docs help support faq resources guides kb knowledge articles handbook learn tutorial troubleshooting'.freeze
MIN_ARTICLES = 3
Skipped = Onboarding::HelpCenterErrors::CurationSkipped