feat: broaden search scope for help center generation (#14880)
**Broaden the Firecrawl `map` search term list so help-center onboarding doesn't skip sites with non-standard docs paths** Help-center onboarding was skipping ~70% of new accounts, and ~60% of those skips came from a single failure: `"map returned no links"`. The root cause was an overly narrow hardcoded search query passed to Firecrawl's `map` endpoint. The curator (`Onboarding::HelpCenterCurator`) calls `Firecrawl.map(url, search: MAP_SEARCH)` to discover candidate pages before the LLM curation step. `MAP_SEARCH` was hardcoded to `"docs help support faq"` — a 4-term list that only matched sites whose help content sat at `/docs`, `/help`, `/support`, or `/faq`. Sites using `/resources`, `/guides`, `/kb`, `/articles`, `/handbook`, `/learn`, `/how-to`, `/tutorial`, `/troubleshooting`, or a docs subdomain found nothing, so the job raised `CurationSkipped` and left the portal empty. Firecrawl's `search` param is a grep-style substring filter across URL, title, and description (not a semantic query), so the fix is to broaden the term list rather than drop it. The LLM curator downstream (`Captain::Llm::HelpCenterCurationService`) already filters returned links by quality — it has the full URL-path-priority prompt and a 25-article hard ceiling — so a wider crawl net is safe and doesn't change the final article quality bar. **What changed** - `enterprise/app/services/onboarding/help_center_curator.rb`: `MAP_SEARCH` broadened from `"docs help support faq"` to a 13-term list covering common help-content path hints. Comment added documenting why the term list exists and that the LLM curator does the real filtering.
This commit is contained in:
@@ -1,6 +1,12 @@
|
||||
class Onboarding::HelpCenterCurator
|
||||
MAP_LIMIT = 500
|
||||
MAP_SEARCH = 'docs help support faq'.freeze
|
||||
# Firecrawl `map` `search` is a substring filter (grep-style) across URL,
|
||||
# title, and description — not a semantic query. The original 4-term list
|
||||
# (`docs help support faq`) missed sites whose help content lives at
|
||||
# non-standard paths, producing ~60% of all onboarding skips via
|
||||
# "map returned no links". Broaden the term list so more paths match; the
|
||||
# LLM curator (HelpCenterCurationService) filters the results by quality.
|
||||
MAP_SEARCH = 'docs help support faq resources guides kb knowledge articles handbook learn tutorial troubleshooting'.freeze
|
||||
MIN_ARTICLES = 3
|
||||
|
||||
Skipped = Onboarding::HelpCenterErrors::CurationSkipped
|
||||
|
||||
Reference in New Issue
Block a user