Files
Shivam MishraandGitHub 8670f66155 feat: broaden search scope for help center generation (#14880)
**Broaden the Firecrawl `map` search term list so help-center onboarding
doesn't skip sites with non-standard docs paths**

Help-center onboarding was skipping ~70% of new accounts, and ~60% of
those skips came from a single failure: `"map returned no links"`. The
root cause was an overly narrow hardcoded search query passed to
Firecrawl's `map` endpoint.

The curator (`Onboarding::HelpCenterCurator`) calls `Firecrawl.map(url,
search: MAP_SEARCH)` to discover candidate pages before the LLM curation
step. `MAP_SEARCH` was hardcoded to `"docs help support faq"` — a 4-term
list that only matched sites whose help content sat at `/docs`, `/help`,
`/support`, or `/faq`. Sites using `/resources`, `/guides`, `/kb`,
`/articles`, `/handbook`, `/learn`, `/how-to`, `/tutorial`,
`/troubleshooting`, or a docs subdomain found nothing, so the job raised
`CurationSkipped` and left the portal empty.

Firecrawl's `search` param is a grep-style substring filter across URL,
title, and description (not a semantic query), so the fix is to broaden
the term list rather than drop it. The LLM curator downstream
(`Captain::Llm::HelpCenterCurationService`) already filters returned
links by quality — it has the full URL-path-priority prompt and a
25-article hard ceiling — so a wider crawl net is safe and doesn't
change the final article quality bar.

**What changed**

- `enterprise/app/services/onboarding/help_center_curator.rb`:
`MAP_SEARCH` broadened from `"docs help support faq"` to a 13-term list
covering common help-content path hints. Comment added documenting why
the term list exists and that the LLM curator does the real filtering.
2026-06-30 14:37:15 +05:30
..