From 8670f661559fcc0e6abc77de407cc1e3d897ddec Mon Sep 17 00:00:00 2001 From: Shivam Mishra Date: Tue, 30 Jun 2026 14:37:15 +0530 Subject: [PATCH] feat: broaden search scope for help center generation (#14880) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Broaden the Firecrawl `map` search term list so help-center onboarding doesn't skip sites with non-standard docs paths** Help-center onboarding was skipping ~70% of new accounts, and ~60% of those skips came from a single failure: `"map returned no links"`. The root cause was an overly narrow hardcoded search query passed to Firecrawl's `map` endpoint. The curator (`Onboarding::HelpCenterCurator`) calls `Firecrawl.map(url, search: MAP_SEARCH)` to discover candidate pages before the LLM curation step. `MAP_SEARCH` was hardcoded to `"docs help support faq"` — a 4-term list that only matched sites whose help content sat at `/docs`, `/help`, `/support`, or `/faq`. Sites using `/resources`, `/guides`, `/kb`, `/articles`, `/handbook`, `/learn`, `/how-to`, `/tutorial`, `/troubleshooting`, or a docs subdomain found nothing, so the job raised `CurationSkipped` and left the portal empty. Firecrawl's `search` param is a grep-style substring filter across URL, title, and description (not a semantic query), so the fix is to broaden the term list rather than drop it. The LLM curator downstream (`Captain::Llm::HelpCenterCurationService`) already filters returned links by quality — it has the full URL-path-priority prompt and a 25-article hard ceiling — so a wider crawl net is safe and doesn't change the final article quality bar. **What changed** - `enterprise/app/services/onboarding/help_center_curator.rb`: `MAP_SEARCH` broadened from `"docs help support faq"` to a 13-term list covering common help-content path hints. Comment added documenting why the term list exists and that the LLM curator does the real filtering. --- enterprise/app/services/onboarding/help_center_curator.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/enterprise/app/services/onboarding/help_center_curator.rb b/enterprise/app/services/onboarding/help_center_curator.rb index 03ab7e407..501ff500a 100644 --- a/enterprise/app/services/onboarding/help_center_curator.rb +++ b/enterprise/app/services/onboarding/help_center_curator.rb @@ -1,6 +1,12 @@ class Onboarding::HelpCenterCurator MAP_LIMIT = 500 - MAP_SEARCH = 'docs help support faq'.freeze + # Firecrawl `map` `search` is a substring filter (grep-style) across URL, + # title, and description — not a semantic query. The original 4-term list + # (`docs help support faq`) missed sites whose help content lives at + # non-standard paths, producing ~60% of all onboarding skips via + # "map returned no links". Broaden the term list so more paths match; the + # LLM curator (HelpCenterCurationService) filters the results by quality. + MAP_SEARCH = 'docs help support faq resources guides kb knowledge articles handbook learn tutorial troubleshooting'.freeze MIN_ARTICLES = 3 Skipped = Onboarding::HelpCenterErrors::CurationSkipped