Shivam MishraandGitHub
8670f66155
**Broaden the Firecrawl `map` search term list so help-center onboarding doesn't skip sites with non-standard docs paths** Help-center onboarding was skipping ~70% of new accounts, and ~60% of those skips came from a single failure: `"map returned no links"`. The root cause was an overly narrow hardcoded search query passed to Firecrawl's `map` endpoint. The curator (`Onboarding::HelpCenterCurator`) calls `Firecrawl.map(url, search: MAP_SEARCH)` to discover candidate pages before the LLM curation step. `MAP_SEARCH` was hardcoded to `"docs help support faq"` — a 4-term list that only matched sites whose help content sat at `/docs`, `/help`, `/support`, or `/faq`. Sites using `/resources`, `/guides`, `/kb`, `/articles`, `/handbook`, `/learn`, `/how-to`, `/tutorial`, `/troubleshooting`, or a docs subdomain found nothing, so the job raised `CurationSkipped` and left the portal empty. Firecrawl's `search` param is a grep-style substring filter across URL, title, and description (not a semantic query), so the fix is to broaden the term list rather than drop it. The LLM curator downstream (`Captain::Llm::HelpCenterCurationService`) already filters returned links by quality — it has the full URL-path-priority prompt and a 25-article hard ceiling — so a wider crawl net is safe and doesn't change the final article quality bar. **What changed** - `enterprise/app/services/onboarding/help_center_curator.rb`: `MAP_SEARCH` broadened from `"docs help support faq"` to a 13-term list covering common help-content path hints. Comment added documenting why the term list exists and that the LLM curator does the real filtering.
72 lines
2.2 KiB
Ruby
72 lines
2.2 KiB
Ruby
class Onboarding::HelpCenterCurator
|
|
MAP_LIMIT = 500
|
|
# Firecrawl `map` `search` is a substring filter (grep-style) across URL,
|
|
# title, and description — not a semantic query. The original 4-term list
|
|
# (`docs help support faq`) missed sites whose help content lives at
|
|
# non-standard paths, producing ~60% of all onboarding skips via
|
|
# "map returned no links". Broaden the term list so more paths match; the
|
|
# LLM curator (HelpCenterCurationService) filters the results by quality.
|
|
MAP_SEARCH = 'docs help support faq resources guides kb knowledge articles handbook learn tutorial troubleshooting'.freeze
|
|
MIN_ARTICLES = 3
|
|
|
|
Skipped = Onboarding::HelpCenterErrors::CurationSkipped
|
|
|
|
def initialize(account:)
|
|
@account = account
|
|
end
|
|
|
|
def perform
|
|
raise Skipped, 'Firecrawl not configured' unless Firecrawl::Configuration.configured?
|
|
raise Skipped, 'no website url' if website_url.blank?
|
|
|
|
links = discover_links
|
|
raise Skipped, 'map returned no links' if links.empty?
|
|
|
|
plan = curate(links)
|
|
raise Skipped, "only #{plan[:articles].size} articles curated (< #{MIN_ARTICLES} threshold)" if plan[:articles].size < MIN_ARTICLES
|
|
|
|
plan.merge(allowed_urls: extract_urls(links)).deep_stringify_keys
|
|
end
|
|
|
|
private
|
|
|
|
def discover_links
|
|
data = Firecrawl::Configuration.client.map(
|
|
website_url,
|
|
Firecrawl::Models::MapOptions.new(limit: MAP_LIMIT, search: MAP_SEARCH)
|
|
)
|
|
Array(data.links)
|
|
end
|
|
|
|
def extract_urls(links)
|
|
Array(links).filter_map do |link|
|
|
link['url'].presence
|
|
end.uniq
|
|
end
|
|
|
|
def curate(links)
|
|
response = Captain::Llm::HelpCenterCurationService.new(account: @account, links: links).perform
|
|
raise Skipped, "curator LLM error: #{response[:error]}" if response[:error]
|
|
|
|
response[:message] || { categories: [], articles: [] }
|
|
end
|
|
|
|
def website_url
|
|
@website_url ||= with_scheme(custom_attributes_website.presence || brand_info[:domain].presence)
|
|
end
|
|
|
|
def with_scheme(raw)
|
|
return raw if raw.blank?
|
|
|
|
raw.match?(%r{\Ahttps?://}i) ? raw : "https://#{raw}"
|
|
end
|
|
|
|
def custom_attributes_website
|
|
@account.custom_attributes['website']
|
|
end
|
|
|
|
def brand_info
|
|
@brand_info ||= (@account.custom_attributes['brand_info'] || {}).deep_symbolize_keys
|
|
end
|
|
end
|