feat: update prompt to ignore unrelated material

feat: add exclude path
2026-06-25 17:28:44 +05:30 · 2026-06-25 17:28:05 +05:30
2 changed files with 13 additions and 1 deletions
@@ -13,6 +13,8 @@ class Captain::Llm::SystemPromptsService

        **Substance over chrome**: Treat as source content only what is actual product, procedural, conceptual, or factual information. Do not generate FAQs from site chrome — navigation, footer, header, breadcrumbs, cookie banners, search widgets, page metadata, or other interface elements.

+        **Support-relevant content only**: Only generate FAQs from content that helps a customer use, set up, troubleshoot, or understand the product or service. Ignore non-support material even when it is substantive — marketing and sales copy, blog or news posts, press releases, company/about/team pages, careers or job listings, investor relations, testimonials or case studies, and legal boilerplate such as terms of service, privacy policies, and cookie notices. If the page is entirely such content, return `{"faqs": []}`.
+
        **Accuracy**: Base answers strictly on the provided text. Do not add assumptions, interpretations, or external knowledge not present in the source material.

        **Structure**: Format output as valid JSON using this exact structure:
@@ -43,7 +45,7 @@ class Captain::Llm::SystemPromptsService
        1. Read the entire provided content carefully
        2. Identify all key information points: procedures, examples, code, identifiers, limits, definitions, warnings, and explanations
        3. For each candidate section, verify the source contains the substance that would answer the question. If the source only points to where the substance lives, skip the section.
-        4. Disregard interface chrome (navigation, footer, header, cookie banners, breadcrumbs, page metadata).
+        4. Disregard interface chrome (navigation, footer, header, cookie banners, breadcrumbs, page metadata) and non-support pages (marketing, blog/news, company info, legal/privacy boilerplate).
        5. Create questions that cover each remaining substantive information point
        6. Write self-contained answers that preserve all relevant details from the source. Be concise where possible, but never trade away steps, examples, warnings, code, IDs, limits, or definitions for brevity.
        7. Verify the combined FAQs represent the complete substantive source content (excluding redirect-only sections and chrome).
@@ -1,6 +1,15 @@
 class Captain::Tools::FirecrawlService
  BASE_URL = 'https://api.firecrawl.dev/v2'.freeze
  FIRECRAWL_EXCLUDE_TAGS = %w[iframe .sidebar .cookie-banner [role=navigation] [role=banner] [role=contentinfo]].freeze
+  # Pathname regex patterns kept out of the crawl so promotional, legal, and
+  # auth pages do not get indexed as Captain documents (and do not consume quota).
+  FIRECRAWL_EXCLUDE_PATHS = %w[
+    ^/blog ^/news ^/press
+    ^/careers ^/jobs ^/about ^/team ^/investors
+    ^/customers ^/testimonials ^/case-studies
+    ^/login ^/signup ^/register
+    ^/legal ^/terms ^/privacy
+  ].freeze

  def self.configured?
    InstallationConfig.find_by(name: 'CAPTAIN_FIRECRAWL_API_KEY')&.value
@@ -38,6 +47,7 @@ class Captain::Tools::FirecrawlService
      maxDiscoveryDepth: 50,
      sitemap: 'include',
      limit: crawl_limit,
+      excludePaths: FIRECRAWL_EXCLUDE_PATHS,
      webhook: { url: webhook_url },
      scrapeOptions: scrape_options
    }.to_json
Author	SHA1	Message	Date
Shivam Mishra	835fb229da	feat: update prompt to ignore unrelated material	2026-06-25 17:28:44 +05:30
Shivam Mishra	6365af872d	feat: add exclude path	2026-06-25 17:28:05 +05:30