codex/docker-ci-cache-optimization
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Shivam MishraandGitHub
|
8670f66155 |
feat: broaden search scope for help center generation (#14880)
**Broaden the Firecrawl `map` search term list so help-center onboarding doesn't skip sites with non-standard docs paths** Help-center onboarding was skipping ~70% of new accounts, and ~60% of those skips came from a single failure: `"map returned no links"`. The root cause was an overly narrow hardcoded search query passed to Firecrawl's `map` endpoint. The curator (`Onboarding::HelpCenterCurator`) calls `Firecrawl.map(url, search: MAP_SEARCH)` to discover candidate pages before the LLM curation step. `MAP_SEARCH` was hardcoded to `"docs help support faq"` — a 4-term list that only matched sites whose help content sat at `/docs`, `/help`, `/support`, or `/faq`. Sites using `/resources`, `/guides`, `/kb`, `/articles`, `/handbook`, `/learn`, `/how-to`, `/tutorial`, `/troubleshooting`, or a docs subdomain found nothing, so the job raised `CurationSkipped` and left the portal empty. Firecrawl's `search` param is a grep-style substring filter across URL, title, and description (not a semantic query), so the fix is to broaden the term list rather than drop it. The LLM curator downstream (`Captain::Llm::HelpCenterCurationService`) already filters returned links by quality — it has the full URL-path-priority prompt and a 25-article hard ceiling — so a wider crawl net is safe and doesn't change the final article quality bar. **What changed** - `enterprise/app/services/onboarding/help_center_curator.rb`: `MAP_SEARCH` broadened from `"docs help support faq"` to a 13-term list covering common help-content path hints. Comment added documenting why the term list exists and that the LLM curator does the real filtering. |
||
|
Shivam MishraandGitHub
|
d9c07fe2e9 |
feat: expose onboarding help center generation status (#14671)
This PR adds a reload-safe onboarding Help Center generation status path. Previously, generation progress was pushed through ActionCable, which meant the onboarding UI could lose context after a page reload or missed websocket event. The new endpoint exposes the current generation id, raw Redis generation state, and Help Center article/category counts so clients can recover state by fetching from the backend. **What changed** - Added an onboarding Help Center generation status endpoint. - Persisted `help_center_generation_id` when generation is enqueued. - Removed Help Center generation ActionCable broadcasts completely. - Kept generation progress in Redis as the backend source of truth. - Kept Help Center generation out of the onboarding hot path; the existing onboarding controller does not start generation yet. **How to test** 1. Start Help Center generation for an account. 2. Fetch the onboarding generation status endpoint and verify it returns the generation id, Redis state, and article/category counts. 3. Reload the onboarding UI or client state and fetch again to confirm progress can be recovered without relying on websocket events. 4. Verify skipped/completed generation states are reflected from Redis. ## Related PRs - https://github.com/chatwoot/chatwoot/pull/14569 - https://github.com/chatwoot/chatwoot/pull/14568 - https://github.com/chatwoot/chatwoot/pull/14567 - https://github.com/chatwoot/chatwoot/pull/14619 - https://github.com/chatwoot/chatwoot/pull/14565 (Primary onboarding PR) |
||
|
Shivam MishraandGitHub
|
3d20a7b049 |
feat: generate Help Center for Onboarding (#14370)
## Manually triggering help center generation
Open a Rails console (`bundle exec rails console`):
```ruby
account = Account.find(<ACCOUNT_ID>)
user = account.users.first
# Optional: refresh brand info from the customer's website
domain = 'example.com'
result = WebsiteBrandingService.new("noreply@#{domain}").perform
account.update!(
name: result[:title].presence || account.name,
custom_attributes: account.custom_attributes.merge('website' => domain, 'brand_info' => result)
)
# Optional: wipe existing portals so a fresh one is created
account.portals.destroy_all
Onboarding::HelpCenterCreationService.new(account, user).perform
```
Sidekiq must be running — articles are written by
`Onboarding::HelpCenterArticleGenerationJob`. Avoid running on
production; generation calls the LLM provider.
### Generation flow (Happy Path)
```mermaid
sequenceDiagram
autonumber
participant Kickoff as HelpCenterCreationService
participant DB as DB
participant GenJob as HelpCenterArticleGenerationJob
participant Curator as HelpCenterCurator
participant Firecrawl as Firecrawl
participant CuratorLLM as Curation LLM
participant Redis as Redis Progress
participant WriterJob as HelpCenterArticleWriterJob
participant Builder as HelpCenterArticleBuilder
participant WriterLLM as Writer LLM
participant Cable as ActionCable
Kickoff->>DB: Create portal for account<br/>homepage_link=https://chatwoot.com
Kickoff->>DB: Attach brand logo if available
Kickoff->>GenJob: Enqueue generation job<br/>account_id, portal_id, user_id, generation_id
GenJob->>Curator: Curate help center plan
Curator->>Firecrawl: map https://chatwoot.com<br/>search: docs help support faq
Firecrawl-->>Curator: Return discovered links
Curator->>CuratorLLM: Select categories + article plans<br/>from discovered links only
CuratorLLM-->>Curator: Return categories, articles, allowed_urls
GenJob->>DB: Create portal categories
GenJob->>GenJob: Stamp articles with category_id
GenJob->>GenJob: Filter article URLs against allowed_urls
GenJob->>GenJob: Drop articles with no category<br/>or no approved source URLs
GenJob->>Redis: Start progress<br/>status=generating, total=N, finished=0
loop For each approved article
GenJob->>WriterJob: Enqueue writer job<br/>title, category_id, approved URLs
end
par Writer jobs run independently
WriterJob->>Builder: Build article from approved URLs
Builder->>Firecrawl: batch_scrape approved URLs
Firecrawl-->>Builder: Return Markdown source pages
Builder->>WriterLLM: Rewrite sources into one article
WriterLLM-->>Builder: Return title, description, Markdown content
Builder->>DB: Create draft portal article<br/>meta.source_urls
WriterJob->>Redis: Increment finished count
WriterJob->>Cable: Broadcast help_center.article_generated
end
WriterJob->>Redis: If finished >= total<br/>mark status=completed
WriterJob->>Cable: Broadcast help_center.generation_completed
```
### Redis State Management
```mermaid
stateDiagram-v2
[*] --> active_pointer_set
active_pointer_set --> generating: generation job creates valid plan
active_pointer_set --> skipped: curation skipped/failed
generating --> generating: each writer job increments finished
generating --> completed: finished == total
generating --> ignored_completion: generation_id superseded
skipped --> [*]
completed --> [*]
ignored_completion --> [*]
```
|