akadmin/chatwoot
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

chore: remove the guides

Browse Source
This commit is contained in:
Muhsin
2026-03-04 20:38:11 +04:00
parent c3606d22ea
commit 63abd18bde
3 changed files with 0 additions and 459 deletions
Download Patch File Download Diff File Expand all files Collapse all files
captain-shopify-gap-competitor-plan.md
-128
View File
@@ -1,128 +0,0 @@
# Captain + Shopify Gap Analysis and Competitor Comparison Plan
## 1. Purpose
Create a clear plan to:
1. Identify product and technical gaps in current Captain + Shopify experience.
2. Compare current behavior with leading support/commerce competitors.
3. Convert findings into a prioritized implementation roadmap.
## 2. Current Baseline (Chatwoot)
Current implemented scope:
1. Shopify product search via Captain tool.
2. Shopify order lookup by contact email/phone.
3. Captain V2 + Shopify connection gating.
4. Fallback identity extraction from conversation text/history.
Out of scope in current version:
1. Order lookup by order ID.
2. Checkout/cart workflows.
3. Deep post-purchase workflows (returns/exchanges automation).
4. GraphQL-first Shopify data layer.
## 3. Competitor Set
Primary competitors for comparison:
1. Gorgias (Shopify-first support workflows).
2. Zendesk + Shopify integration.
3. Intercom (Fin + commerce integrations).
4. Tidio / similar SMB Shopify-focused AI support stacks.
Note: This plan focuses on capability and UX patterns, not pricing.
## 4. Comparison Dimensions
Evaluate each competitor and Chatwoot against the same rubric:
1. Identity resolution robustness (email/phone/order context).
2. Order lookup depth (status, fulfillment, tracking, edits).
3. Product intelligence quality (search relevance, detail depth).
4. Agent handoff quality after failed AI attempts.
5. Conversation memory and context carry-forward.
6. Actionability (read-only vs operational actions).
7. Error transparency and recovery UX.
8. Admin visibility (logs, traceability, debugging).
9. Setup friction (scopes, auth, reauthorization handling).
10. Reliability under ambiguous user messages.
## 5. Gap Assessment Method
### Step 1: Baseline test scenarios (Chatwoot)
Run a fixed scenario pack:
1. Product availability questions.
2. Specific product detail follow-up.
3. Order status lookup with:
- email present in contact
- email only in user message
- confirmation-only follow-up (`Yes`)
- phone-only identity
4. No-result and provider-error paths.
### Step 2: Competitor behavior capture
For each scenario:
1. Capture response quality.
2. Capture number of turns to resolution.
3. Capture failure behavior and escalation quality.
### Step 3: Scorecard
Use 0-3 scoring per dimension:
1. `0` = missing
2. `1` = weak/manual
3. `2` = functional
4. `3` = strong/best-in-class
## 6. Initial Gap Hypotheses
Likely current gaps to validate:
1. Order retrieval flexibility:
- No order-id lookup path yet.
2. Product detail richness:
- Limited details when Shopify product metadata is sparse.
3. Proactive recovery:
- AI asks for identity but does not persist verified identity for next turns.
4. Action workflows:
- No built-in operational actions (refund/cancel/update address/escalate with structured payload).
5. Observability UX:
- Logs exist, but no dedicated operational dashboard for Shopify tool outcomes.
## 7. Roadmap Framework (Post-Analysis)
### Phase A (Quick Wins)
1. Improve prompt/tool guidance for identity confirmation.
2. Add operator-facing diagnostics for identity source and lookup failure reason.
3. Expand product response template quality.
### Phase B (Core Capability)
1. Add optional order lookup by order ID (feature-flagged).
2. Persist verified contact identity with safeguards.
3. Add richer order timeline output (payment, fulfillment, tracking where available).
### Phase C (Strategic)
1. Move Shopify reads to GraphQL Admin API.
2. Add structured post-purchase workflows.
3. Add KPI dashboard for resolution rate and lookup success rate.
## 8. Success Metrics
Track before/after:
1. First-response resolution rate for order status requests.
2. Median turns to order lookup completion.
3. Percentage of order lookups failing with missing identifier.
4. Product query relevance satisfaction (manual QA scoring).
5. Human handoff rate for Shopify intents.
## 9. Deliverables
1. Gap matrix (Chatwoot vs each competitor by dimension).
2. Ranked backlog of gaps with impact and effort.
3. Implementation proposal with phased milestones.
4. QA plan for each planned feature.
## 10. Execution Checklist
1. Finalize scenario pack.
2. Run Chatwoot baseline and record outputs.
3. Run competitor comparisons and record outputs.
4. Complete scoring matrix.
5. Convert top gaps into engineering tickets.
6. Validate each implemented item against the same scenario pack.
captain-shopify-upgrade-guide.md
-180
View File
@@ -1,180 +0,0 @@
# Captain + Shopify Upgrade — Investigation and Implementation
## 1. Objective
Document the investigation, fixes, and validation steps for Shopify-powered Captain flows in Chatwoot, focused on:
- Product search reliability
- Order lookup reliability
- Runtime observability (logs)
This guide reflects the final state where **order lookup is by contact email/phone only** (not by order ID).
## 2. Reported Symptoms
### Product flow
- Captain returned incomplete/incorrect product answers for known products.
- Shopify REST API warning was observed for `products.json` deprecation.
### Order flow
- Customer had a valid Shopify order visible in UI.
- Captain still responded with:
- `I need the contact email or phone number...`
- Logs showed:
- `shopify_get_orders_identity_resolved ... email_present: false, phone_present: false`
- `missing_identifier`
## 3. Root Cause
For order lookup, Captain tool identity was initially resolved from `state[:contact]` only.
- In the failing conversation, contact state had:
- `contact.email = nil`
- `contact.phone_number = nil`
- User shared email in message text, but tool did not use message text/history for identity.
## 4. Final Implementation Summary
## 4.1 Product search improvements
- Added stronger logging around Shopify client initialization and product search.
- Added fallback matching strategy in product search when strict title filtering produced empty results.
## 4.2 Order lookup improvements (final behavior)
- Kept Shopify order lookup API in service as:
- `orders_for_contact(email:, phone_number:, limit:)`
- Added robust identity resolution for Captain order tool:
- Explicit tool args (if provided by model)
- Contact state
- Current user input text (`captain_v2_trace_current_input`)
- User trace history (`captain_v2_trace_input`) for confirmation turns like `Yes`
- Final outcome: order lookup remains **email/phone based**, but now survives missing contact fields when identity exists in conversation text.
## 4.3 Error handling and UX
- Missing identifier message is:
- `I need the contact email or phone number to look up Shopify orders.`
- Domain/provider errors are mapped to safe user-facing messages.
## 4.4 Prompt guidance
- Captain prompt guidance for Shopify order queries remains generic for order lookup via Shopify order tool.
## 5. Key Files
- Service:
- `app/services/integrations/shopify/client_service.rb`
- `app/services/integrations/shopify/products_service.rb`
- `app/services/integrations/shopify/orders_service.rb`
- Captain tools:
- `enterprise/lib/captain/tools/shopify_base_tool.rb`
- `enterprise/lib/captain/tools/shopify_search_products_tool.rb`
- `enterprise/lib/captain/tools/shopify_get_orders_tool.rb`
- Prompt:
- `enterprise/lib/captain/prompts/assistant.liquid`
- Specs:
- `spec/services/integrations/shopify/products_service_spec.rb`
- `spec/services/integrations/shopify/orders_service_spec.rb`
- `spec/enterprise/lib/captain/tools/shopify_search_products_tool_spec.rb`
- `spec/enterprise/lib/captain/tools/shopify_base_tool_spec.rb`
- `spec/enterprise/lib/captain/tools/shopify_get_orders_tool_spec.rb`
## 6. QA Checklist
### Product search
1. Ask for a known product keyword (example: `Do you have snowboard?`).
2. Verify Captain returns matching Shopify products with links.
3. Check logs for:
- `shopify_search_products_requested`
- `ProductsService ... search_products`
- fallback log path when strict title match is empty
### Order lookup (email/phone)
1. Ensure contact sidebar may have empty email/phone.
2. Ask: `Where is my order russel.winfield@example.com?`
3. Verify Captain resolves identity and attempts Shopify lookup.
4. Confirmation flow:
- User provides email in earlier turn
- Later replies only `Yes`
- Verify tool still resolves email from trace history and proceeds
5. Check logs for:
- `shopify_get_orders_requested`
- `shopify_get_orders_identity_resolved` with `email_present: true` or `phone_present: true`
- no `missing_identifier` when identity exists in message/history
## 7. Spec/Lint Commands Used
```bash
bundle exec rspec spec/services/integrations/shopify/orders_service_spec.rb \
spec/enterprise/lib/captain/tools/shopify_base_tool_spec.rb \
spec/enterprise/lib/captain/tools/shopify_get_orders_tool_spec.rb
```
```bash
bundle exec rubocop \
app/services/integrations/shopify/orders_service.rb \
enterprise/lib/captain/tools/shopify_base_tool.rb \
enterprise/lib/captain/tools/shopify_get_orders_tool.rb \
spec/services/integrations/shopify/orders_service_spec.rb \
spec/enterprise/lib/captain/tools/shopify_get_orders_tool_spec.rb
```
## 8. Known Non-Blocking Noise
- Sidekiq scheduled `Discord::PollAllChannelsJob` may fail with:
- `NameError: uninitialized constant Discord`
- This is unrelated to Shopify Captain product/order behavior.
## 9. Rollback Guidance
If required, rollback can be done by reverting Shopify-related tool/service commits only:
- Revert the Shopify service/tool/spec files listed in Section 5.
- Re-run the spec and rubocop commands in Section 7.
## 10. Missed in This Iteration (Planned for Next Version)
The following items were identified during implementation/testing but intentionally deferred:
1. Shopify order lookup by order ID (optional mode)
- We validated this path technically, but final behavior in this release is email/phone-only.
- Next version plan:
- Add `order_id` lookup behind a feature flag.
- Keep email/phone as default to avoid accidental misrouting.
2. Contact identity persistence from conversation
- Current fix resolves identity from message/trace at runtime.
- Next version plan:
- Optionally persist verified email/phone back to contact profile (with guardrails), so subsequent turns need fewer recoveries.
3. Shopify API modernization (REST -> GraphQL)
- Logs still show REST deprecation warnings for product endpoints.
- Next version plan:
- Move product/order reads to Shopify GraphQL Admin API.
- Preserve response contracts expected by Captain tools.
4. End-to-end regression suite for conversational identity flows
- Current tests are service + tool unit specs.
- Next version plan:
- Add integration-level specs for real multi-turn conversation flows:
- email in first turn + `Yes` confirmation turn
- phone-only lookup
- empty identity -> prompt for identifier
5. Better operator-facing diagnostics
- Logs are present, but triage still needs manual correlation.
- Next version plan:
- Add structured log keys and dashboards for:
- identity source used (`contact`, `current_input`, `trace_history`)
- tool success/failure reason distribution
- Shopify error categories by account
6. Non-Shopify scheduled job noise
- `Discord::PollAllChannelsJob` `NameError` is unrelated but pollutes logs.
- Next version plan:
- Gate/disable scheduler for disconnected integrations in dev/test to reduce noise during Captain investigations.
shopify_advanced.md
-151
View File
@@ -1,151 +0,0 @@
# Shopify Advanced (Implementation-Aligned Runbook)
## Summary
This document reflects the implemented Shopify support for Captain in Chatwoot:
1. Enterprise-only
2. Captain V2-only (`captain_integration_v2`)
3. Phase 1 scope: product search + order lookup
4. Order lookup is by contact identity (`email` / `phone_number`) only
No Captain V1 behavior and no checkout/cart features are included.
## Implemented Tool Surface
1. `shopify_search_products`
2. `shopify_get_orders`
Both tools are registered in:
[`/Users/muhsink/Documents/chatwoot/config/agents/tools.yml`](/Users/muhsink/Documents/chatwoot/config/agents/tools.yml)
## Shopify Scope Requirements
1. Integration helper includes:
- `read_customers`
- `read_orders`
- `read_fulfillments`
- `read_products`
2. `read_products` is required for product search behavior.
Reference:
[`/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb`](/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb)
## Core Files
### Services
1. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/client_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/client_service.rb)
2. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/products_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/products_service.rb)
3. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/orders_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/orders_service.rb)
### Captain tools
1. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_base_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_base_tool.rb)
2. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_search_products_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_search_products_tool.rb)
3. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_get_orders_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_get_orders_tool.rb)
### Gating + prompt
1. [`/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb`](/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb)
2. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid)
## Service Behavior
### `Integrations::Shopify::ClientService`
1. Loads enabled Shopify hook (`app_id: 'shopify'`).
2. Builds Shopify REST admin client.
3. Parses granted scopes from hook settings.
4. Returns normalized results:
- `:not_connected`
- `:provider_error`
5. Emits structured Rails logs for connection and scope context.
### `Integrations::Shopify::ProductsService`
Method:
- `search_products(query:, limit: 10)`
Behavior:
1. Rejects blank query with `:no_results`.
2. Validates connection and `read_products` scope.
3. Primary search: active products by title.
4. Fallback search: fetch active products with larger limit, then in-memory keyword match against title/vendor/type.
5. Normalizes product payload:
- `id`, `title`, `vendor`, `product_type`, `handle`, `storefront_url`
- first variant `price`
- computed `availability` summary
6. Error mapping:
- `:insufficient_scope`, `:no_results`, `:provider_error`
### `Integrations::Shopify::OrdersService`
Method:
- `orders_for_contact(email:, phone_number:, limit: 10)`
Behavior:
1. Requires at least one identifier (`email` or `phone_number`).
2. Validates connection and scopes `read_customers` + `read_orders`.
3. Runs customer search using `email OR phone`.
4. Fetches orders for first matching customer.
5. Normalizes order payload:
- `id`, `name`, `created_at`, `total_price`, `currency`
- `financial_status`, `fulfillment_status`
- `line_items` (capped)
- `admin_url`
6. Error mapping:
- `:missing_identifier`, `:insufficient_scope`, `:no_results`, `:provider_error`
## Tool Behavior
### Base tool (`Captain::Tools::ShopifyBaseTool`)
1. `active?` requires:
- account feature flag `captain_integration_v2`
- enabled Shopify integration hook
2. Common deterministic domain error formatting.
3. Identity resolution order for order lookup:
- explicit tool args
- `state.contact`
- `state.captain_v2_trace_current_input` (message text parsing)
- `state.captain_v2_trace_input` history (supports confirmation turns like `Yes`)
### `shopify_search_products`
Params:
1. `query` (required)
Behavior:
1. Calls `ProductsService#search_products(query:, limit: 10)`.
2. Returns compact text list with title, price, availability, and product URL.
3. Returns safe fallback messages on domain/provider errors.
### `shopify_get_orders`
Params:
1. `email` (optional)
2. `phone_number` (optional)
Behavior:
1. Resolves identity through base-tool identity resolution.
2. Calls `OrdersService#orders_for_contact(email:, phone_number:, limit: 10)`.
3. Returns compact text list with order metadata, admin URL, and top line items.
4. Returns deterministic fallback messages for missing identity/no results/scope/provider errors.
## Gating Rules
1. Shopify tools are exposed only when Captain V2 is enabled and Shopify is connected.
2. If tool execution is attempted while ineligible, tools return safe non-crashing messages.
3. Existing `available_tool_ids` validation remains source of truth for scenario/tool validity.
## Current Constraints
1. Order lookup is email/phone-based only in this version.
2. No order-ID lookup path is enabled in this implementation.
3. Product and order responses are plain text lists for Captain tool output.
## Verification Coverage
### Service specs
1. Product search success/no-results/scope/provider-error.
2. Product title-empty fallback keyword flow.
3. Orders success by email and by phone.
4. Orders missing identifier/no customer/provider-error.
### Tool specs
1. Active-state gating checks (`V2 off`, `Shopify disconnected`, eligible case).
2. Product tool success/error messaging.
3. Orders tool success/error messaging.
4. Orders identity extraction from:
- state contact
- current input text
- trace history
- explicit tool arguments overriding inferred values
## Notes for Next Version
1. Optional order lookup by explicit order ID (feature-flagged).
2. Optional verified identity persistence to contact profile.
3. REST-to-GraphQL Shopify migration to address deprecation trajectory.
4. More end-to-end conversation-level regression coverage.