From 63abd18bdeacd1093fbe6a126cc162cc836c607e Mon Sep 17 00:00:00 2001 From: Muhsin Date: Wed, 4 Mar 2026 20:38:11 +0400 Subject: [PATCH] chore: remove the guides --- captain-shopify-gap-competitor-plan.md | 128 ------------------ captain-shopify-upgrade-guide.md | 180 ------------------------- shopify_advanced.md | 151 --------------------- 3 files changed, 459 deletions(-) delete mode 100644 captain-shopify-gap-competitor-plan.md delete mode 100644 captain-shopify-upgrade-guide.md delete mode 100644 shopify_advanced.md diff --git a/captain-shopify-gap-competitor-plan.md b/captain-shopify-gap-competitor-plan.md deleted file mode 100644 index 1b31d5330..000000000 --- a/captain-shopify-gap-competitor-plan.md +++ /dev/null @@ -1,128 +0,0 @@ -# Captain + Shopify Gap Analysis and Competitor Comparison Plan - -## 1. Purpose - -Create a clear plan to: -1. Identify product and technical gaps in current Captain + Shopify experience. -2. Compare current behavior with leading support/commerce competitors. -3. Convert findings into a prioritized implementation roadmap. - -## 2. Current Baseline (Chatwoot) - -Current implemented scope: -1. Shopify product search via Captain tool. -2. Shopify order lookup by contact email/phone. -3. Captain V2 + Shopify connection gating. -4. Fallback identity extraction from conversation text/history. - -Out of scope in current version: -1. Order lookup by order ID. -2. Checkout/cart workflows. -3. Deep post-purchase workflows (returns/exchanges automation). -4. GraphQL-first Shopify data layer. - -## 3. Competitor Set - -Primary competitors for comparison: -1. Gorgias (Shopify-first support workflows). -2. Zendesk + Shopify integration. -3. Intercom (Fin + commerce integrations). -4. Tidio / similar SMB Shopify-focused AI support stacks. - -Note: This plan focuses on capability and UX patterns, not pricing. - -## 4. Comparison Dimensions - -Evaluate each competitor and Chatwoot against the same rubric: -1. Identity resolution robustness (email/phone/order context). -2. Order lookup depth (status, fulfillment, tracking, edits). -3. Product intelligence quality (search relevance, detail depth). -4. Agent handoff quality after failed AI attempts. -5. Conversation memory and context carry-forward. -6. Actionability (read-only vs operational actions). -7. Error transparency and recovery UX. -8. Admin visibility (logs, traceability, debugging). -9. Setup friction (scopes, auth, reauthorization handling). -10. Reliability under ambiguous user messages. - -## 5. Gap Assessment Method - -### Step 1: Baseline test scenarios (Chatwoot) -Run a fixed scenario pack: -1. Product availability questions. -2. Specific product detail follow-up. -3. Order status lookup with: -- email present in contact -- email only in user message -- confirmation-only follow-up (`Yes`) -- phone-only identity -4. No-result and provider-error paths. - -### Step 2: Competitor behavior capture -For each scenario: -1. Capture response quality. -2. Capture number of turns to resolution. -3. Capture failure behavior and escalation quality. - -### Step 3: Scorecard -Use 0-3 scoring per dimension: -1. `0` = missing -2. `1` = weak/manual -3. `2` = functional -4. `3` = strong/best-in-class - -## 6. Initial Gap Hypotheses - -Likely current gaps to validate: -1. Order retrieval flexibility: -- No order-id lookup path yet. -2. Product detail richness: -- Limited details when Shopify product metadata is sparse. -3. Proactive recovery: -- AI asks for identity but does not persist verified identity for next turns. -4. Action workflows: -- No built-in operational actions (refund/cancel/update address/escalate with structured payload). -5. Observability UX: -- Logs exist, but no dedicated operational dashboard for Shopify tool outcomes. - -## 7. Roadmap Framework (Post-Analysis) - -### Phase A (Quick Wins) -1. Improve prompt/tool guidance for identity confirmation. -2. Add operator-facing diagnostics for identity source and lookup failure reason. -3. Expand product response template quality. - -### Phase B (Core Capability) -1. Add optional order lookup by order ID (feature-flagged). -2. Persist verified contact identity with safeguards. -3. Add richer order timeline output (payment, fulfillment, tracking where available). - -### Phase C (Strategic) -1. Move Shopify reads to GraphQL Admin API. -2. Add structured post-purchase workflows. -3. Add KPI dashboard for resolution rate and lookup success rate. - -## 8. Success Metrics - -Track before/after: -1. First-response resolution rate for order status requests. -2. Median turns to order lookup completion. -3. Percentage of order lookups failing with missing identifier. -4. Product query relevance satisfaction (manual QA scoring). -5. Human handoff rate for Shopify intents. - -## 9. Deliverables - -1. Gap matrix (Chatwoot vs each competitor by dimension). -2. Ranked backlog of gaps with impact and effort. -3. Implementation proposal with phased milestones. -4. QA plan for each planned feature. - -## 10. Execution Checklist - -1. Finalize scenario pack. -2. Run Chatwoot baseline and record outputs. -3. Run competitor comparisons and record outputs. -4. Complete scoring matrix. -5. Convert top gaps into engineering tickets. -6. Validate each implemented item against the same scenario pack. diff --git a/captain-shopify-upgrade-guide.md b/captain-shopify-upgrade-guide.md deleted file mode 100644 index 22f091016..000000000 --- a/captain-shopify-upgrade-guide.md +++ /dev/null @@ -1,180 +0,0 @@ -# Captain + Shopify Upgrade — Investigation and Implementation - -## 1. Objective - -Document the investigation, fixes, and validation steps for Shopify-powered Captain flows in Chatwoot, focused on: - -- Product search reliability -- Order lookup reliability -- Runtime observability (logs) - -This guide reflects the final state where **order lookup is by contact email/phone only** (not by order ID). - -## 2. Reported Symptoms - -### Product flow - -- Captain returned incomplete/incorrect product answers for known products. -- Shopify REST API warning was observed for `products.json` deprecation. - -### Order flow - -- Customer had a valid Shopify order visible in UI. -- Captain still responded with: - - `I need the contact email or phone number...` -- Logs showed: - - `shopify_get_orders_identity_resolved ... email_present: false, phone_present: false` - - `missing_identifier` - -## 3. Root Cause - -For order lookup, Captain tool identity was initially resolved from `state[:contact]` only. - -- In the failing conversation, contact state had: - - `contact.email = nil` - - `contact.phone_number = nil` -- User shared email in message text, but tool did not use message text/history for identity. - -## 4. Final Implementation Summary - -## 4.1 Product search improvements - -- Added stronger logging around Shopify client initialization and product search. -- Added fallback matching strategy in product search when strict title filtering produced empty results. - -## 4.2 Order lookup improvements (final behavior) - -- Kept Shopify order lookup API in service as: - - `orders_for_contact(email:, phone_number:, limit:)` -- Added robust identity resolution for Captain order tool: - - Explicit tool args (if provided by model) - - Contact state - - Current user input text (`captain_v2_trace_current_input`) - - User trace history (`captain_v2_trace_input`) for confirmation turns like `Yes` -- Final outcome: order lookup remains **email/phone based**, but now survives missing contact fields when identity exists in conversation text. - -## 4.3 Error handling and UX - -- Missing identifier message is: - - `I need the contact email or phone number to look up Shopify orders.` -- Domain/provider errors are mapped to safe user-facing messages. - -## 4.4 Prompt guidance - -- Captain prompt guidance for Shopify order queries remains generic for order lookup via Shopify order tool. - -## 5. Key Files - -- Service: - - `app/services/integrations/shopify/client_service.rb` - - `app/services/integrations/shopify/products_service.rb` - - `app/services/integrations/shopify/orders_service.rb` -- Captain tools: - - `enterprise/lib/captain/tools/shopify_base_tool.rb` - - `enterprise/lib/captain/tools/shopify_search_products_tool.rb` - - `enterprise/lib/captain/tools/shopify_get_orders_tool.rb` -- Prompt: - - `enterprise/lib/captain/prompts/assistant.liquid` -- Specs: - - `spec/services/integrations/shopify/products_service_spec.rb` - - `spec/services/integrations/shopify/orders_service_spec.rb` - - `spec/enterprise/lib/captain/tools/shopify_search_products_tool_spec.rb` - - `spec/enterprise/lib/captain/tools/shopify_base_tool_spec.rb` - - `spec/enterprise/lib/captain/tools/shopify_get_orders_tool_spec.rb` - -## 6. QA Checklist - -### Product search - -1. Ask for a known product keyword (example: `Do you have snowboard?`). -2. Verify Captain returns matching Shopify products with links. -3. Check logs for: - - `shopify_search_products_requested` - - `ProductsService ... search_products` - - fallback log path when strict title match is empty - -### Order lookup (email/phone) - -1. Ensure contact sidebar may have empty email/phone. -2. Ask: `Where is my order russel.winfield@example.com?` -3. Verify Captain resolves identity and attempts Shopify lookup. -4. Confirmation flow: - - User provides email in earlier turn - - Later replies only `Yes` - - Verify tool still resolves email from trace history and proceeds -5. Check logs for: - - `shopify_get_orders_requested` - - `shopify_get_orders_identity_resolved` with `email_present: true` or `phone_present: true` - - no `missing_identifier` when identity exists in message/history - -## 7. Spec/Lint Commands Used - -```bash -bundle exec rspec spec/services/integrations/shopify/orders_service_spec.rb \ - spec/enterprise/lib/captain/tools/shopify_base_tool_spec.rb \ - spec/enterprise/lib/captain/tools/shopify_get_orders_tool_spec.rb -``` - -```bash -bundle exec rubocop \ - app/services/integrations/shopify/orders_service.rb \ - enterprise/lib/captain/tools/shopify_base_tool.rb \ - enterprise/lib/captain/tools/shopify_get_orders_tool.rb \ - spec/services/integrations/shopify/orders_service_spec.rb \ - spec/enterprise/lib/captain/tools/shopify_get_orders_tool_spec.rb -``` - -## 8. Known Non-Blocking Noise - -- Sidekiq scheduled `Discord::PollAllChannelsJob` may fail with: - - `NameError: uninitialized constant Discord` -- This is unrelated to Shopify Captain product/order behavior. - -## 9. Rollback Guidance - -If required, rollback can be done by reverting Shopify-related tool/service commits only: - -- Revert the Shopify service/tool/spec files listed in Section 5. -- Re-run the spec and rubocop commands in Section 7. - -## 10. Missed in This Iteration (Planned for Next Version) - -The following items were identified during implementation/testing but intentionally deferred: - -1. Shopify order lookup by order ID (optional mode) -- We validated this path technically, but final behavior in this release is email/phone-only. -- Next version plan: - - Add `order_id` lookup behind a feature flag. - - Keep email/phone as default to avoid accidental misrouting. - -2. Contact identity persistence from conversation -- Current fix resolves identity from message/trace at runtime. -- Next version plan: - - Optionally persist verified email/phone back to contact profile (with guardrails), so subsequent turns need fewer recoveries. - -3. Shopify API modernization (REST -> GraphQL) -- Logs still show REST deprecation warnings for product endpoints. -- Next version plan: - - Move product/order reads to Shopify GraphQL Admin API. - - Preserve response contracts expected by Captain tools. - -4. End-to-end regression suite for conversational identity flows -- Current tests are service + tool unit specs. -- Next version plan: - - Add integration-level specs for real multi-turn conversation flows: - - email in first turn + `Yes` confirmation turn - - phone-only lookup - - empty identity -> prompt for identifier - -5. Better operator-facing diagnostics -- Logs are present, but triage still needs manual correlation. -- Next version plan: - - Add structured log keys and dashboards for: - - identity source used (`contact`, `current_input`, `trace_history`) - - tool success/failure reason distribution - - Shopify error categories by account - -6. Non-Shopify scheduled job noise -- `Discord::PollAllChannelsJob` `NameError` is unrelated but pollutes logs. -- Next version plan: - - Gate/disable scheduler for disconnected integrations in dev/test to reduce noise during Captain investigations. diff --git a/shopify_advanced.md b/shopify_advanced.md deleted file mode 100644 index 46b148ae2..000000000 --- a/shopify_advanced.md +++ /dev/null @@ -1,151 +0,0 @@ -# Shopify Advanced (Implementation-Aligned Runbook) - -## Summary -This document reflects the implemented Shopify support for Captain in Chatwoot: -1. Enterprise-only -2. Captain V2-only (`captain_integration_v2`) -3. Phase 1 scope: product search + order lookup -4. Order lookup is by contact identity (`email` / `phone_number`) only - -No Captain V1 behavior and no checkout/cart features are included. - -## Implemented Tool Surface -1. `shopify_search_products` -2. `shopify_get_orders` - -Both tools are registered in: -[`/Users/muhsink/Documents/chatwoot/config/agents/tools.yml`](/Users/muhsink/Documents/chatwoot/config/agents/tools.yml) - -## Shopify Scope Requirements -1. Integration helper includes: -- `read_customers` -- `read_orders` -- `read_fulfillments` -- `read_products` -2. `read_products` is required for product search behavior. - -Reference: -[`/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb`](/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb) - -## Core Files -### Services -1. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/client_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/client_service.rb) -2. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/products_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/products_service.rb) -3. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/orders_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/orders_service.rb) - -### Captain tools -1. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_base_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_base_tool.rb) -2. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_search_products_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_search_products_tool.rb) -3. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_get_orders_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_get_orders_tool.rb) - -### Gating + prompt -1. [`/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb`](/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb) -2. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid) - -## Service Behavior -### `Integrations::Shopify::ClientService` -1. Loads enabled Shopify hook (`app_id: 'shopify'`). -2. Builds Shopify REST admin client. -3. Parses granted scopes from hook settings. -4. Returns normalized results: -- `:not_connected` -- `:provider_error` -5. Emits structured Rails logs for connection and scope context. - -### `Integrations::Shopify::ProductsService` -Method: -- `search_products(query:, limit: 10)` - -Behavior: -1. Rejects blank query with `:no_results`. -2. Validates connection and `read_products` scope. -3. Primary search: active products by title. -4. Fallback search: fetch active products with larger limit, then in-memory keyword match against title/vendor/type. -5. Normalizes product payload: -- `id`, `title`, `vendor`, `product_type`, `handle`, `storefront_url` -- first variant `price` -- computed `availability` summary -6. Error mapping: -- `:insufficient_scope`, `:no_results`, `:provider_error` - -### `Integrations::Shopify::OrdersService` -Method: -- `orders_for_contact(email:, phone_number:, limit: 10)` - -Behavior: -1. Requires at least one identifier (`email` or `phone_number`). -2. Validates connection and scopes `read_customers` + `read_orders`. -3. Runs customer search using `email OR phone`. -4. Fetches orders for first matching customer. -5. Normalizes order payload: -- `id`, `name`, `created_at`, `total_price`, `currency` -- `financial_status`, `fulfillment_status` -- `line_items` (capped) -- `admin_url` -6. Error mapping: -- `:missing_identifier`, `:insufficient_scope`, `:no_results`, `:provider_error` - -## Tool Behavior -### Base tool (`Captain::Tools::ShopifyBaseTool`) -1. `active?` requires: -- account feature flag `captain_integration_v2` -- enabled Shopify integration hook -2. Common deterministic domain error formatting. -3. Identity resolution order for order lookup: -- explicit tool args -- `state.contact` -- `state.captain_v2_trace_current_input` (message text parsing) -- `state.captain_v2_trace_input` history (supports confirmation turns like `Yes`) - -### `shopify_search_products` -Params: -1. `query` (required) - -Behavior: -1. Calls `ProductsService#search_products(query:, limit: 10)`. -2. Returns compact text list with title, price, availability, and product URL. -3. Returns safe fallback messages on domain/provider errors. - -### `shopify_get_orders` -Params: -1. `email` (optional) -2. `phone_number` (optional) - -Behavior: -1. Resolves identity through base-tool identity resolution. -2. Calls `OrdersService#orders_for_contact(email:, phone_number:, limit: 10)`. -3. Returns compact text list with order metadata, admin URL, and top line items. -4. Returns deterministic fallback messages for missing identity/no results/scope/provider errors. - -## Gating Rules -1. Shopify tools are exposed only when Captain V2 is enabled and Shopify is connected. -2. If tool execution is attempted while ineligible, tools return safe non-crashing messages. -3. Existing `available_tool_ids` validation remains source of truth for scenario/tool validity. - -## Current Constraints -1. Order lookup is email/phone-based only in this version. -2. No order-ID lookup path is enabled in this implementation. -3. Product and order responses are plain text lists for Captain tool output. - -## Verification Coverage -### Service specs -1. Product search success/no-results/scope/provider-error. -2. Product title-empty fallback keyword flow. -3. Orders success by email and by phone. -4. Orders missing identifier/no customer/provider-error. - -### Tool specs -1. Active-state gating checks (`V2 off`, `Shopify disconnected`, eligible case). -2. Product tool success/error messaging. -3. Orders tool success/error messaging. -4. Orders identity extraction from: -- state contact -- current input text -- trace history -- explicit tool arguments overriding inferred values - -## Notes for Next Version -1. Optional order lookup by explicit order ID (feature-flagged). -2. Optional verified identity persistence to contact profile. -3. REST-to-GraphQL Shopify migration to address deprecation trajectory. -4. More end-to-end conversation-level regression coverage.