From c3606d22ea6044199b643fd5826a80e88a752370 Mon Sep 17 00:00:00 2001 From: Muhsin Date: Sat, 28 Feb 2026 17:54:29 +0400 Subject: [PATCH] chore: update docs --- captain-shopify-gap-competitor-plan.md | 128 ++++++++++++ shopify_advanced.md | 262 ++++++++++--------------- 2 files changed, 235 insertions(+), 155 deletions(-) create mode 100644 captain-shopify-gap-competitor-plan.md diff --git a/captain-shopify-gap-competitor-plan.md b/captain-shopify-gap-competitor-plan.md new file mode 100644 index 000000000..1b31d5330 --- /dev/null +++ b/captain-shopify-gap-competitor-plan.md @@ -0,0 +1,128 @@ +# Captain + Shopify Gap Analysis and Competitor Comparison Plan + +## 1. Purpose + +Create a clear plan to: +1. Identify product and technical gaps in current Captain + Shopify experience. +2. Compare current behavior with leading support/commerce competitors. +3. Convert findings into a prioritized implementation roadmap. + +## 2. Current Baseline (Chatwoot) + +Current implemented scope: +1. Shopify product search via Captain tool. +2. Shopify order lookup by contact email/phone. +3. Captain V2 + Shopify connection gating. +4. Fallback identity extraction from conversation text/history. + +Out of scope in current version: +1. Order lookup by order ID. +2. Checkout/cart workflows. +3. Deep post-purchase workflows (returns/exchanges automation). +4. GraphQL-first Shopify data layer. + +## 3. Competitor Set + +Primary competitors for comparison: +1. Gorgias (Shopify-first support workflows). +2. Zendesk + Shopify integration. +3. Intercom (Fin + commerce integrations). +4. Tidio / similar SMB Shopify-focused AI support stacks. + +Note: This plan focuses on capability and UX patterns, not pricing. + +## 4. Comparison Dimensions + +Evaluate each competitor and Chatwoot against the same rubric: +1. Identity resolution robustness (email/phone/order context). +2. Order lookup depth (status, fulfillment, tracking, edits). +3. Product intelligence quality (search relevance, detail depth). +4. Agent handoff quality after failed AI attempts. +5. Conversation memory and context carry-forward. +6. Actionability (read-only vs operational actions). +7. Error transparency and recovery UX. +8. Admin visibility (logs, traceability, debugging). +9. Setup friction (scopes, auth, reauthorization handling). +10. Reliability under ambiguous user messages. + +## 5. Gap Assessment Method + +### Step 1: Baseline test scenarios (Chatwoot) +Run a fixed scenario pack: +1. Product availability questions. +2. Specific product detail follow-up. +3. Order status lookup with: +- email present in contact +- email only in user message +- confirmation-only follow-up (`Yes`) +- phone-only identity +4. No-result and provider-error paths. + +### Step 2: Competitor behavior capture +For each scenario: +1. Capture response quality. +2. Capture number of turns to resolution. +3. Capture failure behavior and escalation quality. + +### Step 3: Scorecard +Use 0-3 scoring per dimension: +1. `0` = missing +2. `1` = weak/manual +3. `2` = functional +4. `3` = strong/best-in-class + +## 6. Initial Gap Hypotheses + +Likely current gaps to validate: +1. Order retrieval flexibility: +- No order-id lookup path yet. +2. Product detail richness: +- Limited details when Shopify product metadata is sparse. +3. Proactive recovery: +- AI asks for identity but does not persist verified identity for next turns. +4. Action workflows: +- No built-in operational actions (refund/cancel/update address/escalate with structured payload). +5. Observability UX: +- Logs exist, but no dedicated operational dashboard for Shopify tool outcomes. + +## 7. Roadmap Framework (Post-Analysis) + +### Phase A (Quick Wins) +1. Improve prompt/tool guidance for identity confirmation. +2. Add operator-facing diagnostics for identity source and lookup failure reason. +3. Expand product response template quality. + +### Phase B (Core Capability) +1. Add optional order lookup by order ID (feature-flagged). +2. Persist verified contact identity with safeguards. +3. Add richer order timeline output (payment, fulfillment, tracking where available). + +### Phase C (Strategic) +1. Move Shopify reads to GraphQL Admin API. +2. Add structured post-purchase workflows. +3. Add KPI dashboard for resolution rate and lookup success rate. + +## 8. Success Metrics + +Track before/after: +1. First-response resolution rate for order status requests. +2. Median turns to order lookup completion. +3. Percentage of order lookups failing with missing identifier. +4. Product query relevance satisfaction (manual QA scoring). +5. Human handoff rate for Shopify intents. + +## 9. Deliverables + +1. Gap matrix (Chatwoot vs each competitor by dimension). +2. Ranked backlog of gaps with impact and effort. +3. Implementation proposal with phased milestones. +4. QA plan for each planned feature. + +## 10. Execution Checklist + +1. Finalize scenario pack. +2. Run Chatwoot baseline and record outputs. +3. Run competitor comparisons and record outputs. +4. Complete scoring matrix. +5. Convert top gaps into engineering tickets. +6. Validate each implemented item against the same scenario pack. diff --git a/shopify_advanced.md b/shopify_advanced.md index 49497437d..46b148ae2 100644 --- a/shopify_advanced.md +++ b/shopify_advanced.md @@ -1,199 +1,151 @@ -# `shopify_advanced.md` + Implementation Runbook (Enterprise, Captain V2, Products+Orders) +# Shopify Advanced (Implementation-Aligned Runbook) ## Summary -Implement advanced Shopify support for Captain as **Enterprise + Captain V2 only**, with **phase 1 limited to product search and order lookup**. -No Captain V1 work, no cart/checkout work in this phase. +This document reflects the implemented Shopify support for Captain in Chatwoot: +1. Enterprise-only +2. Captain V2-only (`captain_integration_v2`) +3. Phase 1 scope: product search + order lookup +4. Order lookup is by contact identity (`email` / `phone_number`) only -## Deliverables -1. Root doc file: `shopify_advanced.md` (this plan content). -2. Backend Shopify service layer (shared, read-only). -3. Two new Captain built-in tools: -- `shopify_search_products` -- `shopify_get_orders` -4. Tool exposure/runtime gating to Captain V2 + connected Shopify only. -5. Scope update: add `read_products`. -6. Specs for services, tools, and gating. +No Captain V1 behavior and no checkout/cart features are included. -## Important Interface Changes -1. New Captain tool IDs: -- `shopify_search_products` -- `shopify_get_orders` -2. Shopify required scopes update: -- add `read_products` to `REQUIRED_SCOPES`. -3. No new public REST endpoints for Captain. +## Implemented Tool Surface +1. `shopify_search_products` +2. `shopify_get_orders` -## Files to Create +Both tools are registered in: +[`/Users/muhsink/Documents/chatwoot/config/agents/tools.yml`](/Users/muhsink/Documents/chatwoot/config/agents/tools.yml) -### 1) Shared Shopify services -1. `app/services/integrations/shopify/client_service.rb` -2. `app/services/integrations/shopify/products_service.rb` -3. `app/services/integrations/shopify/orders_service.rb` +## Shopify Scope Requirements +1. Integration helper includes: +- `read_customers` +- `read_orders` +- `read_fulfillments` +- `read_products` +2. `read_products` is required for product search behavior. -### 2) Enterprise Captain tools -1. `enterprise/lib/captain/tools/shopify_base_tool.rb` -2. `enterprise/lib/captain/tools/shopify_search_products_tool.rb` -3. `enterprise/lib/captain/tools/shopify_get_orders_tool.rb` +Reference: +[`/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb`](/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb) -### 3) Specs -1. `spec/services/integrations/shopify/products_service_spec.rb` -2. `spec/services/integrations/shopify/orders_service_spec.rb` -3. `spec/enterprise/lib/captain/tools/shopify_base_tool_spec.rb` -4. `spec/enterprise/lib/captain/tools/shopify_search_products_tool_spec.rb` -5. `spec/enterprise/lib/captain/tools/shopify_get_orders_tool_spec.rb` +## Core Files +### Services +1. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/client_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/client_service.rb) +2. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/products_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/products_service.rb) +3. [`/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/orders_service.rb`](/Users/muhsink/Documents/chatwoot/app/services/integrations/shopify/orders_service.rb) -## Files to Modify +### Captain tools +1. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_base_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_base_tool.rb) +2. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_search_products_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_search_products_tool.rb) +3. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_get_orders_tool.rb`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/tools/shopify_get_orders_tool.rb) -1. [`/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb`](/Users/muhsink/Documents/chatwoot/app/helpers/shopify/integration_helper.rb) -- Add `read_products` to `REQUIRED_SCOPES`. +### Gating + prompt +1. [`/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb`](/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb) +2. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid) -2. [`/Users/muhsink/Documents/chatwoot/config/agents/tools.yml`](/Users/muhsink/Documents/chatwoot/config/agents/tools.yml) -- Register the 2 new tools with title/description/icon. - -3. [`/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb`](/Users/muhsink/Documents/chatwoot/enterprise/app/models/captain/assistant.rb) -- Add helper `shopify_tools_enabled_for_v2?`. -- Auto-append Shopify tools in `agent_tools` when eligible. -- Filter tool metadata in `available_agent_tools` unless eligible. - -4. [`/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid`](/Users/muhsink/Documents/chatwoot/enterprise/lib/captain/prompts/assistant.liquid) -- Add short instruction: use Shopify tools for product/order queries before FAQ fallback. - -5. Optional refactor only (no contract change): -- [`/Users/muhsink/Documents/chatwoot/app/controllers/api/v1/accounts/integrations/shopify_controller.rb`](/Users/muhsink/Documents/chatwoot/app/controllers/api/v1/accounts/integrations/shopify_controller.rb) -- Reuse `OrdersService` internally for consistency. - -## Implementation Details - -### A. `ClientService` -Responsibilities: -1. Load connected hook (`app_id: 'shopify'`, status enabled). -2. Build Shopify REST client from hook token. -3. Parse granted scopes from hook settings. -4. Return normalized failure objects for: +## Service Behavior +### `Integrations::Shopify::ClientService` +1. Loads enabled Shopify hook (`app_id: 'shopify'`). +2. Builds Shopify REST admin client. +3. Parses granted scopes from hook settings. +4. Returns normalized results: - `:not_connected` - `:provider_error` +5. Emits structured Rails logs for connection and scope context. -### B. `ProductsService` +### `Integrations::Shopify::ProductsService` Method: - `search_products(query:, limit: 10)` Behavior: -1. Validate connection + `read_products`. -2. Fetch active products by title query. -3. Normalize output: +1. Rejects blank query with `:no_results`. +2. Validates connection and `read_products` scope. +3. Primary search: active products by title. +4. Fallback search: fetch active products with larger limit, then in-memory keyword match against title/vendor/type. +5. Normalizes product payload: - `id`, `title`, `vendor`, `product_type`, `handle`, `storefront_url` - first variant `price` -- availability summary from variants. -4. Return normalized result/error: -- `:insufficient_scope`, `:no_results`, `:provider_error`. +- computed `availability` summary +6. Error mapping: +- `:insufficient_scope`, `:no_results`, `:provider_error` -### C. `OrdersService` +### `Integrations::Shopify::OrdersService` Method: - `orders_for_contact(email:, phone_number:, limit: 10)` Behavior: -1. Require at least one identifier (`email` or `phone_number`). -2. Validate connection + scopes `read_customers` and `read_orders`. -3. Search customer by `email OR phone`. -4. Fetch orders for first matched customer. -5. Normalize output: +1. Requires at least one identifier (`email` or `phone_number`). +2. Validates connection and scopes `read_customers` + `read_orders`. +3. Runs customer search using `email OR phone`. +4. Fetches orders for first matching customer. +5. Normalizes order payload: - `id`, `name`, `created_at`, `total_price`, `currency` - `financial_status`, `fulfillment_status` - `line_items` (capped) -- `admin_url`. -6. Return normalized result/error: -- `:missing_identifier`, `:no_results`, `:provider_error`. +- `admin_url` +6. Error mapping: +- `:missing_identifier`, `:insufficient_scope`, `:no_results`, `:provider_error` -### D. `ShopifyBaseTool` -Common behavior: -1. Extend `BasePublicTool`. -2. `active?` true only when: -- `captain_integration_v2` enabled -- Shopify hook enabled for account. -3. Common helpers: -- `v2_enabled?` -- `shopify_connected?` -- `resolve_contact_identity(tool_context.state)` -- scope guard + standardized reconnect message. -4. Tool-safe deterministic messages, no exceptions leaked. +## Tool Behavior +### Base tool (`Captain::Tools::ShopifyBaseTool`) +1. `active?` requires: +- account feature flag `captain_integration_v2` +- enabled Shopify integration hook +2. Common deterministic domain error formatting. +3. Identity resolution order for order lookup: +- explicit tool args +- `state.contact` +- `state.captain_v2_trace_current_input` (message text parsing) +- `state.captain_v2_trace_input` history (supports confirmation turns like `Yes`) -### E. `shopify_search_products` tool +### `shopify_search_products` Params: -1. `query` (required string). +1. `query` (required) Behavior: -1. Call `ProductsService#search_products`. -2. Format compact plain text list (max 10): -- title -- price -- availability -- product URL. -3. Deterministic fallback messages for each domain error. +1. Calls `ProductsService#search_products(query:, limit: 10)`. +2. Returns compact text list with title, price, availability, and product URL. +3. Returns safe fallback messages on domain/provider errors. -### F. `shopify_get_orders` tool +### `shopify_get_orders` Params: -1. none (phase 1). +1. `email` (optional) +2. `phone_number` (optional) Behavior: -1. Resolve contact email/phone from conversation state. -2. Call `OrdersService#orders_for_contact`. -3. Format compact plain text list (max 10): -- order name/date/total/status/admin URL -- top line items. -4. Deterministic fallback messages for each domain error. +1. Resolves identity through base-tool identity resolution. +2. Calls `OrdersService#orders_for_contact(email:, phone_number:, limit: 10)`. +3. Returns compact text list with order metadata, admin URL, and top line items. +4. Returns deterministic fallback messages for missing identity/no results/scope/provider errors. -## Gating Rules (Decision-Complete) -1. Tool visibility in UI: -- Hidden unless `captain_integration_v2` and Shopify connected. -2. Tool execution: -- If invoked when ineligible, return non-execution message (no crash). -3. Scenario validity: -- Existing `available_tool_ids`-based validation remains source of truth. +## Gating Rules +1. Shopify tools are exposed only when Captain V2 is enabled and Shopify is connected. +2. If tool execution is attempted while ineligible, tools return safe non-crashing messages. +3. Existing `available_tool_ids` validation remains source of truth for scenario/tool validity. -## Test Cases +## Current Constraints +1. Order lookup is email/phone-based only in this version. +2. No order-ID lookup path is enabled in this implementation. +3. Product and order responses are plain text lists for Captain tool output. +## Verification Coverage ### Service specs -1. Product search success with normalized fields. -2. Product no results. -3. Product missing scope. -4. Orders success by email. -5. Orders success by phone. -6. Orders missing identifier. -7. Orders no matching customer. -8. Provider error mapping for both services. +1. Product search success/no-results/scope/provider-error. +2. Product title-empty fallback keyword flow. +3. Orders success by email and by phone. +4. Orders missing identifier/no customer/provider-error. ### Tool specs -1. `active?` false when V2 disabled. -2. `active?` false when Shopify disconnected. -3. Product tool success formatting. -4. Product tool missing scope message. -5. Orders tool success formatting. -6. Orders tool missing identity message. -7. Orders tool no-result message. -8. Provider error safe messaging. +1. Active-state gating checks (`V2 off`, `Shopify disconnected`, eligible case). +2. Product tool success/error messaging. +3. Orders tool success/error messaging. +4. Orders identity extraction from: +- state contact +- current input text +- trace history +- explicit tool arguments overriding inferred values -### Model/gating specs -1. `available_agent_tools` hides Shopify tools unless eligible. -2. `agent_tools` includes Shopify tools only when eligible. - -### Regression -1. Existing Shopify integration request specs still pass. -2. Existing Captain tools unaffected. - -## Acceptance Criteria -1. With V2 enabled + Shopify connected + proper scopes: -- “Do you have sneakers?” triggers `shopify_search_products` and returns relevant products with links. -- “Where is my order?” triggers `shopify_get_orders` and returns recent order details. -2. With older Shopify install missing product scope: -- Product tool returns reconnect guidance; no crash. -3. With V2 off or no Shopify connection: -- Shopify tools are not shown and not executed. - -## Assumptions -1. Enterprise-only feature is acceptable. -2. Captain V2 only. -3. Phase 1 excludes abandoned checkouts/cart. -4. Plain text output only. -5. English copy updates only where needed. - -## `shopify_advanced.md` Content -Use this entire document as the initial content of `shopify_advanced.md` at repo root. +## Notes for Next Version +1. Optional order lookup by explicit order ID (feature-flagged). +2. Optional verified identity persistence to contact profile. +3. REST-to-GraphQL Shopify migration to address deprecation trajectory. +4. More end-to-end conversation-level regression coverage.