feat: add captain general guidelines migration helpers (#14909)
This PR adds internal tooling and planning docs for migrating existing Captain assistant instructions into the new General Guidelines structure. **Summary** This PR adds a controlled migration path for moving existing Captain V1 assistant instructions into the structured Captain architecture. It introduces a classifier that reads the current `config.instructions` and produces reviewed migration drafts with separate sections for: - assistant description / business context - response guidelines - guardrails - scenario candidates - conversation messages - FAQ/document candidates - needs-review items The migration is intentionally staged. It only targets V1-style assistants that still have custom instructions, are connected to inboxes, and do not already have structured response guidelines, guardrails, or scenario records. When applied, the task writes the extracted business context to the assistant description, response guidelines to `response_guidelines`, guardrails to `guardrails`, and stores scenario candidates / FAQ candidates / review notes under `config["assistant_migration"]`. Scenario candidates are also flattened into response guidelines for now so customer behavior is preserved before we create real `Captain::Scenario` records in a later rollout. The applier stores the original assistant values under migration metadata so conversation message config can be restored if needed. It does not create scenario records yet. **How to generate drafts** For specific assistant IDs: ```bash bundle exec rake captain:assistant_migration:generate \ IDS=546,636,819 \ LIMIT=0 \ OUTPUT=tmp/captain_migration_drafts.jsonl ``` For the first 50 eligible assistants: ```bash bundle exec rake captain:assistant_migration:generate \ OUTPUT=tmp/captain_migration_drafts.jsonl ``` For all eligible assistants: ```bash bundle exec rake captain:assistant_migration:generate \ LIMIT=0 \ OUTPUT=tmp/captain_migration_drafts.jsonl ``` **How to apply drafts** Dry run first: ```bash bundle exec rake captain:assistant_migration:apply \ INPUT=tmp/captain_migration_drafts.jsonl \ DRY_RUN=true ``` Apply changes: ```bash bundle exec rake captain:assistant_migration:apply \ INPUT=tmp/captain_migration_drafts.jsonl \ DRY_RUN=false ``` **How to restore conversation messages** If extracted `welcome_message`, `handoff_message`, or `resolution_message` need to be reverted to their pre-migration values: ```bash bundle exec rake captain:assistant_migration:restore_messages \ IDS=546,636,819 \ DRY_RUN=true ``` ```bash bundle exec rake captain:assistant_migration:restore_messages \ IDS=546,636,819 \ DRY_RUN=false ``` **Notes** - `LIMIT=0` means no limit. - `generate` overwrites the output file. - The apply task skips assistants that are no longer V1 migration candidates. - This PR does not create `Captain::Scenario` records; scenario candidates are staged in assistant config for a future migration. --------- Co-authored-by: Muhsin <12408980+muhsin-k@users.noreply.github.com> Co-authored-by: aakashb95 <aakashbakhle@gmail.com> Co-authored-by: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com>
This commit is contained in:
Muhsin Keloth
co-authored by
Muhsin
aakashb95
Aakash Bakhle
Muhsin
aakashb95
Aakash Bakhle
parent
7caa4e0bbc
commit
8fc5c7a5c8
@@ -0,0 +1,199 @@
|
||||
class Captain::AssistantMigration::DraftApplier
|
||||
ASSISTANT_DESCRIPTION_LIMIT = 500
|
||||
CONFIG_KEY = 'assistant_migration'.freeze
|
||||
SCENARIO_DESCRIPTION_LIMIT = 500
|
||||
ORIGINAL_VALUES_KEY = 'original_values'.freeze
|
||||
|
||||
pattr_initialize [:assistant!, :draft!, { dry_run: true }]
|
||||
|
||||
def perform
|
||||
changes = build_changes
|
||||
apply_changes(changes) unless dry_run
|
||||
|
||||
{
|
||||
assistant_id: assistant.id,
|
||||
dry_run: dry_run,
|
||||
changes: changes
|
||||
}
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def build_changes
|
||||
{
|
||||
description: description_change,
|
||||
response_guidelines: array_change(:response_guidelines, response_guidelines),
|
||||
guardrails: array_change(:guardrails, guardrails),
|
||||
config: config_change
|
||||
}.compact
|
||||
end
|
||||
|
||||
def apply_changes(changes)
|
||||
assistant.transaction do
|
||||
assistant.update!(assistant_update_attributes(changes)) if assistant_update_attributes(changes).present?
|
||||
end
|
||||
end
|
||||
|
||||
def assistant_update_attributes(changes)
|
||||
{}.tap do |attributes|
|
||||
attributes[:description] = changes.dig(:description, :to) if changes[:description].present?
|
||||
attributes[:response_guidelines] = changes.dig(:response_guidelines, :to) if changes[:response_guidelines].present?
|
||||
attributes[:guardrails] = changes.dig(:guardrails, :to) if changes[:guardrails].present?
|
||||
attributes[:config] = changes.dig(:config, :to) if changes[:config].present?
|
||||
end
|
||||
end
|
||||
|
||||
def description_change
|
||||
value = assistant_description_value
|
||||
return if value.blank? || value == assistant.description
|
||||
|
||||
{ from: assistant.description, to: value }
|
||||
end
|
||||
|
||||
def assistant_description_value
|
||||
value = item_values(:business_product_context).join(' ').presence
|
||||
return if value.blank?
|
||||
|
||||
raise ArgumentError, "Assistant description exceeds #{ASSISTANT_DESCRIPTION_LIMIT} characters" if value.length > ASSISTANT_DESCRIPTION_LIMIT
|
||||
|
||||
value
|
||||
end
|
||||
|
||||
def response_guidelines
|
||||
(item_values(:response_guidelines) + scenario_response_guidelines).uniq
|
||||
end
|
||||
|
||||
def guardrails
|
||||
item_values(:guardrails)
|
||||
end
|
||||
|
||||
def array_change(field, values)
|
||||
return if values.blank?
|
||||
|
||||
current = Array(assistant.public_send(field)).map(&:to_s)
|
||||
return if current == values
|
||||
|
||||
{ from: current, to: values }
|
||||
end
|
||||
|
||||
def config_change
|
||||
updated_config = assistant.config.deep_dup
|
||||
conversation_messages.each do |key, value|
|
||||
next if value.blank?
|
||||
next if updated_config[key].present?
|
||||
|
||||
updated_config[key] = value
|
||||
end
|
||||
updated_config[CONFIG_KEY] = migration_config
|
||||
|
||||
return if updated_config == assistant.config
|
||||
|
||||
{ from: assistant.config, to: updated_config }
|
||||
end
|
||||
|
||||
def migration_config
|
||||
existing_migration_config.merge(
|
||||
ORIGINAL_VALUES_KEY => existing_original_values,
|
||||
'scenario_candidates' => staged_scenario_candidates,
|
||||
'faq_document_candidates' => normalized_faq_document_candidates,
|
||||
'needs_review' => normalized_instruction_items(:needs_review)
|
||||
)
|
||||
end
|
||||
|
||||
def existing_migration_config
|
||||
config = assistant.config[CONFIG_KEY]
|
||||
config.is_a?(Hash) ? config : {}
|
||||
end
|
||||
|
||||
def existing_original_values
|
||||
existing_migration_config[ORIGINAL_VALUES_KEY].presence || original_values
|
||||
end
|
||||
|
||||
def original_values
|
||||
{
|
||||
'name' => assistant.name,
|
||||
'description' => assistant.description,
|
||||
'config' => original_config,
|
||||
'response_guidelines' => Array(assistant.response_guidelines),
|
||||
'guardrails' => Array(assistant.guardrails)
|
||||
}
|
||||
end
|
||||
|
||||
def original_config
|
||||
assistant.config.except(CONFIG_KEY)
|
||||
end
|
||||
|
||||
def conversation_messages
|
||||
messages = draft_hash.fetch(:conversation_messages, {})
|
||||
messages = messages.deep_stringify_keys
|
||||
|
||||
{
|
||||
'welcome_message' => messages['welcome_message'].to_s.strip,
|
||||
'handoff_message' => messages['handoff_message'].to_s.strip,
|
||||
'resolution_message' => messages['resolution_message'].to_s.strip
|
||||
}
|
||||
end
|
||||
|
||||
def staged_scenario_candidates
|
||||
scenario_candidates.map do |candidate|
|
||||
candidate.transform_keys(&:to_s)
|
||||
end
|
||||
end
|
||||
|
||||
def scenario_response_guidelines
|
||||
scenario_candidates.filter_map { |candidate| candidate[:response_guideline].presence }
|
||||
end
|
||||
|
||||
def scenario_tool_ids(tool_ids)
|
||||
Array(tool_ids).filter_map { |tool_id| tool_id.to_s.squish.presence }.uniq
|
||||
end
|
||||
|
||||
def scenario_candidates
|
||||
Array(draft_hash[:scenario_candidates]).filter_map do |candidate|
|
||||
normalized_scenario_candidate(candidate)
|
||||
end
|
||||
end
|
||||
|
||||
def normalized_scenario_candidate(candidate)
|
||||
return unless candidate.is_a?(Hash)
|
||||
|
||||
candidate = candidate.deep_symbolize_keys
|
||||
normalized_candidate = {
|
||||
title: candidate[:title].to_s.squish,
|
||||
description: candidate[:description].to_s.squish.truncate(SCENARIO_DESCRIPTION_LIMIT),
|
||||
instruction: candidate[:instruction].to_s.squish,
|
||||
response_guideline: candidate[:response_guideline].to_s.squish,
|
||||
tool_ids: scenario_tool_ids(candidate[:tool_ids])
|
||||
}
|
||||
return if normalized_candidate.values_at(:title, :description, :instruction).any?(&:blank?)
|
||||
|
||||
normalized_candidate
|
||||
end
|
||||
|
||||
def item_values(key)
|
||||
Array(draft_hash[key]).filter_map do |item|
|
||||
item.to_s.squish.presence
|
||||
end.uniq
|
||||
end
|
||||
|
||||
def normalized_instruction_items(key)
|
||||
item_values(key)
|
||||
end
|
||||
|
||||
def normalized_faq_document_candidates
|
||||
Array(draft_hash[:faq_document_candidates]).map do |candidate|
|
||||
raise ArgumentError, 'FAQ document candidates must be question and answer objects' unless candidate.is_a?(Hash)
|
||||
|
||||
candidate = candidate.deep_symbolize_keys
|
||||
question = candidate[:question].to_s.squish
|
||||
answer = candidate[:answer].to_s.squish
|
||||
raise ArgumentError, 'FAQ document candidates must include a question and answer' if question.blank? || answer.blank?
|
||||
|
||||
{ 'question' => question, 'answer' => answer }
|
||||
end.uniq
|
||||
end
|
||||
|
||||
def draft_hash
|
||||
@draft_hash ||= draft.deep_symbolize_keys
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,148 @@
|
||||
class Captain::AssistantMigration::InstructionClassifier < Captain::BaseTaskService
|
||||
RESPONSE_SCHEMA = Captain::AssistantMigration::InstructionClassifierSchema
|
||||
CLASSIFIER_MODEL = 'gpt-5.2'.freeze
|
||||
MAX_INSTRUCTIONS_LENGTH = 20_000
|
||||
|
||||
pattr_initialize [:assistant!]
|
||||
|
||||
def perform
|
||||
response = make_api_call(model: CLASSIFIER_MODEL, messages: messages, schema: RESPONSE_SCHEMA)
|
||||
return error_response(response) if response[:error]
|
||||
|
||||
{
|
||||
assistant: assistant_metadata,
|
||||
draft: normalized_payload(response[:message]),
|
||||
usage: response[:usage],
|
||||
request_messages: response[:request_messages]
|
||||
}
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def account
|
||||
assistant.account
|
||||
end
|
||||
|
||||
def messages
|
||||
[
|
||||
{ role: 'system', content: system_prompt },
|
||||
{ role: 'user', content: user_prompt }
|
||||
]
|
||||
end
|
||||
|
||||
def system_prompt
|
||||
Captain::PromptRenderer.render('instruction_classifier')
|
||||
end
|
||||
|
||||
def user_prompt
|
||||
JSON.pretty_generate(assistant_payload)
|
||||
end
|
||||
|
||||
def assistant_payload # rubocop:disable Metrics/AbcSize
|
||||
{
|
||||
assistant_id: assistant.id,
|
||||
account_id: assistant.account_id,
|
||||
account_name: assistant.account.name,
|
||||
name: assistant.name,
|
||||
description: assistant.description,
|
||||
product_name: assistant.config['product_name'],
|
||||
instructions: truncated_instructions,
|
||||
welcome_message: assistant.config['welcome_message'],
|
||||
handoff_message: assistant.config['handoff_message'],
|
||||
resolution_message: assistant.config['resolution_message'],
|
||||
existing_response_guidelines: assistant.response_guidelines || [],
|
||||
existing_guardrails: assistant.guardrails || [],
|
||||
existing_scenarios: existing_scenarios,
|
||||
available_agent_tools: available_agent_tools,
|
||||
feature_settings: feature_settings
|
||||
}
|
||||
end
|
||||
|
||||
def truncated_instructions
|
||||
instructions = assistant.config['instructions'].to_s
|
||||
return instructions if instructions.length <= MAX_INSTRUCTIONS_LENGTH
|
||||
|
||||
"#{instructions.first(MAX_INSTRUCTIONS_LENGTH)}\n\n[TRUNCATED]"
|
||||
end
|
||||
|
||||
def existing_scenarios
|
||||
assistant.scenarios.map do |scenario|
|
||||
{
|
||||
id: scenario.id,
|
||||
title: scenario.title,
|
||||
description: scenario.description,
|
||||
instruction: scenario.instruction,
|
||||
enabled: scenario.enabled
|
||||
}
|
||||
end
|
||||
end
|
||||
|
||||
def available_agent_tools
|
||||
tools = assistant.respond_to?(:available_agent_tools) ? assistant.available_agent_tools : Captain::Assistant.built_in_agent_tools
|
||||
tools.map { |tool| tool.slice(:id, :title, :description) }
|
||||
end
|
||||
|
||||
def feature_settings
|
||||
assistant.config.slice(
|
||||
'feature_faq',
|
||||
'feature_memory',
|
||||
'feature_citation',
|
||||
'feature_contact_attributes',
|
||||
'temperature'
|
||||
)
|
||||
end
|
||||
|
||||
def normalized_payload(message)
|
||||
payload = message.is_a?(Hash) ? message.deep_symbolize_keys : {}
|
||||
payload.reverse_merge(
|
||||
business_product_context: [],
|
||||
response_guidelines: [],
|
||||
guardrails: [],
|
||||
scenario_candidates: [],
|
||||
conversation_messages: {},
|
||||
faq_document_candidates: [],
|
||||
needs_review: [],
|
||||
classification_notes: []
|
||||
)
|
||||
end
|
||||
|
||||
def assistant_metadata # rubocop:disable Metrics/AbcSize
|
||||
{
|
||||
id: assistant.id,
|
||||
name: assistant.name,
|
||||
account_id: assistant.account_id,
|
||||
account_name: assistant.account.name,
|
||||
inbox_count: assistant.captain_inboxes.size,
|
||||
instruction_length: assistant.config['instructions'].to_s.length,
|
||||
original_instructions: assistant.config['instructions'].to_s,
|
||||
welcome_message: assistant.config['welcome_message'].to_s,
|
||||
handoff_message: assistant.config['handoff_message'].to_s,
|
||||
resolution_message: assistant.config['resolution_message'].to_s
|
||||
}
|
||||
end
|
||||
|
||||
def error_response(response)
|
||||
{
|
||||
assistant: assistant_metadata,
|
||||
error: response[:error],
|
||||
error_code: response[:error_code],
|
||||
request_messages: response[:request_messages]
|
||||
}
|
||||
end
|
||||
|
||||
def event_name
|
||||
'assistant_migration_instruction_classifier'
|
||||
end
|
||||
|
||||
def captain_tasks_enabled?
|
||||
true
|
||||
end
|
||||
|
||||
def counts_toward_usage?
|
||||
false
|
||||
end
|
||||
|
||||
def build_follow_up_context?
|
||||
false
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,91 @@
|
||||
class Captain::AssistantMigration::InstructionClassifierSchema < RubyLLM::Schema
|
||||
DESCRIPTION_LENGTH_LIMIT = 500
|
||||
|
||||
def self.instruction_items(field_name, description:, max_items: 20)
|
||||
array field_name,
|
||||
description: "#{description} Return plain standalone sentences without numbering, bullets, or section labels.",
|
||||
max_items: max_items,
|
||||
of: :string
|
||||
end
|
||||
|
||||
array :business_product_context,
|
||||
description: "Single compact root assistant description for the root orchestrator prompt, maximum #{DESCRIPTION_LENGTH_LIMIT} characters: " \
|
||||
'preserve the existing assistant description and enrich it only with relevant business/product context from the ' \
|
||||
'custom instructions. Include assistant identity, product scope, high-level mission, and high-level source/routing ' \
|
||||
'priorities only. Do not include workflows, procedures, attribute glossaries, policy details, or long inventories. ' \
|
||||
'Return complete plain prose without numbering, bullets, section labels, or a truncated final sentence.',
|
||||
min_items: 1,
|
||||
max_items: 1 do
|
||||
string max_length: DESCRIPTION_LENGTH_LIMIT
|
||||
end
|
||||
|
||||
instruction_items :response_guidelines,
|
||||
description: 'Tone, language, answer length, formatting, and clarification behavior.',
|
||||
max_items: 20
|
||||
|
||||
instruction_items :guardrails,
|
||||
description: 'Refusal rules, escalation boundaries, source boundaries, safety limits, and things the assistant must not do.',
|
||||
max_items: 20
|
||||
|
||||
array :scenario_candidates,
|
||||
description: 'Review-stage specialized-agent candidates. These are also temporarily flattened into response guidelines.',
|
||||
max_items: 15 do
|
||||
object do
|
||||
string :title,
|
||||
description: 'Short scenario agent title for a distinct user-intent workflow.',
|
||||
max_length: 80
|
||||
string :description,
|
||||
description: 'When this specialized scenario should be used. This is shown to the orchestrator for routing.',
|
||||
max_length: 500
|
||||
string :instruction,
|
||||
description: 'How the specialized agent should handle the workflow. Include only evidence-backed markdown tool links. ' \
|
||||
'Do not include confidence labels or review notes.',
|
||||
max_length: 2000
|
||||
string :response_guideline,
|
||||
description: 'Same-language, customer-visible response guideline that preserves this scenario behavior when flattened. ' \
|
||||
'Do not include tool syntax, tool names, labels, private-note instructions, or internal implementation details.',
|
||||
max_length: 1000
|
||||
array :tool_ids,
|
||||
description: 'Available tool IDs explicitly referenced in instruction using markdown links. Empty when no tools are required.',
|
||||
max_items: 10,
|
||||
of: :string
|
||||
end
|
||||
end
|
||||
|
||||
object :conversation_messages, description: 'Exact globally reusable customer-facing message copy found in instructions. ' \
|
||||
'Leave empty for conditional, placeholder, or workflow-specific copy.' do
|
||||
string :welcome_message, description: 'Exact globally reusable initial greeting copy from instructions, or empty string. ' \
|
||||
'Do not convert an instruction about greeting into message copy.',
|
||||
max_length: 1000
|
||||
string :handoff_message,
|
||||
description: 'Exact globally reusable human-handoff message copy from instructions, or empty string. ' \
|
||||
'Do not use scenario-specific, team-specific, placeholder, or conditional handoff copy.',
|
||||
max_length: 1000
|
||||
string :resolution_message,
|
||||
description: 'Exact globally reusable resolution/closing message copy from instructions, or empty string. ' \
|
||||
'Do not use conditional or placeholder closing copy.',
|
||||
max_length: 1000
|
||||
end
|
||||
|
||||
array :faq_document_candidates,
|
||||
description: 'Pending FAQ candidates for factual or product-specific knowledge such as pricing, policy, setup, troubleshooting, ' \
|
||||
'or operational details. These candidates remain inactive until reviewed and approved.',
|
||||
max_items: 25 do
|
||||
object do
|
||||
string :question,
|
||||
description: 'Natural, standalone customer question about factual product or business knowledge.',
|
||||
max_length: 255
|
||||
string :answer,
|
||||
description: 'Self-contained factual answer using only the existing instructions. Do not include assistant behavior, ' \
|
||||
'tool use, or message copy. Preserve exact values, conditions, and exceptions.',
|
||||
max_length: 2000
|
||||
end
|
||||
end
|
||||
|
||||
instruction_items :needs_review,
|
||||
description: 'Unclear, conflicting, risky, duplicated, or uncertain content that needs human review. ' \
|
||||
'Include the reason in the item text.',
|
||||
max_items: 20
|
||||
|
||||
array :classification_notes, description: 'Short notes about important migration decisions or risks.', max_items: 10, of: :string
|
||||
end
|
||||
@@ -0,0 +1,137 @@
|
||||
You are migrating Captain assistant instructions into a structured configuration.
|
||||
|
||||
Classify the existing assistant instructions into these sections:
|
||||
1. Business/Product Context
|
||||
2. Response Guidelines
|
||||
3. Guardrails
|
||||
4. Scenario Candidates
|
||||
5. Conversation Messages
|
||||
6. FAQs/Documents Candidates
|
||||
7. Needs Review
|
||||
|
||||
## General Rules
|
||||
|
||||
- Preserve behavior as closely as possible.
|
||||
- Do not duplicate the same content across sections.
|
||||
- Return clean migrated values only. Do not include source excerpts, source labels, citations, or "Source:" text in any migrated field.
|
||||
- Do not rewrite customer-facing message copy unless necessary to classify an exact copy from instructions.
|
||||
- Do not include confidence labels, review labels, bracketed reviewer comments, or schema labels inside migrated values.
|
||||
- For Business/Product Context, Response Guidelines, and Guardrails, return each item as a plain standalone sentence.
|
||||
Do not prefix items with numbers, bullets, section labels, or list markers such as "1.", "-", or "*".
|
||||
- When several instructions share the same trigger, condition, or subject, combine them into one concise item instead
|
||||
of repeating the same trigger across multiple items. Preserve every required action, prohibition, and routing
|
||||
outcome from the source instruction when combining.
|
||||
- If unsure, place content in Needs Review and include the reason in that item.
|
||||
- Return data that matches the provided schema.
|
||||
|
||||
## Business/Product Context
|
||||
|
||||
- Business/Product Context maps to the root assistant description and is injected into the root orchestrator prompt.
|
||||
- Return exactly one Business/Product Context item.
|
||||
- Start with the existing assistant description and preserve its meaning.
|
||||
- Enrich it only with relevant business or product context found in the custom instructions.
|
||||
- Produce one coherent description rather than appending a second context block or repeating the existing description.
|
||||
- Keep it at most 500 characters because that is the assistant description limit in the UI and model.
|
||||
- Prefer roughly 300-450 characters when the source needs detail, leaving room below the hard limit.
|
||||
- Finish the description cleanly. Never end mid-word, mid-clause, after an opening bracket, or with a dangling separator.
|
||||
- Make it a compact summary of assistant identity, product scope, high-level mission, and high-level source or routing priorities.
|
||||
- Do not include detailed workflows, step-by-step procedures, long support-scope inventories, attribute glossaries,
|
||||
policy details, scenario-specific handling, tool instructions, or customer-facing message copy.
|
||||
|
||||
## Conversation Messages
|
||||
|
||||
- Existing welcome_message, handoff_message, and resolution_message config values are provided separately.
|
||||
- Treat welcome_message, handoff_message, and resolution_message as conversation message config fields.
|
||||
- Extract exact welcome, handoff, or resolution message copy from instructions into conversation_messages when present.
|
||||
- Only classify handoff copy as conversation_messages.handoff_message when it is generic enough to reuse for any human handoff.
|
||||
- If handoff copy is scenario-specific, keep it inside that scenario instruction; if it is only a rule about when or how to hand off, classify it as a Response Guideline or Guardrail.
|
||||
- Do not extract a conversation message from an instruction about what to say, from a placeholder template,
|
||||
from conditional copy, from role/team-specific copy, or from text that only applies inside one workflow.
|
||||
- If a message contains placeholders such as a blank name, team name, bracketed variable, business-hours state,
|
||||
or dynamic runtime condition, do not place it in conversation_messages. Keep it in the relevant workflow or
|
||||
Needs Review.
|
||||
- Do not copy message values from existing config into conversation_messages.
|
||||
- Do not decide whether existing config values should be overwritten. Migration code handles applying extracted
|
||||
conversation_messages only when the corresponding config value is blank.
|
||||
|
||||
## Scenario Candidates
|
||||
|
||||
- In the current architecture, a scenario becomes a specialized sub-agent with its own title, description,
|
||||
instructions, and optional tools.
|
||||
- During this migration, scenario candidates are also temporarily flattened into response guidelines so existing
|
||||
assistant behavior is preserved before scenario records are created.
|
||||
- For every scenario candidate, write a response_guideline that is the flattened version of that scenario for
|
||||
the root assistant's response guidelines.
|
||||
- The response_guideline must be in the same language as the original scenario or source instruction.
|
||||
- The response_guideline must preserve the intended customer-visible behavior, trigger, information to collect,
|
||||
and routing/escalation outcome.
|
||||
- The response_guideline must not include tool syntax, tool:// links, markdown tool links, tool names, label
|
||||
updates, priority updates, private-note instructions, custom-tool instructions, or internal implementation details.
|
||||
- If the scenario uses internal tools such as labels, priorities, private notes, or custom tools, describe only
|
||||
the customer-visible behavior and expected routing/escalation outcome in response_guideline.
|
||||
- If human handoff is needed, describe it in natural language such as route/escalate/transfer to a human; do not
|
||||
mention the handoff tool in response_guideline.
|
||||
- Keep scenario titles, descriptions, instructions, and response_guidelines clear, self-contained, and reviewable.
|
||||
- Only create scenario candidates for distinct user-intent workflows that should be routed to a specialized agent.
|
||||
A candidate must be narrow enough to become a named specialist assistant with domain-specific handling instructions.
|
||||
- Good scenario candidates include multi-step intake workflows, qualification flows, specialized troubleshooting
|
||||
workflows, booking flows, lead-capture flows, recommendation flows, fulfillment workflows, or tool-use procedures
|
||||
for a specific user intent.
|
||||
- A scenario candidate should answer "yes" to this test: would a named specialist sub-agent improve handling
|
||||
beyond the base assistant's global FAQ, guardrail, response-guideline, and human-handoff behavior?
|
||||
- Do not create scenario candidates for global escalation rules, generic handoff policy, missing-information
|
||||
behavior, source-boundary rules, refusal rules, tone, formatting, answer length, or one-step fallback behavior.
|
||||
- Do not create scenario candidates whose main purpose is to escalate or hand off. "Identify the trigger, avoid
|
||||
guessing, tell the user support will review, and hand off" is a guardrail/handoff boundary, not a scenario,
|
||||
even though it contains multiple statements.
|
||||
- Do create scenario candidates when the instructions define a concrete intake, qualification, troubleshooting,
|
||||
booking, lead-capture, recommendation, or fulfillment workflow, even when the workflow eventually hands off
|
||||
to a human.
|
||||
- Do not create scenario candidates for simple routing triggers such as "user asks for a human", "immediately
|
||||
hand off this category", or "route sales questions to the sales team" when there is no concrete workflow to run.
|
||||
- Handoff behavior is a scenario candidate only when part of a larger intake, qualification, or specialized handling workflow.
|
||||
- Global rules like "if not in docs, escalate", "ask one clarifying question", "do not answer account-specific
|
||||
questions", or "tell the user support will review" belong in Guardrails or Response Guidelines, not Scenario Candidates.
|
||||
- Broad buckets like "account-specific issue escalation", "unknown question escalation", "contact support",
|
||||
"fallback to human", or "documentation unavailable" are not scenario candidates.
|
||||
|
||||
## Tool Use
|
||||
|
||||
- If a scenario candidate requires tools, reference the available tool explicitly inside the scenario instruction
|
||||
using markdown tool links such as [Handoff to Human](tool://handoff).
|
||||
- Use only tool IDs listed in available_agent_tools. If a needed tool is unavailable or the workflow depends on
|
||||
unavailable runtime data such as FAQ relevance scores or business-hours status, place it in Needs Review instead.
|
||||
- Do not map an unavailable named tool to a different available tool. For example, do not treat FAQ Lookup as
|
||||
Product Search, Order Status, website browsing, pricing lookup, agent availability, business-hours detection,
|
||||
ticket creation, or custom-attribute assignment unless the instructions explicitly say that the available
|
||||
tool provides that behavior.
|
||||
- If a workflow cannot run without an unavailable tool or runtime signal, do not create a tool-backed scenario
|
||||
for it. Preserve the instruction in Needs Review with the missing capability named.
|
||||
|
||||
## FAQs/Documents Candidates
|
||||
|
||||
- Convert factual or product-specific knowledge into pending FAQ candidates with a natural customer question and a self-contained answer.
|
||||
- FAQ candidates are review-stage data only. They are not active assistant knowledge until a human reviews and approves them.
|
||||
- Use only facts stated in the existing instructions. Do not invent, generalize, update, or fill in missing details.
|
||||
- Preserve exact prices, limits, dates, time zones, conditions, exceptions, product names, and operational details in the answer.
|
||||
- Write each question as a standalone question a customer might naturally ask. Make it specific enough to retrieve the corresponding answer.
|
||||
- Write each answer so it fully answers its question without relying on another FAQ candidate or surrounding context.
|
||||
- Split unrelated facts into separate candidates. Keep related conditions and exceptions together when separating them would make an answer incomplete.
|
||||
- Do not create FAQ candidates about what the assistant should say or do, how it should use sources or tools, when it should route or escalate,
|
||||
or which exact message it should send. Classify those as Response Guidelines, Guardrails, Scenario Candidates, Conversation Messages,
|
||||
or Needs Review as appropriate.
|
||||
- FAQ questions must ask about the product or business, not about the assistant. Do not write questions such as "What should the assistant answer?",
|
||||
"What should I say?", "Which source should the assistant use?", or "Which tool should be called?".
|
||||
- FAQ answers must contain customer-facing knowledge, not instructions to call tools, inspect internal data, update records, transfer conversations,
|
||||
or follow internal workflows.
|
||||
- When factual sources conflict and the instructions do not explicitly establish which fact overrides the others, put the conflict in Needs Review
|
||||
instead of creating an FAQ candidate. Use an explicitly stated override or superseding fact when one is present.
|
||||
- Only factual or product-specific knowledge should become FAQs/Documents candidates.
|
||||
- Generic capability statements such as "answer product questions", "help with billing",
|
||||
"troubleshoot common issues", or "direct to documentation" are not FAQ/document candidates.
|
||||
Put them in Business/Product Context or Response Guidelines when useful.
|
||||
- Product facts, pricing, policies, setup steps, troubleshooting facts, support hours, emergency contacts,
|
||||
and operational details should become pending FAQ candidates, not Response Guidelines or trusted approved knowledge.
|
||||
- Do not create FAQ/document candidates for topic labels or unsupported capabilities when the factual content is
|
||||
missing. Put "pricing details are needed", "same-day delivery schedule details are needed", or similar gaps in
|
||||
Needs Review instead.
|
||||
@@ -0,0 +1,278 @@
|
||||
require 'json'
|
||||
require 'fileutils'
|
||||
require 'csv'
|
||||
|
||||
# rubocop:disable Metrics/BlockLength
|
||||
namespace :captain do
|
||||
namespace :assistant_migration do
|
||||
desc 'Generate structured migration drafts. Usage: rake captain:assistant_migration:generate IDS=1,2,3 LIMIT=50 ' \
|
||||
'OUTPUT=tmp/captain_migration.jsonl'
|
||||
task generate: :environment do
|
||||
assistants = CaptainAssistantMigrationTask.assistants
|
||||
output_path = ENV.fetch('OUTPUT', Rails.root.join('tmp/captain_assistant_migration_drafts.jsonl').to_s)
|
||||
|
||||
FileUtils.mkdir_p(File.dirname(output_path))
|
||||
processed = 0
|
||||
|
||||
File.open(output_path, 'w') do |file|
|
||||
CaptainAssistantMigrationTask.each_assistant(assistants) do |assistant|
|
||||
result = Captain::AssistantMigration::InstructionClassifier.new(assistant: assistant).perform
|
||||
file.puts(JSON.generate(result))
|
||||
processed += 1
|
||||
puts "Generated migration draft for assistant #{assistant.id} (#{processed}/#{CaptainAssistantMigrationTask.assistant_count(assistants)})"
|
||||
end
|
||||
end
|
||||
|
||||
puts "Wrote #{processed} migration drafts to #{output_path}"
|
||||
end
|
||||
|
||||
desc 'Apply reviewed migration drafts. Usage: rake captain:assistant_migration:apply INPUT=tmp/reviewed.jsonl DRY_RUN=true'
|
||||
task apply: :environment do
|
||||
input_path = ENV.fetch('INPUT')
|
||||
dry_run = CaptainAssistantMigrationTask.truthy?('DRY_RUN', default: true)
|
||||
|
||||
results = CaptainAssistantMigrationTask.apply_drafts(
|
||||
input_path: input_path,
|
||||
dry_run: dry_run
|
||||
)
|
||||
|
||||
results.each { |result| puts(JSON.generate(result)) }
|
||||
puts "Processed #{results.size} migration drafts from #{input_path}"
|
||||
puts 'Dry run only. Re-run with DRY_RUN=false to write changes.' if dry_run
|
||||
end
|
||||
|
||||
desc 'Restore conversation message config from migration backup. Usage: rake captain:assistant_migration:restore_messages IDS=1,2 DRY_RUN=true'
|
||||
task restore_messages: :environment do
|
||||
dry_run = CaptainAssistantMigrationTask.truthy?('DRY_RUN', default: true)
|
||||
results = CaptainAssistantMigrationTask.restore_conversation_messages(dry_run: dry_run)
|
||||
|
||||
results.each { |result| puts(JSON.generate(result)) }
|
||||
puts "Processed #{results.size} assistant message restores"
|
||||
puts 'Dry run only. Re-run with DRY_RUN=false to restore conversation messages.' if dry_run
|
||||
end
|
||||
end
|
||||
end
|
||||
# rubocop:enable Metrics/BlockLength
|
||||
|
||||
# rubocop:disable Style/OneClassPerFile
|
||||
class CaptainAssistantMigrationTask
|
||||
CsvAccount = Struct.new(:id, :name, keyword_init: true) do
|
||||
def captain_models
|
||||
{}
|
||||
end
|
||||
|
||||
def conversations
|
||||
CsvRelation.new
|
||||
end
|
||||
end
|
||||
|
||||
CsvAssociation = Struct.new(:inbox_count, keyword_init: true) do
|
||||
def size
|
||||
inbox_count
|
||||
end
|
||||
end
|
||||
|
||||
class CsvRelation
|
||||
def find_by(*)
|
||||
nil
|
||||
end
|
||||
|
||||
def exists?
|
||||
false
|
||||
end
|
||||
end
|
||||
|
||||
CsvAssistant = Struct.new(
|
||||
:id,
|
||||
:name,
|
||||
:account_id,
|
||||
:account,
|
||||
:description,
|
||||
:config,
|
||||
:response_guidelines,
|
||||
:guardrails,
|
||||
:captain_inboxes,
|
||||
:scenarios,
|
||||
keyword_init: true
|
||||
)
|
||||
|
||||
class << self
|
||||
def assistants
|
||||
return csv_assistants if ENV['CSV_INPUT'].present?
|
||||
|
||||
scope = Captain::Assistant.includes(:account, :captain_inboxes, :scenarios)
|
||||
|
||||
ids = ENV.fetch('IDS', '').split(',').filter_map { |id| id.strip.presence }
|
||||
scope = scope.where(id: ids) if ids.any?
|
||||
|
||||
scope = migration_eligible_scope(scope).order(:id)
|
||||
|
||||
limit = ENV.fetch('LIMIT', 50).to_i
|
||||
limit.positive? ? scope.limit(limit) : scope
|
||||
end
|
||||
|
||||
def each_assistant(assistants, &)
|
||||
return assistants.find_each(&) if assistants.respond_to?(:find_each)
|
||||
|
||||
assistants.each(&)
|
||||
end
|
||||
|
||||
def assistant_count(assistants)
|
||||
assistants.respond_to?(:size) ? assistants.size : assistants.count
|
||||
end
|
||||
|
||||
def restore_conversation_messages(dry_run:)
|
||||
ENV.fetch('IDS').split(',').filter_map { |id| id.strip.presence }.map do |assistant_id|
|
||||
assistant = Captain::Assistant.find(assistant_id)
|
||||
restore_conversation_messages_for(assistant, dry_run: dry_run)
|
||||
rescue ActiveRecord::RecordNotFound
|
||||
{ assistant_id: assistant_id, error: 'Assistant not found' }
|
||||
end
|
||||
end
|
||||
|
||||
def apply_drafts(input_path:, dry_run:)
|
||||
File.readlines(input_path, chomp: true).filter_map.with_index(1) do |line, line_number|
|
||||
next if line.blank?
|
||||
|
||||
apply_draft(JSON.parse(line), line_number: line_number, dry_run: dry_run)
|
||||
rescue JSON::ParserError => e
|
||||
{ line_number: line_number, error: "Invalid JSON: #{e.message}" }
|
||||
end
|
||||
end
|
||||
|
||||
def apply_draft(payload, line_number:, dry_run:)
|
||||
return { line_number: line_number, skipped: true, reason: payload['error'] } if payload['error'].present?
|
||||
|
||||
assistant_id = payload.dig('assistant', 'id') || payload['assistant_id']
|
||||
assistant = Captain::Assistant.find(assistant_id)
|
||||
return skipped_result(line_number, assistant_id, 'Assistant is not a V1 migration candidate') unless migration_candidate?(assistant)
|
||||
|
||||
draft = payload['draft'] || payload
|
||||
|
||||
Captain::AssistantMigration::DraftApplier.new(
|
||||
assistant: assistant,
|
||||
draft: draft,
|
||||
dry_run: dry_run
|
||||
).perform.merge(line_number: line_number)
|
||||
rescue ActiveRecord::RecordNotFound
|
||||
{ line_number: line_number, assistant_id: assistant_id, error: 'Assistant not found' }
|
||||
end
|
||||
|
||||
def truthy?(key, default:)
|
||||
value = ENV.fetch(key, nil)
|
||||
return default if value.nil?
|
||||
|
||||
value.to_s.downcase.in?(%w[1 true yes y])
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def restore_conversation_messages_for(assistant, dry_run:)
|
||||
original_config = assistant.config.dig(
|
||||
Captain::AssistantMigration::DraftApplier::CONFIG_KEY,
|
||||
Captain::AssistantMigration::DraftApplier::ORIGINAL_VALUES_KEY,
|
||||
'config'
|
||||
)
|
||||
return skipped_result(nil, assistant.id, 'No stored migration original config found') if original_config.nil?
|
||||
|
||||
config, changes = restored_message_config(assistant.config.deep_dup, original_config)
|
||||
assistant.update!(config: config) if !dry_run && changes.present?
|
||||
|
||||
{ assistant_id: assistant.id, dry_run: dry_run, changes: changes }
|
||||
end
|
||||
|
||||
def restored_message_config(config, original_config)
|
||||
changes = {}
|
||||
%w[welcome_message handoff_message resolution_message].each do |key|
|
||||
original_present = original_config.key?(key)
|
||||
next if config[key] == original_config[key] && config.key?(key) == original_present
|
||||
|
||||
changes[key] = { from: config[key], to: original_config[key] }
|
||||
original_present ? config[key] = original_config[key] : config.delete(key)
|
||||
end
|
||||
[config, changes]
|
||||
end
|
||||
|
||||
def skipped_result(line_number, assistant_id, reason)
|
||||
{
|
||||
line_number: line_number,
|
||||
assistant_id: assistant_id,
|
||||
skipped: true,
|
||||
reason: reason
|
||||
}
|
||||
end
|
||||
|
||||
def migration_eligible_scope(scope)
|
||||
scope.left_outer_joins(:scenarios)
|
||||
.joins(:captain_inboxes)
|
||||
.where("NULLIF(captain_assistants.config->>'instructions', '') IS NOT NULL")
|
||||
.where("captain_assistants.response_guidelines IS NULL OR captain_assistants.response_guidelines = '[]'::jsonb")
|
||||
.where("captain_assistants.guardrails IS NULL OR captain_assistants.guardrails = '[]'::jsonb")
|
||||
.where(captain_scenarios: { id: nil })
|
||||
.distinct
|
||||
end
|
||||
|
||||
def migration_candidate?(assistant)
|
||||
assistant.config['instructions'].present? &&
|
||||
assistant.captain_inboxes.size.positive? &&
|
||||
Array(assistant.response_guidelines).blank? &&
|
||||
Array(assistant.guardrails).blank? &&
|
||||
!scenarios_exist?(assistant)
|
||||
end
|
||||
|
||||
def scenarios_exist?(assistant)
|
||||
scenarios = assistant.scenarios
|
||||
return scenarios.exists? if scenarios.respond_to?(:exists?)
|
||||
|
||||
scenarios.present?
|
||||
end
|
||||
|
||||
def csv_assistants # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
||||
rows = CSV.read(ENV.fetch('CSV_INPUT'), headers: true)
|
||||
ids = ENV.fetch('IDS', '').split(',').filter_map { |id| id.strip.presence }
|
||||
status = ENV.fetch('STATUS', '').presence
|
||||
|
||||
assistants = rows.filter_map do |row|
|
||||
next if ids.any? && ids.exclude?(row['id'].to_s)
|
||||
next if status.present? && row['status'].to_s != status
|
||||
|
||||
assistant = csv_assistant(row)
|
||||
next unless migration_candidate?(assistant)
|
||||
|
||||
assistant
|
||||
end
|
||||
|
||||
limit = ENV.fetch('LIMIT', 50).to_i
|
||||
limit.positive? ? assistants.first(limit) : assistants
|
||||
end
|
||||
|
||||
def csv_assistant(row)
|
||||
config = parse_json(row['config'], fallback: {})
|
||||
CsvAssistant.new(
|
||||
id: normalize_integer(row['id']),
|
||||
name: row['name'].to_s,
|
||||
account_id: normalize_integer(row['account_id']),
|
||||
account: CsvAccount.new(id: normalize_integer(row['account_id']), name: row['account_name'].to_s),
|
||||
description: row['description'].to_s,
|
||||
config: config,
|
||||
response_guidelines: parse_json(row['response_guidelines'], fallback: []),
|
||||
guardrails: parse_json(row['guardrails'], fallback: []),
|
||||
captain_inboxes: CsvAssociation.new(inbox_count: normalize_integer(row['inbox_count'])),
|
||||
scenarios: []
|
||||
)
|
||||
end
|
||||
|
||||
def parse_json(value, fallback:)
|
||||
return fallback if value.blank?
|
||||
|
||||
JSON.parse(value)
|
||||
rescue JSON::ParserError
|
||||
fallback
|
||||
end
|
||||
|
||||
def normalize_integer(value)
|
||||
value.to_s.delete(',').to_i
|
||||
end
|
||||
end
|
||||
end
|
||||
# rubocop:enable Style/OneClassPerFile
|
||||
@@ -0,0 +1,116 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe Captain::AssistantMigration::DraftApplier do
|
||||
let(:account) { create(:account) }
|
||||
let(:assistant) do
|
||||
create(
|
||||
:captain_assistant,
|
||||
account: account,
|
||||
config: { 'product_name' => 'Test Product', 'instructions' => 'Legacy V1 custom instructions.' },
|
||||
response_guidelines: [],
|
||||
guardrails: []
|
||||
)
|
||||
end
|
||||
let(:scenario_candidate) do
|
||||
{
|
||||
'title' => 'Billing Investigation',
|
||||
'description' => 'Use when a customer reports an account-specific billing issue.',
|
||||
'instruction' => 'Collect the invoice number and summarize the issue before escalating.',
|
||||
'response_guideline' => 'For account-specific billing issues, collect the invoice number and summarize the issue before escalating.',
|
||||
'tool_ids' => []
|
||||
}
|
||||
end
|
||||
let(:faq_document_candidate) do
|
||||
{
|
||||
'question' => 'When is support available?',
|
||||
'answer' => 'Support is available Monday to Friday.'
|
||||
}
|
||||
end
|
||||
let(:draft) do
|
||||
{
|
||||
business_product_context: ['Support assistant for Test Product.'],
|
||||
response_guidelines: ['Be concise.'],
|
||||
guardrails: ['Do not guess.'],
|
||||
conversation_messages: {},
|
||||
scenario_candidates: [scenario_candidate],
|
||||
faq_document_candidates: [faq_document_candidate],
|
||||
needs_review: ['Pricing details are missing because factual details are absent from the source instructions.']
|
||||
}
|
||||
end
|
||||
|
||||
describe '#perform' do
|
||||
it 'reports staged scenario candidates in dry run without writing to the assistant' do
|
||||
result = described_class.new(assistant: assistant, draft: draft, dry_run: true).perform
|
||||
|
||||
expect(result.dig(:changes, :config, :to, 'assistant_migration', 'scenario_candidates')).to eq([scenario_candidate])
|
||||
expect(result.dig(:changes, :response_guidelines, :to)).to include(
|
||||
'For account-specific billing issues, collect the invoice number and summarize the issue before escalating.'
|
||||
)
|
||||
expect(assistant.reload.config).not_to have_key('assistant_migration')
|
||||
expect(assistant.scenarios.count).to eq(0)
|
||||
end
|
||||
|
||||
it 'stores scenario candidates in assistant config and flattens them into response guidelines' do
|
||||
described_class.new(assistant: assistant, draft: draft, dry_run: false).perform
|
||||
|
||||
assistant.reload
|
||||
expect(assistant.config.dig('assistant_migration', 'scenario_candidates')).to eq([scenario_candidate])
|
||||
expect(assistant.config.dig('assistant_migration', 'faq_document_candidates')).to contain_exactly(faq_document_candidate)
|
||||
expect(assistant.config.dig('assistant_migration', 'needs_review')).to contain_exactly(
|
||||
'Pricing details are missing because factual details are absent from the source instructions.'
|
||||
)
|
||||
expect(assistant.response_guidelines).to include(
|
||||
'For account-specific billing issues, collect the invoice number and summarize the issue before escalating.'
|
||||
)
|
||||
expect(assistant.response_guidelines).not_to include(faq_document_candidate['answer'])
|
||||
expect(assistant.scenarios.count).to eq(0)
|
||||
end
|
||||
|
||||
it 'rejects stale drafts whose FAQ candidates use the old string format' do
|
||||
stale_draft = draft.merge(faq_document_candidates: ['Support is available Monday to Friday.'])
|
||||
|
||||
expect do
|
||||
described_class.new(assistant: assistant, draft: stale_draft, dry_run: false).perform
|
||||
end.to raise_error(ArgumentError, 'FAQ document candidates must be question and answer objects')
|
||||
|
||||
expect(assistant.reload.config).not_to have_key('assistant_migration')
|
||||
end
|
||||
|
||||
it 'preserves original values in migration config before applying classifier output' do
|
||||
assistant.update!(
|
||||
description: 'Existing assistant description.',
|
||||
response_guidelines: ['Use plain language.'],
|
||||
guardrails: ['Do not disclose internal notes.']
|
||||
)
|
||||
|
||||
described_class.new(assistant: assistant, draft: draft, dry_run: false).perform
|
||||
|
||||
assistant.reload
|
||||
expect(assistant.description).to eq('Support assistant for Test Product.')
|
||||
expect(assistant.response_guidelines).to include('Be concise.')
|
||||
expect(assistant.guardrails).to eq(['Do not guess.'])
|
||||
expect(assistant.config.dig('assistant_migration', 'original_values')).to include(
|
||||
'name' => assistant.name,
|
||||
'description' => 'Existing assistant description.',
|
||||
'config' => { 'product_name' => 'Test Product', 'instructions' => 'Legacy V1 custom instructions.' },
|
||||
'response_guidelines' => ['Use plain language.'],
|
||||
'guardrails' => ['Do not disclose internal notes.']
|
||||
)
|
||||
end
|
||||
|
||||
it 'rejects an oversized assistant description from a stale draft' do
|
||||
long_context = 'This assistant supports a very broad product surface with many long details. ' * 10
|
||||
original_description = assistant.description
|
||||
|
||||
expect do
|
||||
described_class.new(
|
||||
assistant: assistant,
|
||||
draft: draft.merge(business_product_context: [long_context]),
|
||||
dry_run: false
|
||||
).perform
|
||||
end.to raise_error(ArgumentError, 'Assistant description exceeds 500 characters')
|
||||
|
||||
expect(assistant.reload.description).to eq(original_description)
|
||||
end
|
||||
end
|
||||
end
|
||||
Reference in New Issue
Block a user