Aakash BakhleandGitHub
70f799ab35
# Pull Request Template ## Description Captain (v1) makes false promises by saying it will handoff but doesn't. This happens due to an exact string match comparison and the prompt gives the model a lot of responsibilities: - identity - what to respond - obey custom instructions - decide on tool calls This PR decouples responsibility, the core prompt responds, and an additional llm call evaluates if handoff was needed or not after that message. ## Type of change - [x] Bug fix (non-breaking change which fixes an issue) ## How Has This Been Tested? Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration. Locally ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] Any dependent changes have been merged and published in downstream modules
79 lines
2.2 KiB
Ruby
79 lines
2.2 KiB
Ruby
module Captain::ChatResponseHelper
|
|
include Integrations::LlmInstrumentationConstants
|
|
|
|
private
|
|
|
|
def build_response(response)
|
|
Rails.logger.debug { "#{self.class.name} Assistant: #{@assistant.id}, Received response #{response}" }
|
|
|
|
parsed = parse_json_response(response.content)
|
|
apply_credit_usage_metadata(parsed)
|
|
|
|
persist_message(parsed, 'assistant')
|
|
parsed
|
|
end
|
|
|
|
def parse_json_response(content)
|
|
content = content.gsub('```json', '').gsub('```', '')
|
|
content = content.strip
|
|
JSON.parse(content)
|
|
rescue JSON::ParserError => e
|
|
Rails.logger.error "#{self.class.name} Assistant: #{@assistant.id}, Error parsing JSON response: #{e.message}"
|
|
{ 'content' => content }
|
|
end
|
|
|
|
def apply_credit_usage_metadata(parsed_response)
|
|
return unless captain_v1_assistant?
|
|
|
|
OpenTelemetry::Trace.current_span.set_attribute(
|
|
format(ATTR_LANGFUSE_METADATA, 'credit_used'),
|
|
credit_used_for_response?(parsed_response).to_s
|
|
)
|
|
rescue StandardError => e
|
|
Rails.logger.warn "#{self.class.name} Assistant: #{@assistant.id}, Failed to set credit usage metadata: #{e.message}"
|
|
end
|
|
|
|
def credit_used_for_response?(parsed_response)
|
|
response = parsed_response['response']
|
|
|
|
# The classifier can still decide to hand off after this trace is written.
|
|
# Actual response usage is charged later in ResponseBuilderJob, so billing stays correct.
|
|
response.present? && response != 'conversation_handoff' && parsed_response['action'] != 'handoff'
|
|
end
|
|
|
|
def captain_v1_assistant?
|
|
feature_name == 'assistant' && !@assistant.account.feature_enabled?('captain_integration_v2')
|
|
end
|
|
|
|
def persist_thinking_message(tool_call)
|
|
return if @copilot_thread.blank?
|
|
|
|
tool_name = tool_call.name.to_s
|
|
|
|
persist_message(
|
|
{
|
|
'content' => "Using #{tool_name}",
|
|
'function_name' => tool_name
|
|
},
|
|
'assistant_thinking'
|
|
)
|
|
end
|
|
|
|
def persist_tool_completion
|
|
return if @copilot_thread.blank?
|
|
|
|
tool_call = @pending_tool_calls&.pop
|
|
return unless tool_call
|
|
|
|
tool_name = tool_call.name.to_s
|
|
|
|
persist_message(
|
|
{
|
|
'content' => "Completed #{tool_name}",
|
|
'function_name' => tool_name
|
|
},
|
|
'assistant_thinking'
|
|
)
|
|
end
|
|
end
|