Concepts

Shadow Content

The shadow_content field allows you to capture an alternative agent response alongside the production response, enabling A/B testing and comparison of different agent behaviors.

How it works:

This separation allows you to:

• Compare agent versions: Test how a new agent version would respond without affecting production
• Evaluate profile effectiveness: See how different user profile configurations affect agent responses
• A/B test playbook integration: Compare responses with and without specific agent playbook entries applied

When to use shadow_content:

• Testing a new agent version's responses against the current production version
• Evaluating how different user profile configurations affect agent behavior
• Comparing responses with different playbook/prompt improvements applied
• Running shadow evaluations without impacting the user experience

Example: Comparing Agent Responses

# Agent v1.0 is in production, testing v2.0 response in shadow
client.publish_interaction(
    user_id="user_123",
    interactions=[
        {
            "role": "User",
            "content": "I would like to get a refund"
        },
        {
            "role": "Agent",
            # Production response (v1.0) - generic, asks for info
            "content": "Happy to help you with that. What is your email or phone number?",
            # Shadow response (v2.0) - personalized, uses known user data
            "shadow_content": "Happy to assist you with that. I can see your phone number is 123-456-7890. Is that the right number to send the refund confirmation to?"
        }
    ],
    session_id="support_session_001",
    agent_version="v1.0"
)

In this example:

• content stores the production response from agent v1.0
• shadow_content captures how agent v2.0 (with access to user profiles) would have responded
• Profile extraction and evaluation use shadow_content to assess the v2.0 approach
• You can compare evaluation results to decide whether to promote v2.0 to production

Expert Content

The expert_content field lets you provide a human expert's ideal response alongside the agent's actual response. Reflexio compares the two and generates actionable playbook entries -- SOPs (Situation/Trigger, Instruction, Pitfall) -- so your agent learns to match expert behavior.

How it works:

When expert_content is present on any interaction in a session, Reflexio automatically switches to an expert playbook extraction pipeline that:

1. Pairs each expert response with the agent's response and the preceding user question
2. Analyzes alignment gaps between the agent and expert
3. Generates structured SOPs: what triggered the gap, what the agent should do instead, and what pitfalls to avoid

When to use expert_content:

• Training from gold-standard responses: When domain experts review agent conversations and provide ideal answers
• Quality assurance workflows: Compare production agent output against expert-vetted responses
• Bootstrapping new agents: Seed playbook entries from existing expert knowledge before you have enough user interactions
• Compliance or accuracy-critical domains: Ensure agent responses meet expert standards (legal, medical, financial)

Example: Learning from Expert Responses

client.publish_interaction(
    user_id="user_123",
    interactions=[
        {
            "role": "User",
            "content": "What is your return policy for electronics?"
        },
        {
            "role": "Agent",
            # Agent's actual response — incomplete
            "content": "You can return electronics within 30 days of purchase.",
            # Expert's ideal response — comprehensive
            "expert_content": (
                "Electronics can be returned within 30 days of purchase with the original receipt. "
                "Items must be in original packaging and include all accessories. "
                "Opened software and personal care electronics are final sale. "
                "For defective items, the return window extends to 90 days. "
                "Refunds are processed to the original payment method within 5-7 business days."
            )
        }
    ],
    session_id="expert_review_001",
    source="expert_review",
    agent_version="v1.0"
)

In this example:

• content stores the agent's actual (incomplete) response
• expert_content provides the expert's comprehensive ideal response
• Reflexio compares the two and extracts playbook entries like: "Agent should mention packaging requirements, exceptions for opened software, extended window for defective items, and refund processing timeline"
• The extracted entries enter the standard aggregation pipeline, improving the agent for all users