Reflexio Docs
Getting StartedConfiguration

Extractors & Evaluation

Configure profile extractors, playbook aggregation, and agent success evaluation in Reflexio.

Extractors & Evaluation

Extractor settings define how Reflexio extracts profiles, collects playbook entries, and evaluates agent success.

Profile Extractor Configuration

Profile extractors automatically generate user profiles from interactions. Each extractor defines what information to look for and how to categorize it.

from reflexio.models.config_schema import ProfileExtractorConfig

profile_config = ProfileExtractorConfig(
    # Required: Unique name for this extractor
    extractor_name="customer_info",
    # Required: What information to extract
    profile_content_definition_prompt="""
Extract the following user information:
- Name and contact details
- Preferences and interests
- Goals and intent
""",
    # Optional: Context about the interaction type
    context_prompt="""
This is a conversation between a sales agent and a potential customer.
Extract any relevant customer information.
""",
    # Optional: How to categorize extracted profiles
    metadata_definition_prompt="""
Categorize extracted profiles as one of:
- 'basic_info': Name, contact, demographics
- 'preferences': Likes, dislikes, style preferences
- 'intent': Goals, purchase intent, timeline
""",
    # Optional: Only process interactions from these sources
    request_sources_enabled=["chat", "email"],
    # Optional: Require manual triggering instead of auto extraction
    manual_trigger=False
)
FieldTypeRequiredDefaultDescription
extractor_namestrYes-Unique name identifier for this extractor. Used to reference specific extractors in rerun_profile_generation
profile_content_definition_promptstrYes-Describes what user information to extract from interactions
context_promptstrNoNoneProvides context about the interaction type to improve extraction accuracy
metadata_definition_promptstrNoNoneDefines metadata categories to attach to profiles for filtering and organization
should_extract_profile_prompt_overridestrNoNoneCustom logic to determine when profile extraction should run
request_sources_enabledlist[str]NoNoneLimits extraction to specific sources (e.g., "chat", "email"). If not set, processes all sources
manual_triggerboolNoFalseIf True, skip auto extraction and require manual triggering via rerun_profile_generation

Multiple Profile Extractors

Configure multiple extractors when you need to capture different types of information separately. Each extractor runs independently and can have its own source filters.

# Extractor for demographics - runs on all sources
demographics_config = ProfileExtractorConfig(
    extractor_name="demographics",
    profile_content_definition_prompt="Extract name, age, location, occupation",
)

# Extractor for preferences - only from chat interactions
preferences_config = ProfileExtractorConfig(
    extractor_name="preferences",
    profile_content_definition_prompt="Extract product preferences, style choices, budget range",
    metadata_definition_prompt="choose from 'style_preference' or 'budget_preference'",
    request_sources_enabled=["chat"]
)

# Extractor that requires manual triggering
sentiment_config = ProfileExtractorConfig(
    extractor_name="sentiment_analysis",
    profile_content_definition_prompt="Extract user sentiment and emotional state",
    manual_trigger=True  # Only runs when explicitly triggered
)

config.profile_extractor_configs = [demographics_config, preferences_config, sentiment_config]

# Later, run only specific extractors manually:
client.rerun_profile_generation(
    user_id="user_123",
    extractor_names=["sentiment_analysis"],
    wait_for_response=True
)

Agent Playbook Configuration

Playbook configuration defines how Reflexio learns from user interactions to improve agent behavior. The playbook system works in two stages:

  1. User Playbooks - Extracted from each interaction and stored per user/agent version
  2. Agent Playbooks - Consolidated from multiple user playbooks into actionable insights for agent improvement
from reflexio.models.config_schema import (
    PlaybookConfig,
    PlaybookAggregatorConfig
)

playbook_config = PlaybookConfig(
    # Required: Unique name for this playbook type
    playbook_name="customer_satisfaction",
    # Required: What to extract from interactions
    playbook_definition_prompt="""
Analyze the interaction and extract playbook entries about:
- Was the customer satisfied with the response?
- What could have been done better?
- Any specific complaints or praise?
""",
    # Optional: How to categorize the playbook entries
    metadata_definition_prompt="""
Rate satisfaction: 'positive', 'neutral', 'negative'
""",
    # Optional: When to aggregate user playbooks
    playbook_aggregator_config=PlaybookAggregatorConfig(
        min_cluster_size=5,
        reaggregation_trigger_count=3
    )
)

PlaybookConfig

FieldTypeRequiredDescription
playbook_namestrYesUnique identifier for this playbook type
playbook_definition_promptstrYesDescribes what to extract from each interaction
metadata_definition_promptstrNoDefines metadata categories to attach to playbook entries for filtering and organization
playbook_aggregator_configPlaybookAggregatorConfigNoControls when user playbooks are aggregated into agent playbooks

PlaybookAggregatorConfig

Controls when user playbooks are aggregated into consolidated agent playbooks.

FieldTypeDefaultDescription
min_cluster_sizeint2Minimum user playbooks required before first aggregation runs
reaggregation_trigger_countint2Number of new user playbooks that trigger re-aggregation

Example: With min_cluster_size=5 and reaggregation_trigger_count=3:

  • First aggregation runs after 5 user playbooks are collected
  • Re-aggregation runs every 3 new user playbooks (at 8, 11, 14, etc.)

Agent Success Configuration

Success configuration defines how Reflexio evaluates whether the agent achieved its goals in each interaction.

Best Practice: Define success based on user outcomes and goals, not technical implementation details. Focus on what the user accomplished rather than how the agent responded.

from reflexio.models.config_schema import (
    AgentSuccessConfig,
    ToolUseConfig
)

success_config = AgentSuccessConfig(
    # Required: Unique name for this evaluation
    evaluation_name="booking_success",
    # Required: Define what success looks like (focus on user outcomes)
    success_definition_prompt="""
Evaluate if the agent successfully:
1. Understood the customer's booking request
2. Provided accurate availability information
3. Completed the booking or explained next steps
4. Left the customer satisfied

A successful interaction ends with either:
- A confirmed booking
- Clear next steps agreed upon
- Customer explicitly stating they're satisfied
""",
    # Optional: How to categorize outcomes
    metadata_definition_prompt="""
Classify outcome as:
- 'booking_completed': Customer completed a booking
- 'booking_pending': Customer will return to complete
- 'booking_cancelled': Customer decided not to book
- 'information_only': Customer was just browsing
""",
    # Optional: Evaluate only a portion of interactions (0.5 = 50%)
    sampling_rate=0.5
)

# Configure tools the agent can use at the Config level
# (shared across success evaluation and playbook extraction)
config.tool_can_use = [
    ToolUseConfig(
        tool_name="check_availability",
        tool_description="Check room/service availability for given dates"
    ),
    ToolUseConfig(
        tool_name="create_booking",
        tool_description="Create a new booking for the customer"
    )
]

AgentSuccessConfig

FieldTypeRequiredDefaultDescription
evaluation_namestrYes-Unique identifier for this evaluation type
success_definition_promptstrYes-Describes what constitutes a successful interaction (focus on user outcomes)
metadata_definition_promptstrNoNoneDefines categories to classify evaluation outcomes
sampling_ratefloatNo1.0Fraction of interactions to evaluate (0.0-1.0). Use lower values during development to reduce API costs

ToolUseConfig

Describes tools available to the agent. This provides the evaluator with context about what actions were possible during the interaction.

FieldTypeDescription
tool_namestrName of the tool
tool_descriptionstrDescription of what the tool does