Reflexio Docs
Getting StartedConfiguration

Extractors & Evaluation

Configure profile extractors, playbook aggregation, and agent success evaluation in Reflexio.

Extractors & Evaluation

Extractor settings define how Reflexio extracts profiles, collects playbook entries, and evaluates agent success.

Profile Extractor Configuration

The profile extractor automatically generates user profiles from interactions. It defines what information to look for and how to categorize it.

from reflexio.models.config_schema import ProfileExtractorConfig

profile_config = ProfileExtractorConfig(
    # Required: What information to extract
    extraction_definition_prompt="""
Extract the following user information:
- Name and contact details
- Preferences and interests
- Goals and intent
""",
    # Optional: Context about the interaction type
    context_prompt="""
This is a conversation between a sales agent and a potential customer.
Extract any relevant customer information.
""",
    # Optional: How to categorize extracted profiles
    metadata_definition_prompt="""
Categorize extracted profiles as one of:
- 'basic_info': Name, contact, demographics
- 'preferences': Likes, dislikes, style preferences
- 'intent': Goals, purchase intent, timeline
""",
    # Optional: Only process interactions from these sources
    request_sources_enabled=["chat", "email"],
    # Optional: Require manual triggering instead of auto extraction
    manual_trigger=False
)
FieldTypeRequiredDefaultDescription
extractor_namestrNoNoneDeprecated compatibility field. Accepted in legacy configs but ignored for runtime selection
extraction_definition_promptstrYes-Describes what user information to extract from interactions. Legacy field name profile_content_definition_prompt is accepted and auto-migrated
context_promptstrNoNoneProvides context about the interaction type to improve extraction accuracy
metadata_definition_promptstrNoNoneDefines metadata categories to attach to profiles for filtering and organization
should_extract_profile_prompt_overridestrNoNoneCustom logic to determine when profile extraction should run
request_sources_enabledlist[str]NoNoneLimits extraction to specific sources (e.g., "chat", "email"). If not set, processes all sources
manual_triggerboolNoFalseIf True, skip auto extraction and require manual triggering via rerun_profile_generation
window_size_overrideintNoNoneOverride the global window_size for this extractor
stride_size_overrideintNoNoneOverride the global stride_size for this extractor

Configuring the Profile Extractor

Configure one profile extractor with the scope you want to capture. Use request_sources_enabled and manual_trigger when you need narrower control over when it runs.

profile_config = ProfileExtractorConfig(
    extraction_definition_prompt="Extract product preferences, style choices, budget range",
    metadata_definition_prompt="choose from 'style_preference' or 'budget_preference'",
    request_sources_enabled=["chat"]
)

config.profile_extractor_config = profile_config

# Later, run the configured extractor manually:
client.rerun_profile_generation(
    user_id="user_123",
    wait_for_response=True
)

extractor_names is still accepted by rerun APIs for older clients, but it no longer selects or skips extractors. Reflexio runs the configured singleton profile extractor when it is enabled.


Agent Playbook Configuration

Playbook configuration defines how Reflexio learns from user interactions to improve agent behavior. The playbook system works in two stages:

  1. User Playbooks - Extracted from each interaction and stored per user/agent version
  2. Agent Playbooks - Consolidated from multiple user playbooks into actionable insights for agent improvement
from reflexio.models.config_schema import (
    UserPlaybookExtractorConfig,
    PlaybookAggregatorConfig
)

playbook_config = UserPlaybookExtractorConfig(
    # Required: What to extract from interactions
    extraction_definition_prompt="""
Analyze the interaction and extract playbook entries about:
- Was the customer satisfied with the response?
- What could have been done better?
- Any specific complaints or praise?
""",
    # Optional: How to categorize the playbook entries
    metadata_definition_prompt="""
Rate satisfaction: 'positive', 'neutral', 'negative'
""",
    # Optional: When to aggregate user playbooks
    aggregation_config=PlaybookAggregatorConfig(
        min_cluster_size=5,
        reaggregation_trigger_count=3
    )
)

Set it on config.user_playbook_extractor_config before saving. (PlaybookConfig remains as a deprecated alias, and the legacy field names playbook_name / playbook_definition_prompt / playbook_aggregator_config are still accepted and auto-migrated.)

UserPlaybookExtractorConfig

FieldTypeRequiredDescription
extractor_name / playbook_namestrNoDeprecated compatibility fields. Accepted in legacy configs but ignored for runtime selection
extraction_definition_promptstrYesDescribes what to extract from each interaction. Legacy field playbook_definition_prompt is accepted and auto-migrated
context_promptstrNoAdditional context for playbook extraction
metadata_definition_promptstrNoDefines metadata categories to attach to playbook entries for filtering and organization
aggregation_configPlaybookAggregatorConfigNoControls when user playbooks are aggregated into agent playbooks. Legacy name playbook_aggregator_config is auto-migrated
deduplication_configDeduplicationConfigNoControls deduplication against existing user playbooks
request_sources_enabledlist[str]NoOnly extract from these request sources. If not set, extracts from all sources
window_size_overrideintNoOverride the global window_size for this extractor
stride_size_overrideintNoOverride the global stride_size for this extractor

PlaybookAggregatorConfig

Controls when user playbooks are aggregated into consolidated agent playbooks.

FieldTypeDefaultDescription
min_cluster_sizeint2Minimum user playbooks required before first aggregation runs
reaggregation_trigger_countint2Number of new user playbooks that trigger re-aggregation
clustering_similarityfloat0.3Cosine similarity threshold for clustering (0.0–1.0). Higher = tighter clusters
direction_overlap_thresholdfloat0.6Token overlap threshold for grouping playbooks by direction (0.0–1.0)

Example: With min_cluster_size=5 and reaggregation_trigger_count=3:

  • First aggregation runs after 5 user playbooks are collected
  • Re-aggregation runs every 3 new user playbooks (at 8, 11, 14, etc.)

Agent Success Configuration

Success configuration defines how Reflexio evaluates whether the agent achieved its goals in each interaction.

Best Practice: Define success based on user outcomes and goals, not technical implementation details. Focus on what the user accomplished rather than how the agent responded.

from reflexio.models.config_schema import (
    AgentSuccessConfig,
    ToolUseConfig
)

success_config = AgentSuccessConfig(
    # Required: Define what success looks like (focus on user outcomes)
    success_definition_prompt="""
Evaluate if the agent successfully:
1. Understood the customer's booking request
2. Provided accurate availability information
3. Completed the booking or explained next steps
4. Left the customer satisfied

A successful interaction ends with either:
- A confirmed booking
- Clear next steps agreed upon
- Customer explicitly stating they're satisfied
""",
    # Optional: How to categorize outcomes
    metadata_definition_prompt="""
Classify outcome as:
- 'booking_completed': Customer completed a booking
- 'booking_pending': Customer will return to complete
- 'booking_cancelled': Customer decided not to book
- 'information_only': Customer was just browsing
""",
    # Optional: Evaluate only a portion of interactions (0.5 = 50%)
    sampling_rate=0.5
)

# Configure tools the agent can use at the Config level
# (shared across success evaluation and playbook extraction)
config.tool_can_use = [
    ToolUseConfig(
        tool_name="check_availability",
        tool_description="Check room/service availability for given dates"
    ),
    ToolUseConfig(
        tool_name="create_booking",
        tool_description="Create a new booking for the customer"
    )
]

AgentSuccessConfig

FieldTypeRequiredDefaultDescription
evaluation_namestrNoNoneDeprecated compatibility field. Accepted in legacy configs and requests but ignored for runtime selection
success_definition_promptstrYes-Describes what constitutes a successful interaction (focus on user outcomes)
metadata_definition_promptstrNoNoneDefines categories to classify evaluation outcomes
sampling_ratefloatNo1.0Fraction of interactions to evaluate (0.0-1.0). Use lower values during development to reduce API costs
window_size_overrideintNoNoneOverride the global window_size for this evaluator
stride_size_overrideintNoNoneOverride the global stride_size for this evaluator

ToolUseConfig

Describes tools available to the agent. This provides the evaluator with context about what actions were possible during the interaction.

FieldTypeDescription
tool_namestrName of the tool
tool_descriptionstrDescription of what the tool does