Extractors & Evaluation
Configure profile extractors, playbook aggregation, and agent success evaluation in Reflexio.
Extractors & Evaluation
Extractor settings define how Reflexio extracts profiles, collects playbook entries, and evaluates agent success.
Profile Extractor Configuration
Profile extractors automatically generate user profiles from interactions. Each extractor defines what information to look for and how to categorize it.
from reflexio.models.config_schema import ProfileExtractorConfig
profile_config = ProfileExtractorConfig(
# Required: Unique name for this extractor
extractor_name="customer_info",
# Required: What information to extract
profile_content_definition_prompt="""
Extract the following user information:
- Name and contact details
- Preferences and interests
- Goals and intent
""",
# Optional: Context about the interaction type
context_prompt="""
This is a conversation between a sales agent and a potential customer.
Extract any relevant customer information.
""",
# Optional: How to categorize extracted profiles
metadata_definition_prompt="""
Categorize extracted profiles as one of:
- 'basic_info': Name, contact, demographics
- 'preferences': Likes, dislikes, style preferences
- 'intent': Goals, purchase intent, timeline
""",
# Optional: Only process interactions from these sources
request_sources_enabled=["chat", "email"],
# Optional: Require manual triggering instead of auto extraction
manual_trigger=False
)| Field | Type | Required | Default | Description |
|---|---|---|---|---|
extractor_name | str | Yes | - | Unique name identifier for this extractor. Used to reference specific extractors in rerun_profile_generation |
profile_content_definition_prompt | str | Yes | - | Describes what user information to extract from interactions |
context_prompt | str | No | None | Provides context about the interaction type to improve extraction accuracy |
metadata_definition_prompt | str | No | None | Defines metadata categories to attach to profiles for filtering and organization |
should_extract_profile_prompt_override | str | No | None | Custom logic to determine when profile extraction should run |
request_sources_enabled | list[str] | No | None | Limits extraction to specific sources (e.g., "chat", "email"). If not set, processes all sources |
manual_trigger | bool | No | False | If True, skip auto extraction and require manual triggering via rerun_profile_generation |
Multiple Profile Extractors
Configure multiple extractors when you need to capture different types of information separately. Each extractor runs independently and can have its own source filters.
# Extractor for demographics - runs on all sources
demographics_config = ProfileExtractorConfig(
extractor_name="demographics",
profile_content_definition_prompt="Extract name, age, location, occupation",
)
# Extractor for preferences - only from chat interactions
preferences_config = ProfileExtractorConfig(
extractor_name="preferences",
profile_content_definition_prompt="Extract product preferences, style choices, budget range",
metadata_definition_prompt="choose from 'style_preference' or 'budget_preference'",
request_sources_enabled=["chat"]
)
# Extractor that requires manual triggering
sentiment_config = ProfileExtractorConfig(
extractor_name="sentiment_analysis",
profile_content_definition_prompt="Extract user sentiment and emotional state",
manual_trigger=True # Only runs when explicitly triggered
)
config.profile_extractor_configs = [demographics_config, preferences_config, sentiment_config]
# Later, run only specific extractors manually:
client.rerun_profile_generation(
user_id="user_123",
extractor_names=["sentiment_analysis"],
wait_for_response=True
)Agent Playbook Configuration
Playbook configuration defines how Reflexio learns from user interactions to improve agent behavior. The playbook system works in two stages:
- User Playbooks - Extracted from each interaction and stored per user/agent version
- Agent Playbooks - Consolidated from multiple user playbooks into actionable insights for agent improvement
from reflexio.models.config_schema import (
PlaybookConfig,
PlaybookAggregatorConfig
)
playbook_config = PlaybookConfig(
# Required: Unique name for this playbook type
playbook_name="customer_satisfaction",
# Required: What to extract from interactions
playbook_definition_prompt="""
Analyze the interaction and extract playbook entries about:
- Was the customer satisfied with the response?
- What could have been done better?
- Any specific complaints or praise?
""",
# Optional: How to categorize the playbook entries
metadata_definition_prompt="""
Rate satisfaction: 'positive', 'neutral', 'negative'
""",
# Optional: When to aggregate user playbooks
playbook_aggregator_config=PlaybookAggregatorConfig(
min_cluster_size=5,
reaggregation_trigger_count=3
)
)PlaybookConfig
| Field | Type | Required | Description |
|---|---|---|---|
playbook_name | str | Yes | Unique identifier for this playbook type |
playbook_definition_prompt | str | Yes | Describes what to extract from each interaction |
metadata_definition_prompt | str | No | Defines metadata categories to attach to playbook entries for filtering and organization |
playbook_aggregator_config | PlaybookAggregatorConfig | No | Controls when user playbooks are aggregated into agent playbooks |
PlaybookAggregatorConfig
Controls when user playbooks are aggregated into consolidated agent playbooks.
| Field | Type | Default | Description |
|---|---|---|---|
min_cluster_size | int | 2 | Minimum user playbooks required before first aggregation runs |
reaggregation_trigger_count | int | 2 | Number of new user playbooks that trigger re-aggregation |
Example: With min_cluster_size=5 and reaggregation_trigger_count=3:
- First aggregation runs after 5 user playbooks are collected
- Re-aggregation runs every 3 new user playbooks (at 8, 11, 14, etc.)
Agent Success Configuration
Success configuration defines how Reflexio evaluates whether the agent achieved its goals in each interaction.
Best Practice: Define success based on user outcomes and goals, not technical implementation details. Focus on what the user accomplished rather than how the agent responded.
from reflexio.models.config_schema import (
AgentSuccessConfig,
ToolUseConfig
)
success_config = AgentSuccessConfig(
# Required: Unique name for this evaluation
evaluation_name="booking_success",
# Required: Define what success looks like (focus on user outcomes)
success_definition_prompt="""
Evaluate if the agent successfully:
1. Understood the customer's booking request
2. Provided accurate availability information
3. Completed the booking or explained next steps
4. Left the customer satisfied
A successful interaction ends with either:
- A confirmed booking
- Clear next steps agreed upon
- Customer explicitly stating they're satisfied
""",
# Optional: How to categorize outcomes
metadata_definition_prompt="""
Classify outcome as:
- 'booking_completed': Customer completed a booking
- 'booking_pending': Customer will return to complete
- 'booking_cancelled': Customer decided not to book
- 'information_only': Customer was just browsing
""",
# Optional: Evaluate only a portion of interactions (0.5 = 50%)
sampling_rate=0.5
)
# Configure tools the agent can use at the Config level
# (shared across success evaluation and playbook extraction)
config.tool_can_use = [
ToolUseConfig(
tool_name="check_availability",
tool_description="Check room/service availability for given dates"
),
ToolUseConfig(
tool_name="create_booking",
tool_description="Create a new booking for the customer"
)
]AgentSuccessConfig
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
evaluation_name | str | Yes | - | Unique identifier for this evaluation type |
success_definition_prompt | str | Yes | - | Describes what constitutes a successful interaction (focus on user outcomes) |
metadata_definition_prompt | str | No | None | Defines categories to classify evaluation outcomes |
sampling_rate | float | No | 1.0 | Fraction of interactions to evaluate (0.0-1.0). Use lower values during development to reduce API costs |
ToolUseConfig
Describes tools available to the agent. This provides the evaluator with context about what actions were possible during the interaction.
| Field | Type | Description |
|---|---|---|
tool_name | str | Name of the tool |
tool_description | str | Description of what the tool does |