Extractors & Evaluation
Configure profile extractors, playbook aggregation, and agent success evaluation in Reflexio.
Extractors & Evaluation
Extractor settings define how Reflexio extracts profiles, collects playbook entries, and evaluates agent success.
Profile Extractor Configuration
The profile extractor automatically generates user profiles from interactions. It defines what information to look for and how to categorize it.
from reflexio.models.config_schema import ProfileExtractorConfig
profile_config = ProfileExtractorConfig(
# Required: What information to extract
extraction_definition_prompt="""
Extract the following user information:
- Name and contact details
- Preferences and interests
- Goals and intent
""",
# Optional: Context about the interaction type
context_prompt="""
This is a conversation between a sales agent and a potential customer.
Extract any relevant customer information.
""",
# Optional: How to categorize extracted profiles
metadata_definition_prompt="""
Categorize extracted profiles as one of:
- 'basic_info': Name, contact, demographics
- 'preferences': Likes, dislikes, style preferences
- 'intent': Goals, purchase intent, timeline
""",
# Optional: Only process interactions from these sources
request_sources_enabled=["chat", "email"],
# Optional: Require manual triggering instead of auto extraction
manual_trigger=False
)| Field | Type | Required | Default | Description |
|---|---|---|---|---|
extractor_name | str | No | None | Deprecated compatibility field. Accepted in legacy configs but ignored for runtime selection |
extraction_definition_prompt | str | Yes | - | Describes what user information to extract from interactions. Legacy field name profile_content_definition_prompt is accepted and auto-migrated |
context_prompt | str | No | None | Provides context about the interaction type to improve extraction accuracy |
metadata_definition_prompt | str | No | None | Defines metadata categories to attach to profiles for filtering and organization |
should_extract_profile_prompt_override | str | No | None | Custom logic to determine when profile extraction should run |
request_sources_enabled | list[str] | No | None | Limits extraction to specific sources (e.g., "chat", "email"). If not set, processes all sources |
manual_trigger | bool | No | False | If True, skip auto extraction and require manual triggering via rerun_profile_generation |
window_size_override | int | No | None | Override the global window_size for this extractor |
stride_size_override | int | No | None | Override the global stride_size for this extractor |
Configuring the Profile Extractor
Configure one profile extractor with the scope you want to capture. Use request_sources_enabled and manual_trigger when you need narrower control over when it runs.
profile_config = ProfileExtractorConfig(
extraction_definition_prompt="Extract product preferences, style choices, budget range",
metadata_definition_prompt="choose from 'style_preference' or 'budget_preference'",
request_sources_enabled=["chat"]
)
config.profile_extractor_config = profile_config
# Later, run the configured extractor manually:
client.rerun_profile_generation(
user_id="user_123",
wait_for_response=True
)extractor_names is still accepted by rerun APIs for older clients, but it no longer selects or skips extractors. Reflexio runs the configured singleton profile extractor when it is enabled.
Agent Playbook Configuration
Playbook configuration defines how Reflexio learns from user interactions to improve agent behavior. The playbook system works in two stages:
- User Playbooks - Extracted from each interaction and stored per user/agent version
- Agent Playbooks - Consolidated from multiple user playbooks into actionable insights for agent improvement
from reflexio.models.config_schema import (
UserPlaybookExtractorConfig,
PlaybookAggregatorConfig
)
playbook_config = UserPlaybookExtractorConfig(
# Required: What to extract from interactions
extraction_definition_prompt="""
Analyze the interaction and extract playbook entries about:
- Was the customer satisfied with the response?
- What could have been done better?
- Any specific complaints or praise?
""",
# Optional: How to categorize the playbook entries
metadata_definition_prompt="""
Rate satisfaction: 'positive', 'neutral', 'negative'
""",
# Optional: When to aggregate user playbooks
aggregation_config=PlaybookAggregatorConfig(
min_cluster_size=5,
reaggregation_trigger_count=3
)
)Set it on config.user_playbook_extractor_config before saving. (PlaybookConfig remains as a deprecated alias, and the legacy field names playbook_name / playbook_definition_prompt / playbook_aggregator_config are still accepted and auto-migrated.)
UserPlaybookExtractorConfig
| Field | Type | Required | Description |
|---|---|---|---|
extractor_name / playbook_name | str | No | Deprecated compatibility fields. Accepted in legacy configs but ignored for runtime selection |
extraction_definition_prompt | str | Yes | Describes what to extract from each interaction. Legacy field playbook_definition_prompt is accepted and auto-migrated |
context_prompt | str | No | Additional context for playbook extraction |
metadata_definition_prompt | str | No | Defines metadata categories to attach to playbook entries for filtering and organization |
aggregation_config | PlaybookAggregatorConfig | No | Controls when user playbooks are aggregated into agent playbooks. Legacy name playbook_aggregator_config is auto-migrated |
deduplication_config | DeduplicationConfig | No | Controls deduplication against existing user playbooks |
request_sources_enabled | list[str] | No | Only extract from these request sources. If not set, extracts from all sources |
window_size_override | int | No | Override the global window_size for this extractor |
stride_size_override | int | No | Override the global stride_size for this extractor |
PlaybookAggregatorConfig
Controls when user playbooks are aggregated into consolidated agent playbooks.
| Field | Type | Default | Description |
|---|---|---|---|
min_cluster_size | int | 2 | Minimum user playbooks required before first aggregation runs |
reaggregation_trigger_count | int | 2 | Number of new user playbooks that trigger re-aggregation |
clustering_similarity | float | 0.3 | Cosine similarity threshold for clustering (0.0–1.0). Higher = tighter clusters |
direction_overlap_threshold | float | 0.6 | Token overlap threshold for grouping playbooks by direction (0.0–1.0) |
Example: With min_cluster_size=5 and reaggregation_trigger_count=3:
- First aggregation runs after 5 user playbooks are collected
- Re-aggregation runs every 3 new user playbooks (at 8, 11, 14, etc.)
Agent Success Configuration
Success configuration defines how Reflexio evaluates whether the agent achieved its goals in each interaction.
Best Practice: Define success based on user outcomes and goals, not technical implementation details. Focus on what the user accomplished rather than how the agent responded.
from reflexio.models.config_schema import (
AgentSuccessConfig,
ToolUseConfig
)
success_config = AgentSuccessConfig(
# Required: Define what success looks like (focus on user outcomes)
success_definition_prompt="""
Evaluate if the agent successfully:
1. Understood the customer's booking request
2. Provided accurate availability information
3. Completed the booking or explained next steps
4. Left the customer satisfied
A successful interaction ends with either:
- A confirmed booking
- Clear next steps agreed upon
- Customer explicitly stating they're satisfied
""",
# Optional: How to categorize outcomes
metadata_definition_prompt="""
Classify outcome as:
- 'booking_completed': Customer completed a booking
- 'booking_pending': Customer will return to complete
- 'booking_cancelled': Customer decided not to book
- 'information_only': Customer was just browsing
""",
# Optional: Evaluate only a portion of interactions (0.5 = 50%)
sampling_rate=0.5
)
# Configure tools the agent can use at the Config level
# (shared across success evaluation and playbook extraction)
config.tool_can_use = [
ToolUseConfig(
tool_name="check_availability",
tool_description="Check room/service availability for given dates"
),
ToolUseConfig(
tool_name="create_booking",
tool_description="Create a new booking for the customer"
)
]AgentSuccessConfig
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
evaluation_name | str | No | None | Deprecated compatibility field. Accepted in legacy configs and requests but ignored for runtime selection |
success_definition_prompt | str | Yes | - | Describes what constitutes a successful interaction (focus on user outcomes) |
metadata_definition_prompt | str | No | None | Defines categories to classify evaluation outcomes |
sampling_rate | float | No | 1.0 | Fraction of interactions to evaluate (0.0-1.0). Use lower values during development to reduce API costs |
window_size_override | int | No | None | Override the global window_size for this evaluator |
stride_size_override | int | No | None | Override the global stride_size for this evaluator |
ToolUseConfig
Describes tools available to the agent. This provides the evaluator with context about what actions were possible during the interaction.
| Field | Type | Description |
|---|---|---|
tool_name | str | Name of the tool |
tool_description | str | Description of what the tool does |