Extractors & Evaluation

Extractor settings define how Reflexio extracts profiles, collects playbook entries, evaluates agent success, and asynchronously tags their persisted results. Prompt customization is optional; Reflexio ships with defaults so you can publish interactions before tuning extraction behavior.

Profile Extractor Configuration

The profile extractor automatically generates user profiles from interactions. By default, Reflexio looks for durable user facts, preferences, goals, and constraints. Configure the extractor only when you want to narrow what information to capture or add tags.

from reflexio.models.config_schema import ProfileExtractorConfig

profile_config = ProfileExtractorConfig(
    # Optional: override the default profile extraction prompt
    extraction_definition_prompt="""
Extract the following user information:
- Name and contact details
- Preferences and interests
- Goals and intent
""",
    # Optional: Context about the interaction type
    context_prompt="""
This is a conversation between a sales agent and a potential customer.
Extract any relevant customer information.
""",
    # Optional: How to categorize extracted profiles
    tagging_definition_prompt="""
Categorize extracted profiles as one of:
- 'basic_info': Name, contact, demographics
- 'preferences': Likes, dislikes, style preferences
- 'intent': Goals, purchase intent, timeline
""",
    # Optional: Only process interactions from these sources
    request_sources_enabled=["chat", "email"],
    # Optional: Require manual triggering instead of auto extraction
    manual_trigger=False
)

Optional: Configure the Profile Extractor

Configure one profile extractor with the scope you want to capture. Use request_sources_enabled and manual_trigger when you need narrower control over when it runs.

profile_config = ProfileExtractorConfig(
    extraction_definition_prompt="Extract product preferences, style choices, budget range",
    tagging_definition_prompt="choose from 'style_preference' or 'budget_preference'",
    request_sources_enabled=["chat"]
)

config.profile_extractor_config = profile_config

# Later, run the configured extractor manually:
client.rerun_profile_generation(
    user_id="user_123",
    wait_for_response=True
)

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/rerun_profile_generation" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "user_id": "user_123"
}
JSON

extractor_names is still accepted by rerun APIs for older clients, but it no longer selects or skips extractors. Reflexio runs the configured singleton profile extractor when it is enabled.

Agent Playbook Configuration

Playbook configuration defines how Reflexio learns from user interactions to improve agent behavior. By default, Reflexio looks for durable agent improvement signals in published interactions. Configure the playbook extractor only when you want a narrower playbook focus, custom tags, or different aggregation thresholds.

The playbook system works in two stages:

1. User Playbooks - Extracted from each interaction and stored per user/agent version
2. Agent Playbooks - Consolidated from multiple user playbooks into actionable insights for agent improvement

from reflexio.models.config_schema import (
    UserPlaybookExtractorConfig,
    PlaybookAggregatorConfig
)

playbook_config = UserPlaybookExtractorConfig(
    # Optional: override the default playbook extraction prompt
    extraction_definition_prompt="""
Analyze the interaction and extract playbook entries about:
- Was the customer satisfied with the response?
- What could have been done better?
- Any specific complaints or praise?
""",
    # Optional: How to categorize the playbook entries
    tagging_definition_prompt="""
Rate satisfaction: 'positive', 'neutral', 'negative'
""",
    # Optional: When to aggregate user playbooks
    aggregation_config=PlaybookAggregatorConfig(
        min_cluster_size=5,
        reaggregation_trigger_count=3
    )
)

Set it on config.user_playbook_extractor_config before saving. The Python type name PlaybookConfig remains a deprecated alias for UserPlaybookExtractorConfig. Historical configuration fields playbook_name, playbook_definition_prompt, and playbook_aggregator_config require explicit conversion; normal reads and writes reject them.

UserPlaybookExtractorConfig

Playbook prompts use agent_context_prompt and extraction_definition_prompt. The former playbook context_prompt setting never affected extraction and is retired; old saved documents remain readable. Raw upgrades preserve the unused text as opaque data; typed configuration no longer exposes or serializes it. Profile extraction retains its independent context_prompt.

PlaybookAggregatorConfig

Controls when user playbooks are aggregated into consolidated agent playbooks.

Aggregation is enabled by default whenever user-playbook extraction is enabled. Stored configurations where aggregation_config is missing or null are normalized to the defaults above when loaded. To disable only aggregation, set min_cluster_size=1; setting the entire user-playbook extractor to null disables extraction as well.

Example: With min_cluster_size=5 and reaggregation_trigger_count=3:

• First aggregation runs after 5 user playbooks are collected
• Re-aggregation runs every 3 new user playbooks (at 8, 11, 14, etc.)

Agent Success Configuration

Success configuration defines how Reflexio evaluates whether the agent achieved its goals in each session. It is enabled by default with 5% deterministic session sampling.

Best Practice: Define success based on user outcomes and goals, not technical implementation details. Focus on what the user accomplished rather than how the agent responded.

For the full publishing and analysis workflow, including evaluation_only=True, source-set comparison, shadow responses, on-demand grading, and regenerate jobs, see Evaluating Agent Performance.

from reflexio.models.config_schema import (
    AgentSuccessConfig,
    ToolUseConfig
)

success_config = AgentSuccessConfig(
    # Required: Define what success looks like (focus on user outcomes)
    success_definition_prompt="""
Evaluate if the agent successfully:
1. Understood the customer's booking request
2. Provided accurate availability information
3. Completed the booking or explained next steps
4. Left the customer satisfied

A successful interaction ends with either:
- A confirmed booking
- Clear next steps agreed upon
- Customer explicitly stating they're satisfied
""",
    # Optional: How to categorize outcomes
    tagging_definition_prompt="""
Classify outcome as:
- 'booking_completed': Customer completed a booking
- 'booking_pending': Customer will return to complete
- 'booking_cancelled': Customer decided not to book
- 'information_only': Customer was just browsing
""",
    # Optional: Evaluate only a portion of sessions (0.5 = 50%; default is 0.05)
    sampling_rate=0.5,
    # Optional: Give evaluation-only sessions different success-judge coverage.
    # None (the default) dynamically inherits sampling_rate.
    evaluation_only_sampling_rate=1.0,
)

# Configure tools the agent can use at the Config level
# (shared across success evaluation and playbook extraction)
config.tool_can_use = [
    ToolUseConfig(
        tool_name="check_availability",
        tool_description="Check room/service availability for given dates"
    ),
    ToolUseConfig(
        tool_name="create_booking",
        tool_description="Create a new booking for the customer"
    )
]

AgentSuccessConfig

ToolUseConfig

Describes tools available to the agent. This provides the evaluator with context about what actions were possible during the interaction.

See Evaluating Agent Performance for end-to-end examples that use this configuration.

Runtime identities are fixed to profile, playbook, and agent_success; omit retired identity fields from new configuration. Historical playbook/feedback field aliases require explicit conversion before normal reads or writes. Historical result names remain unchanged.

Upgrading saved configuration

Normal constructors, API writes, and stored reads require current configuration field names. For an older local file, run reflexio config upgrade-file old.json --output canonical.json to write a canonical copy while retaining the original source bytes. Managed and self-host native rows require the qualified operator conversion workflow before a strict-reader deployment.

Pending information tools

The organization lifetime pending_tool_call_config.pending_ttl_seconds controls pending-information tools. Zero (default) disables new calls; positive seconds select their exact lifetime. Per-tool TTL overrides cannot reopen a zero base. Profile extraction can attach a known pending request; playbook extraction can also ask a new human question. The extraction loop still runs when these tools are off. Storage is required, and tool TTL, cached-answer freshness and deduplication remain independent settings.

The former global/cohort resumable_extraction_agent rollout policy is retired. Stock installations need no change: the old bundled file was globally enabled. A retained neutral file requires no rewrite. A non-neutral or ambiguous old feature_flags.json/.txt is refused before pending tools can run.

Before replacing a package or image containing a custom cohort policy, preserve its original file and org settings and complete adoption. The guard protects retained files; it cannot reconstruct a policy deleted by an upgrade. Set pending_tool_call_config.pending_ttl_seconds to zero for every previously denied org; keep allowed orgs' existing creation policy. Apply the same opt-out to any custom future-org defaults. Verify the effective settings and drain or refresh workers holding old config before removing the retired policy. Removing the file alone is not adoption. No automatic config migration or customer-data rewrite runs.

The deprecated feature_flags login/provisioning response metadata is empty. It is not an admission setting; use the organization configuration for pending tool availability. Authentication, roles and project/tenant checks still apply.