All

Evaluating Agent Performance

Choose between session A/B tests, response head-to-head comparisons, and retrieved-learning analysis.

Evaluating Agent Performance

Reflexio supports three complementary evaluation methods. Choose the method based on whether you want to measure session outcomes, compare two answers to the same question, or understand the learnings behind an answer.

All three methods use publish_interaction. Reflexio evaluates session success from the requests that share a session_id, using your success rubric plus the user turns, agent turns, and recorded tool use.

Evaluation is session-level

Session success is evaluated after a session becomes inactive. Head-to-head and retrieved-learning signals are attached to individual assistant interactions, then processed as part of that session evaluation.

Choose an Evaluation Method

Session cohortssource + evaluation_only

A/B testing

Measure whether Reflexio improves success across separate control and test sessions.

Publish: A stable source for each arm and evaluation_only=True on the no-Reflexio control arm.
Read: Success, corrections, turns, and escalation metrics for each source set.

One assistant turnshadow_content

Head-to-head comparison

Judge the served response and an alternate response against the same user question.

Publish: The served answer in content and the alternate answer in shadow_content.
Read: Regular wins, shadow wins, ties, and the per-turn comparison details.

Learning attributionretrieved_learnings

Retrieved-learning analysis

Measure whether each profile or playbook applied to an answer was relevant and helpful.

Publish: Every injected learning as a stable {kind, learning_id} reference on the assistant interaction.
Read: Per-learning relevance and positive, negative, or neutral impact verdicts.

A/B and head-to-head answer different questions

A/B testing compares outcomes across separate sessions. Head-to-head comparison evaluates two responses to the same question on one assistant interaction. The evaluation_only flag controls learning eligibility; it is not a cohort label. Use source to label cohorts.

Configure Evaluation

Configure one success rubric before using any evaluation method. The rubric should describe the observable conditions that make a complete session successful. Add tool_can_use when tool choice is part of that judgment.

from reflexio import ReflexioClient
from reflexio.models.config_schema import AgentSuccessConfig, ToolUseConfig

client = ReflexioClient()
config = client.get_config()

config.agent_success_config = AgentSuccessConfig(
    success_definition_prompt="""
Evaluate whether the agent resolved the user's task.

Success means:
- The agent understood the user's goal.
- The answer or action directly addressed that goal.
- Any required next step was clear.
- The user did not need to correct, repeat, or escalate the request.
""",
    request_sources_enabled=[
        "prod_with_reflexio",
        "prod_without_reflexio",
    ],
    sampling_rate=1.0,
    evaluation_only_sampling_rate=1.0,
    retrieved_learning_sampling_rate=1.0,
)

config.tool_can_use = [
    ToolUseConfig(
        tool_name="search_docs",
        tool_description="Search product documentation for grounded answers.",
    ),
    ToolUseConfig(
        tool_name="create_ticket",
        tool_description="Create a support ticket when follow-up is required.",
    ),
]

client.set_config(config)

curl -X GET "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_config" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -o reflexio-config.json

# Update agent_success_config in the downloaded config object, then send
# the complete object back to Reflexio.
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/set_config" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @reflexio-config.json

Field	What it controls
`success_definition_prompt`	The rubric used to decide whether the session succeeded.
`sampling_rate`	Session-success coverage for normal publish traffic. The default is `0.05`.
`evaluation_only_sampling_rate`	Optional session-success coverage for evaluation-only traffic. `null` inherits `sampling_rate`.
`retrieved_learning_sampling_rate`	Independent coverage for retrieved-learning judges. `null` inherits `sampling_rate`.
`request_sources_enabled`	Optional allowlist of `source` values eligible for evaluation.
`tool_can_use`	Root config list describing tools the agent could use.

Use 1.0 during a controlled launch or audit window when every eligible session should be graded. Lower the rates for ongoing production monitoring. Set agent_success_config=None to disable automatic session-success evaluation.

A/B Testing

A/B testing measures Reflexio's effect across separate sessions:

Arm	Agent behavior	Publish behavior
Control	Answer without Reflexio context.	Use a control `source` and set `evaluation_only=True`. The session is graded but excluded from profile and playbook learning.
Test	Retrieve Reflexio context, apply it, and serve the resulting response.	Use a test `source` and publish normally so the interaction can contribute to learning.

Assign sessions to arms at random and keep that assignment stable for the full session_id. Reflexio assigns the session to the source on its first request.

from reflexio import InteractionData, ReflexioClient

client = ReflexioClient()

# Control session: do not retrieve Reflexio context.
control_response = run_agent(user_message, context=[])
client.publish_interaction(
    user_id="user_123",
    session_id="session_control_001",
    source="prod_without_reflexio",
    agent_version="support-agent@2.1.0",
    evaluation_only=True,
    interactions=[
        InteractionData(role="User", content=user_message),
        InteractionData(role="Agent", content=control_response),
    ],
)

# Test session: retrieve Reflexio context and publish normally.
reflexio_context = client.search(user_message)
test_response = run_agent(user_message, context=reflexio_context)
client.publish_interaction(
    user_id="user_456",
    session_id="session_test_001",
    source="prod_with_reflexio",
    agent_version="support-agent@2.1.0",
    interactions=[
        InteractionData(role="User", content=user_message),
        InteractionData(role="Agent", content=test_response),
    ],
)

# Control session
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/publish_interaction" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "user_id": "user_123",
  "session_id": "session_control_001",
  "source": "prod_without_reflexio",
  "agent_version": "support-agent@2.1.0",
  "evaluation_only": true,
  "interaction_data_list": [
    {"role": "User", "content": "Can you help me reset my password?"},
    {"role": "Agent", "content": "Open Account Settings, then choose Security."}
  ]
}
JSON

# Test session
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/publish_interaction" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "user_id": "user_456",
  "session_id": "session_test_001",
  "source": "prod_with_reflexio",
  "agent_version": "support-agent@2.1.0",
  "interaction_data_list": [
    {"role": "User", "content": "Can you help me reset my password?"},
    {"role": "Agent", "content": "Open Account Settings, choose Security, then select Reset Password."}
  ]
}
JSON

Read the cohort comparison

Request labeled source sets from POST /api/get_evaluation_overview:

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_evaluation_overview" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "from_ts": 1782864000,
  "to_ts": 1785456000,
  "bucket": "day",
  "include_shadow": false,
  "source_sets": [
    {
      "label": "Control",
      "sources": ["prod_without_reflexio"]
    },
    {
      "label": "Reflexio",
      "sources": ["prod_with_reflexio"]
    }
  ]
}
JSON

Read each arm under source_set_comparison.sets. Metrics include session count, success rate, corrections, turns to resolution, escalation, and rule attribution. See GetEvaluationOverviewRequest for the complete request and response shapes.

Randomization determines what you can claim

With random session assignment, the source-set gap can support a causal lift measurement. If assignment depends on user type, geography, time, or another non-random rule, treat the comparison as observational.

Head-to-Head Comparison

Head-to-head comparison answers a narrower question: for this user message, which of two candidate responses was better?

Generate both responses from the same input. Put the response shown to the user in content and the alternate response in shadow_content. The labels describe payload position, not which response used Reflexio.

from reflexio import InteractionData, ReflexioClient

client = ReflexioClient()

served_response = run_agent_with_reflexio(user_message)
alternate_response = run_agent_without_reflexio(user_message)

client.publish_interaction(
    user_id="user_123",
    session_id="session_002",
    source="prod_with_reflexio",
    agent_version="support-agent@2.1.0",
    interactions=[
        InteractionData(role="User", content=user_message),
        InteractionData(
            role="Agent",
            content=served_response,
            shadow_content=alternate_response,
        ),
    ],
)

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/publish_interaction" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "user_id": "user_123",
  "session_id": "session_002",
  "source": "prod_with_reflexio",
  "agent_version": "support-agent@2.1.0",
  "interaction_data_list": [
    {
      "role": "User",
      "content": "Can I place the same order as last time?"
    },
    {
      "role": "Agent",
      "content": "Yes. Your last order was one barbecue chicken pizza. Would you like that again?",
      "shadow_content": "Yes. What would you like to order?"
    }
  ]
}
JSON

The comparison judge records whether the regular response won, the shadow response won, or the result was a tie. Use shadow_win_rate_trend from POST /api/get_evaluation_overview for aggregate win-rate analysis. Raw session results also expose regular_vs_shadow.

Evaluation details showing the regular and shadow responses side by side

Keep the two runs comparable: use the same user message, model settings, tool availability, and non-Reflexio context. Change only the Reflexio content or other treatment you intend to test.

Retrieved-Learning Analysis

Retrieved-learning analysis explains the contribution of the Reflexio context applied to an answer. Whenever your agent injects a profile, user playbook, or agent playbook, attach its stable identity to the assistant interaction.

Include every injected learning, even when its effect is unclear. Without these references, Reflexio can still judge overall session success but cannot attribute relevance or impact to individual learnings.

from reflexio import InteractionData, ReflexioClient

client = ReflexioClient()

client.publish_interaction(
    user_id="user_123",
    session_id="session_003",
    source="prod_with_reflexio",
    agent_version="support-agent@2.1.0",
    interactions=[
        InteractionData(
            role="User",
            content="Give me a deployment checklist.",
        ),
        InteractionData(
            role="Agent",
            content="1. Build. 2. Migrate. 3. Verify.",
            retrieved_learnings=[
                {"kind": "profile", "learning_id": "prof-abc123"},
                {"kind": "user_playbook", "learning_id": "42"},
                {"kind": "agent_playbook", "learning_id": "7"},
            ],
        ),
    ],
)

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/publish_interaction" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "user_id": "user_123",
  "session_id": "session_003",
  "source": "prod_with_reflexio",
  "agent_version": "support-agent@2.1.0",
  "interaction_data_list": [
    {
      "role": "User",
      "content": "Give me a deployment checklist."
    },
    {
      "role": "Agent",
      "content": "1. Build. 2. Migrate. 3. Verify.",
      "retrieved_learnings": [
        {"kind": "profile", "learning_id": "prof-abc123"},
        {"kind": "user_playbook", "learning_id": "42"},
        {"kind": "agent_playbook", "learning_id": "7"}
      ]
    }
  ]
}
JSON

Reflexio produces two verdicts for each resolvable learning occurrence:

Relevance: whether the learning applied to the target interaction and response.
Impact: whether the learning moved the response toward success (positive), away from success (negative), or did not materially change it (neutral).

The same learning used on two responses is evaluated separately against each response. Duplicate references on one response are deduplicated. Deleted learning rows are skipped because their content is no longer available to the judge.

Read learning verdicts

verdicts = client.get_retrieved_learning_evaluation_results(
    session_id="session_003",
    limit=100,
)

for verdict in verdicts.results:
    print(
        verdict.interaction_id,
        verdict.kind,
        verdict.learning_id,
        verdict.is_relevant,
        verdict.impact,
    )

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_retrieved_learning_evaluation_results" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"session_id": "session_003", "limit": 100}'

The Evaluation page groups verdicts by interaction so a response counts once in the displayed percentages. Read the RetrievedLearningEvaluationResult schema when you need per-learning reasons or timestamps.

How Evaluation Runs

When an eligible interaction is published:

Reflexio stores the request and interactions.
Session-success and retrieved-learning judges pass through independent deterministic sampling gates.
Reflexio waits for session inactivity, then evaluates the full session. The default inactivity delay is 10 minutes after its latest request.
Reflexio stores session results plus any shadow and retrieved-learning verdicts requested by the published data.

Publishing another request with the same session_id moves the scheduled evaluation later. Sampling happens once per session. A session is scheduled when either judge family samples it, and only the sampled judge families run.

Evaluation-only requests are stored and graded but excluded from profile extraction, playbook extraction, and aggregation. The flag requires a non-empty session_id, cannot be combined with force_extraction=True, and must remain consistent across all publishes in a session.

Grade or Regenerate Explicitly

Automatic evaluation waits for session inactivity. Use these endpoints when you need a result sooner or need to re-score existing sessions.

Action	API	Use it when
Grade one session nowImmediate	`POST /api/evaluations/grade_on_demand`Cached for 24 hours per session and agent version.	A UI, demo, launch check, or operations tool needs a result before the inactivity delay.
Re-score a time windowHistorical	`POST /api/evaluations/regenerateGET /api/evaluations/regenerate/{job_id}`Returns a job id for status polling.	The success rubric, evaluator model, prompt, or analysis window changed.

import time

grade = client.grade_on_demand(
    session_id="session_003",
    agent_version="support-agent@2.1.0",
)

job = client.regenerate_evaluations(
    from_ts=int(time.time()) - 7 * 24 * 60 * 60,
    to_ts=int(time.time()),
)

status = client.get_evaluation_regeneration_status(job.job_id)

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/grade_on_demand" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"session_id": "session_003", "agent_version": "support-agent@2.1.0"}'

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/regenerate" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data '{"from_ts": 1782864000, "to_ts": 1785456000}'

Inspect Session Results

Use get_agent_success_evaluation_results for raw session-level outcomes:

response = client.get_agent_success_evaluation_results(
    agent_version="support-agent@2.1.0",
    limit=100,
)

for result in response.agent_success_evaluation_results:
    print(
        result.session_id,
        result.is_success,
        result.failure_type,
        result.failure_reason,
        result.tags,
    )

Important fields include is_success, failure_type, failure_reason, number_of_correction_per_session, user_turns_to_resolution, is_escalated, tags, regular_vs_shadow, agent_version, user_id, and session_id.

When AgentSuccessConfig.tagging_definition_prompt is configured, Reflexio tags each persisted evaluation summary asynchronously. tags=None means tagging has not completed yet; tags=[] means the pass completed without a match. Tagging uses only the stored outcome summary, not the raw transcript.

number_of_correction_per_session is the judge's count of user turns that corrected or redirected an earlier agent response. A qualifying turn must identify an earlier response as incorrect, incomplete, insufficient, or misaligned and steer a revision. Topic continuity alone does not make a turn corrective: a new question, a separate deliverable, an ordinary follow-up, or an answer to the agent's clarification question does not count. Several issues raised in one user turn count once; later corrective turns count separately. The judge computes this independently from final success, so a successful session can still have one or more corrections.

Evaluation result expanded to show the failure reason and suggested improvement

API Map

Task	API or SDK surface
Publish any evaluation input	`publish_interaction`
Compare A/B source sets and shadow trends	`POST /api/get_evaluation_overview`
Read raw session outcomes	`get_agent_success_evaluation_results`
Read per-learning verdicts	`get_retrieved_learning_evaluation_results`
Grade one session immediately	`grade_on_demand`
Re-score a historical window	`regenerate_evaluations`