Measuring Reflexio's Impact

The Evaluation page reports quality at three levels:

• Session outcomes — whether evaluated conversations succeeded, needed corrections, took more turns, or escalated.
• Retrieved-learning effects — whether the profiles and playbooks supplied to a response were relevant and helped.
• Per-turn comparisons — whether a response using Reflexio beat an alternate response generated without retrieved context.

Start with the metric reference below to understand the numbers. The second half of this page covers integration, sampling, and the methodology required to make source-set comparisons meaningful.

How Filters and Windows Work

Metric Reference

Session outcomes

Every session-outcome metric is calculated from stored agent-success evaluation results. The configured success rubric defines what the judge considers success. The session judge also independently counts corrective user turns and determines escalation.

Correction count and final success are independent: a session may require one or more corrections and still end successfully. Rows evaluated before the judge-derived count was introduced retain their earlier values until they are regenerated. Regenerate every historical window used for a comparison or experiment after upgrading; do not mix legacy and regenerated rows in one analysis.

Success-rate and escalation deltas are shown in percentage points (pp). Corrections and turns deltas are absolute differences. The Source-set comparison applies these same formulas after grouping evaluated sessions by their first-request source.

Retrieved-learning effects

This section appears when published responses include retrieved_learnings. Reflexio judges each attached profile or playbook separately, then groups those verdicts by response so a response with many learnings does not outweigh one with a single learning.

Collective impact uses these response-level categories:

Expand Interaction verdicts on the dashboard to inspect the per-learning relevance, impact, and judge reasons behind these response-level numbers.

Per-turn comparison

These metrics compare each served response that used Reflexio with a shadow response produced without retrieved context.

Learning attribution and imported scorers

Learning attribution is observational: a positive net does not prove that the learning caused the successful outcome. Use a retrieval experiment when you need a causal comparison.

Retrieval A/B Experiments

Retrieval experiments measure the effect of serving learned context while keeping the rest of the production pipeline the same. Reflexio deterministically assigns each user to one of two arms:

Only retrieval is withheld. Do not set evaluation_only=True for the holdout arm: doing so would change the learning pipeline and make the arms less comparable.

Start and monitor the experiment from Experiments in the web portal, or use the client lifecycle methods. There can be only one active experiment per organization, experiment IDs cannot be reused, and stopped experiments remain available in history.

The results scorecard also compares average output tokens per published session. Reflexio stores a canonical token count for each newly published interaction and sums all non-user roles, including assistant, system, and tool content. Sessions containing older non-user interactions without a stored count are excluded from that average, and the scorecard shows measured-session coverage for each arm.

client.start_retrieval_experiment(
    experiment_id="support-agent-2026-08",
    holdout_percentage=10,
)

Search and publish attribution

Pass user_id to every learning-search request. Search does not require a session ID for assignment: the arm is stable for the combination of organization, experiment, and user. When an experiment is active, the response includes experiment_id and arm; a holdout response is successful but has no retrieved profiles or playbooks.

from reflexio import ReflexioClient
from reflexio.models.api_schema.domain.entities import InteractionData

client = ReflexioClient()

search = client.search(query=user_message, user_id=user.id)
agent_response = run_agent(
    user_message,
    profiles=search.profiles,
    agent_playbooks=search.agent_playbooks,
    user_playbooks=search.user_playbooks,
)

interactions = [
    InteractionData(role="user", content=user_message),
    InteractionData(role="assistant", content=agent_response),
]

client.publish_interaction(
    user_id=user.id,
    interactions=interactions,
    session_id=session_id,
    agent_version="v1",
    retrieval_experiment_id=(
        search.experiment.experiment_id if search.experiment else None
    ),
    retrieval_experiment_arm=(search.experiment.arm if search.experiment else None),
)

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/search" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "query": "<user_message>",
  "user_id": "user_123"
}
JSON

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/publish_interaction" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "user_id": "id",
  "interaction_data_list": [
    {
      "role": "user",
      "content": "<user_message>"
    },
    {
      "role": "assistant",
      "content": "<run_agent_result>"
    }
  ],
  "session_id": "<session_id>",
  "agent_version": "v1",
  "retrieval_experiment_id": "<experiment_id returned by search>",
  "retrieval_experiment_arm": "<arm returned by search>"
}
JSON

If no experiment is active, experiment is absent and both publish fields should be omitted. If it is present, send both fields together. Reflexio rejects an unknown experiment, a single missing field, or an arm that does not match the user's deterministic assignment.

Dashboard inspection searches bypass experiment gating so operators can still inspect learned context. Agent traffic should not send the dashboard caller type.

Methodology Contract

Assignment is user-level, but outcomes remain session-level. Reflexio joins each evaluated session to the first attributed request for its user and session, then compares treatment sessions with holdout sessions. This means one user's sessions stay in one arm even though the search request itself has no session ID.

The results page reports success, corrections, turns to resolution, and escalation for both arms. Treatment lift is the difference in session success rates. Its 95% confidence interval is clustered by user so repeat sessions from one person are not treated as independent random assignments. Evaluation coverage shows what share of evaluated sessions could be joined to experiment attribution; investigate untagged sessions before relying on the lift.

Keep these integration invariants:

• Always use the same stable application user_id for search and publish.
• Echo the search response's experiment ID and arm on every related publish.
• Publish both arms normally; only the search response differs.
• Keep the agent version, evaluation rubric, traffic mix, and other runtime behavior comparable across arms.
• Stop an experiment before starting another. Do not reuse an experiment ID.

Sampling and freshness

Regen jobs sample at most 200 sessions per (day × group) stratum by default so cost stays predictable as your traffic grows. The dashboard surfaces the sampled n per point so it's always honest about how many sessions back each number.

If you click into a session that wasn't in the sampled set, Reflexio grades it on demand via POST /api/evaluations/grade_on_demand and caches the result for 24 hours.

To tune the defaults, set eval_sample_n_per_stratum and eval_concurrency_limit in your Config.

API Reference

See Retrieval Experiments for the lifecycle and reporting methods, Search Models for response assignment metadata, and publish_interaction for publish attribution fields.

Per-turn comparison (F1)

The Evaluation page also renders a per-turn head-to-head win rate comparing your agent's response with Reflexio's retrieved context vs. without. F2 measures session-level outcomes; F1 zooms in to individual turns so you can spot-check the judge and surface specific cases where the Reflexio-less response was actually better.

To produce this signal, your agent code generates two responses per turn and uploads both on the same Interaction via the existing publish_interaction() API.

Integration

For each agent turn you want graded, generate the regular response WITH Reflexio rules in context, then re-run your LLM WITHOUT them to produce a shadow response. Publish both on the same agent interaction:

from reflexio import ReflexioClient, InteractionData

client = ReflexioClient()

# Retrieve Reflexio rules for this turn (your retrieval call)
rules = client.search(user_message)

# Generate the regular response WITH rules
regular_response = llm.generate(user_message, rules=rules)

# Generate the shadow response WITHOUT rules
shadow_response = llm.generate(user_message, rules=[])

# Publish both on the same agent Interaction
client.publish_interaction(
    user_id=user.id,
    interactions=[
        InteractionData(role="User", content=user_message),
        InteractionData(
            role="Agent",
            content=regular_response,        # what was served to the user
            shadow_content=shadow_response,  # used only for grading
        ),
    ],
    session_id=session_id,
    source="my-integration",
    agent_version="v1",
)

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/search" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "query": "<user_message>"
}
JSON

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/publish_interaction" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "user_id": "id",
  "interaction_data_list": [
    {
      "role": "User",
      "content": "<user_message>"
    },
    {
      "role": "Agent",
      "content": "<generate_result>",
      "shadow_content": "<generate_result>"
    }
  ],
  "session_id": "<session_id>",
  "source": "my-integration",
  "agent_version": "v1"
}
JSON

Dashboard diagnostics

• A Win rate trend plots daily win and loss rates and shows the underlying wins, losses, ties, and n for each UTC day.
• A View recent comparisons drawer with the most recent judged turns, each rendered side-by-side with the judge's rationale. Backed by GET /api/evaluations/shadow_comparisons/recent.
• A Top disagreements widget listing turns where the shadow response was significantly better than the Reflexio-augmented one (output.is_significantly_better=True losses) — actionable cases for updating your Reflexio rules.

Methodology contract

• The judge compares the two responses against request-local transcript context; no session history outside the current publish request is shown to the judge.
• Position (Request 1 vs Request 2) is randomized per call to mitigate LLM judge position bias. The mapping is recorded on ShadowComparisonVerdict.reflexio_is_request_1 so wins/losses can be derived deterministically downstream.
• The judge prompt version is pinned per org via Config.shadow_comparison_judge_prompt_version. Verdicts are stored with the version that produced them; the dashboard filters to your currently pinned version so a future rubric bump never silently mixes epochs into the headline number.
• Sampling: regen jobs are stratified per day at Config.eval_sample_n_per_stratum (default 200) so cost stays predictable. Clicking into a non-sampled session triggers on-demand grading via POST /api/evaluations/grade_on_demand with a 24h cache.

Where the data lands in the API

The win-rate trend is carried on GetEvaluationOverviewResponse.shadow_win_rate_trend (see the ShadowWinRateTrend schema). Individual verdicts are returned by the recent-verdicts endpoint above; each verdict is a ShadowComparisonVerdict wrapping a ShadowComparisonOutput.