ReflexioDeveloper Docs
Hosted Enterprise startGitHubWebsite
Menu
GitHubWebsitellms.txt
All

Agent Evaluation

Methods for retrieving, regenerating, and grading agent success evaluation results.

Agent Evaluation

get_agent_success_evaluation_results

Get session-level agent success evaluation results to analyze performance. For the end-to-end workflow covering configuration, automatic scheduling, evaluation_only=True, source-set comparison, shadow responses, and explicit regrading, see Evaluating Agent Performance.

response = client.get_agent_success_evaluation_results(
    agent_version=None,
    limit=100
)
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "limit": 100
}
JSON

Prop

Type

Example:

# Get all results
response = client.get_agent_success_evaluation_results(limit=100)

# Calculate success rate
total = len(response.agent_success_evaluation_results)
successful = sum(1 for r in response.agent_success_evaluation_results if r.is_success)
print(f"Success rate: {successful/total*100:.1f}%")

# Review failures
print("\nFailure Analysis:")
for result in response.agent_success_evaluation_results:
    if not result.is_success:
        print(f"Session: {result.session_id}")
        print(f"  Failure Type: {result.failure_type}")
        print(f"  Reason: {result.failure_reason}")

# Compare agent versions
v1_results = client.get_agent_success_evaluation_results(
    agent_version="v1.0.0",
    limit=100
)
v2_results = client.get_agent_success_evaluation_results(
    agent_version="v2.0.0",
    limit=100
)

v1_success = sum(1 for r in v1_results.agent_success_evaluation_results if r.is_success)
v2_success = sum(1 for r in v2_results.agent_success_evaluation_results if r.is_success)

print(f"\nVersion Comparison:")
print(f"v1.0.0: {v1_success}/{len(v1_results.agent_success_evaluation_results)} successful")
print(f"v2.0.0: {v2_success}/{len(v2_results.agent_success_evaluation_results)} successful")

# Group failures by type
from collections import Counter
failure_types = Counter(
    r.failure_type for r in response.agent_success_evaluation_results if not r.is_success
)
print("\nFailure Types:")
for failure_type, count in failure_types.most_common():
    print(f"  {failure_type}: {count}")
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "limit": 100
}
JSON

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "agent_version": "v1.0.0",
  "limit": 100
}
JSON

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "agent_version": "v2.0.0",
  "limit": 100
}
JSON

regenerate_evaluations

Start an asynchronous replay-the-judge job over a Unix timestamp window. Use this after changing the success rubric, evaluator model, or prompt and needing historical sessions re-scored.

import time

from reflexio import ReflexioClient

client = ReflexioClient()

job = client.regenerate_evaluations(
    from_ts=int(time.time()) - 7 * 24 * 60 * 60,
    to_ts=int(time.time()),
)

status = client.get_evaluation_regeneration_status(job.job_id)

# Optional: request cancellation while the job is running.
# client.cancel_evaluation_regeneration(job.job_id)
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/regenerate" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "from_ts": 1764547200,
  "to_ts": 1765152000
}
JSON

curl -X GET "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/regenerate/<job_id>" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY"

curl -X DELETE "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/regenerate/<job_id>" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY"

get_evaluation_regeneration_status

Return the current status and progress for a regeneration job ID.

cancel_evaluation_regeneration

Request cancellation for a running regeneration job. Work already in progress may finish before the job stops.

Prop

Type

grade_on_demand

Grade one session synchronously without waiting for the normal session inactivity delay. Results are cached for 24 hours per session and agent version.

from reflexio import ReflexioClient

client = ReflexioClient()

result = client.grade_on_demand(
    session_id="session_001",
    agent_version="v2.1.0",
)

if result.skipped_reason:
    print(f"Skipped: {result.skipped_reason}")
else:
    print(f"Result id: {result.result_id}, cached: {result.cached}")
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/grade_on_demand" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "session_id": "session_001",
  "agent_version": "v2.1.0"
}
JSON

Prop

Type

get_retrieved_learning_evaluation_results

Read the latest per-learning relevance and impact verdicts produced for a graded session.

verdicts = client.get_retrieved_learning_evaluation_results(
    user_id="user_001",
    session_id="session_001",
    start_time=window_start,
    end_time=window_end,
    limit=100,
)

for verdict in verdicts.results:
    print(
        verdict.interaction_id,
        verdict.kind,
        verdict.learning_id,
        verdict.is_relevant,
        verdict.impact,
    )

Prop

Type

Publish the stable identities of every profile or playbook injected into agent turns through retrieved_learnings. This enables the detailed per-learning verdicts returned here and provides attribution data for future retrieved-learning optimization. Call grade_on_demand before this read when an immediate demo result is needed.

Each result remains a judge verdict for one learning, but it is attributed to the specific response interaction that used that learning. The same learning used on two responses therefore produces two independently targeted verdicts. The Evaluation page groups these rows by interaction so each response counts once in the displayed metrics:

  • relevance is positive when at least one graded learning is relevant;
  • impact is positive when at least one learning is positive and none is negative, negative when at least one is negative and none is positive, mixed when both occur, and neutral when every graded impact is neutral;
  • null verdicts are ignored when another learning on the response was graded, while an all-null response is reported as ungraded.