ReflexioDeveloper Docs
Menu
GitHubWebsitellms.txt
Client Methods

Agent Evaluation

Methods for retrieving, regenerating, and grading agent success evaluation results.

get_agent_success_evaluation_results

Get session-level agent success evaluation results to analyze performance. For the end-to-end workflow covering configuration, automatic scheduling, evaluation_only=True, source-set comparison, shadow responses, and explicit regrading, see Evaluating Agent Performance.

Python
response = client.get_agent_success_evaluation_results(
    agent_version=None,
    limit=100
)
Terminal
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "limit": 100
}
JSON
PropTypeDefaultDescription
agent_version?str—Filter by specific agent version
limit?int100Maximum number of results to return
start_time?datetime—Filter by result created-at start time
end_time?datetime—Filter by result created-at end time

Example:

Python
# Get all results
response = client.get_agent_success_evaluation_results(limit=100)

# Calculate success rate
total = len(response.agent_success_evaluation_results)
successful = sum(1 for r in response.agent_success_evaluation_results if r.is_success)
print(f"Success rate: {successful/total*100:.1f}%")

# Review failures
print("\nFailure Analysis:")
for result in response.agent_success_evaluation_results:
    if not result.is_success:
        print(f"Session: {result.session_id}")
        print(f"  Failure Type: {result.failure_type}")
        print(f"  Reason: {result.failure_reason}")

# Compare agent versions
v1_results = client.get_agent_success_evaluation_results(
    agent_version="v1.0.0",
    limit=100
)
v2_results = client.get_agent_success_evaluation_results(
    agent_version="v2.0.0",
    limit=100
)

v1_success = sum(1 for r in v1_results.agent_success_evaluation_results if r.is_success)
v2_success = sum(1 for r in v2_results.agent_success_evaluation_results if r.is_success)

print(f"\nVersion Comparison:")
print(f"v1.0.0: {v1_success}/{len(v1_results.agent_success_evaluation_results)} successful")
print(f"v2.0.0: {v2_success}/{len(v2_results.agent_success_evaluation_results)} successful")

# Group failures by type
from collections import Counter
failure_types = Counter(
    r.failure_type for r in response.agent_success_evaluation_results if not r.is_success
)
print("\nFailure Types:")
for failure_type, count in failure_types.most_common():
    print(f"  {failure_type}: {count}")
Terminal
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "limit": 100
}
JSON

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "agent_version": "v1.0.0",
  "limit": 100
}
JSON

curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/get_agent_success_evaluation_results" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "agent_version": "v2.0.0",
  "limit": 100
}
JSON

regenerate_evaluations

Start an asynchronous replay-the-judge job over a Unix timestamp window. Use this after changing the success rubric, evaluator model, or prompt and needing historical sessions re-scored.

Python
import time

from reflexio import ReflexioClient

client = ReflexioClient()

job = client.regenerate_evaluations(
    from_ts=int(time.time()) - 7 * 24 * 60 * 60,
    to_ts=int(time.time()),
)

status = client.get_evaluation_regeneration_status(job.job_id)

# Optional: request cancellation while the job is running.
# client.cancel_evaluation_regeneration(job.job_id)
Terminal
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/regenerate" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "from_ts": 1764547200,
  "to_ts": 1765152000
}
JSON

curl -X GET "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/regenerate/<job_id>" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY"

curl -X DELETE "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/regenerate/<job_id>" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY"

get_evaluation_regeneration_status

Return the current status and progress for a regeneration job ID.

cancel_evaluation_regeneration

Request cancellation for a running regeneration job. Work already in progress may finish before the job stops.

PropTypeDefaultDescription
from_tsint—Inclusive lower bound of the window as Unix epoch seconds.
to_tsint—Inclusive upper bound of the window as Unix epoch seconds. Must be greater than from_ts.
evaluation_name?str—Deprecated compatibility field. Accepted but ignored by the singleton evaluator.

grade_on_demand

Grade one session synchronously without waiting for the normal session inactivity delay. Results are cached for 24 hours per session and agent version.

Python
from reflexio import ReflexioClient

client = ReflexioClient()

result = client.grade_on_demand(
    session_id="session_001",
    agent_version="v2.1.0",
)

if result.skipped_reason:
    print(f"Skipped: {result.skipped_reason}")
else:
    print(f"Result id: {result.result_id}, cached: {result.cached}")
Terminal
curl -X POST "${REFLEXIO_URL:-https://www.reflexio.ai}/api/evaluations/grade_on_demand" \
  -H "User-Agent: my-agent-reflexio" \
  -H "Authorization: Bearer $REFLEXIO_API_KEY" \
  -H "Content-Type: application/json" \
  --data @- <<'JSON'
{
  "session_id": "session_001",
  "agent_version": "v2.1.0"
}
JSON
PropTypeDefaultDescription
session_idstr—Target session to grade.
agent_versionstr—Agent version to grade.
evaluation_name?str—Deprecated compatibility field. Accepted but ignored by the singleton evaluator.

get_retrieved_learning_evaluation_results

Read the latest per-learning relevance and impact verdicts produced for a graded session.

Python
verdicts = client.get_retrieved_learning_evaluation_results(
    user_id="user_001",
    session_id="session_001",
    start_time=window_start,
    end_time=window_end,
    limit=100,
)

for verdict in verdicts.results:
    print(
        verdict.interaction_id,
        verdict.kind,
        verdict.learning_id,
        verdict.is_relevant,
        verdict.impact,
    )
PropTypeDefaultDescription
user_id?str—Filter by session owner.
session_id?str—Filter by session.
start_time?datetime—Inclusive lower bound on the target interaction timestamp.
end_time?datetime—Inclusive upper bound on the target interaction timestamp.
limit?int100Maximum learning-verdict rows to return (1-1000).

Publish the stable identities of every profile or playbook injected into agent turns through retrieved_learnings. This enables the detailed per-learning verdicts returned here and provides attribution data for future retrieved-learning optimization. Call grade_on_demand before this read when an immediate demo result is needed, and inspect its retrieved_learning_status, skipped_reason, and cached fields. This read does not trigger grading. See the complete grade-and-read example for reasons, missing-result diagnostics, and sampling guidance.

Each result remains a judge verdict for one learning, but it is attributed to the specific response interaction that used that learning. The same learning used on two responses therefore produces two independently targeted verdicts. The Evaluation page groups these rows by interaction so each response counts once in the displayed metrics:

  • relevance is positive when at least one graded learning is relevant;
  • impact is positive when at least one learning is positive and none is negative, negative when at least one is negative and none is positive, mixed when both occur, and neutral when every graded impact is neutral;
  • null verdicts are ignored when another learning on the response was graded, while an all-null response is reported as ungraded.

Client Methods

Initialize and authenticate the client.

Interaction Models

Inspect the source interaction schema.

Configuration Guide

Configure extraction and evaluation.

Searching Learned Context

Retrieve relevant context for your agent.