Schemas

Evaluation Models

Data structures for agent success evaluation results.

AgentSuccessEvaluationResult

Represents an agent performance evaluation result.

Prop	Type	Default	Description
`result_id?`	`integer`	`0`	Unique identifier for the evaluation result
`user_id?`	`string`	`—`	User whose session slice was evaluated
`agent_version`	`string`	`—`	Version of the agent being evaluated
`session_id`	`string`	`—`	Session ID that was evaluated
`is_success`	`boolean`	`—`	Whether the agent was successful
`failure_type?`	`string`	`—`	Type of failure if unsuccessful
`failure_reason?`	`string`	`—`	Detailed reason for failure
`evaluation_name?`	`string`	`—`	Name of the evaluation configuration that generated this result
`created_at?`	`integer`	`Current time`	Unix timestamp when evaluation was created
`regular_vs_shadow?`	RegularVsShadow	`—`	Comparison result between regular and shadow agent versions
`number_of_correction_per_session?`	`integer`	`0`	Number of user turns in the session that corrected or redirected an earlier agent response
`user_turns_to_resolution?`	`integer`	`—`	Number of user-side turns to reach resolution (only when is_success=True, programmatically counted)
`is_escalated?`	`boolean`	`false`	Whether the user was handed off to a human or another agent (LLM-computed)
`trajectory_visibility_snapshot?`	`string \| null`	`null`	Internal PostgreSQL visibility token from the actual judgment input read. Legacy and non-PostgreSQL verdicts have null; it is not a user-supplied time or policy identifier.
`trajectory_gc_epoch?`	`integer \| null`	`null`	Internal project GC epoch captured with actual judgment inputs. Prevents late writes after departure evidence is reclaimed; null for legacy and non-PostgreSQL verdicts.
`tags?`	`array[string] \| null`	`—`	Tags assigned asynchronously from the persisted evaluation summary. Null means not yet processed; an empty array means no tags matched.

GetAgentSuccessEvaluationResultsRequest

Request model for getting agent success evaluation results.

Prop	Type	Default	Description
`agent_version?`	`string`	`—`	Filter by specific agent version
`limit?`	`integer`	`100`	Maximum number of results to return
`start_time?`	`datetime`	`—`	Filter by result created-at start time
`end_time?`	`datetime`	`—`	Filter by result created-at end time

GetAgentSuccessEvaluationResultsResponse

Response model for getting agent success evaluation results.

Prop	Type	Default	Description
`success`	`boolean`	`—`	Whether the request was successful
`agent_success_evaluation_results`	array[AgentSuccessEvaluationResult]	`—`	List of evaluation results
`msg?`	`string`	`—`	Additional message

RetrievedLearningEvaluationResult

The latest per-learning relevance/impact verdict for one target interaction. One row per (user_id, session_id, interaction_id, kind, learning_id); the stored set is the most recent successfully persisted evaluation for the session, not an append-only history. The same learning used on multiple interactions is evaluated separately against each response. Produced automatically when a session that published interactions with retrieved_learnings goes through group evaluation.

Prop	Type	Default	Description
`result_id?`	`integer`	`—`	Result row identifier
`user_id`	`string`	`—`	Session owner
`session_id`	`string`	`—`	Evaluated session
`agent_version?`	`string`	`""`	Agent version supplied to group evaluation
`interaction_id?`	`integer \| null`	`—`	Response interaction that used this learning. Null only on backward-compatible legacy rows that predate interaction attribution.
`interaction_created_at?`	`integer \| null`	`—`	Unix timestamp of the target interaction. Null only on backward-compatible legacy rows.
`kind`	`"profile" \| "user_playbook" \| "agent_playbook"`	`—`	The learning kind
`learning_id`	`string`	`—`	Stable storage id, matching RetrievedLearning.learning_id
`is_relevant?`	`boolean \| null`	`—`	Whether the learning applies to its target interaction and response; null only when the relevance judge failed
`relevance_reason?`	`string`	`""`	Judge reasoning; empty when is_relevant is null
`impact?`	`"positive" \| "negative" \| "neutral" \| null`	`—`	Relative to the agent's definition of success, whether the learning moved the response toward success, away from it, or did not materially change it; null only when the impact judge failed
`impact_reason?`	`string`	`""`	Judge reasoning; empty when impact is null
`diagnosis?`	`PlaybookDiagnosis \| null`	`null`	Transcript-grounded diagnosis for playbooks; null for profiles or older results. Separate from relevance and impact.
`evaluated_playbook_digest?`	`string \| null`	`null`	Opaque content-and-trigger digest of the user playbook actually evaluated. The tuner compares it with the serve-time digest; not a revision identifier.
`diagnosis_evidence_complete?`	`boolean`	`false`	Whether the judge saw the complete bounded transcript and playbook content/trigger. Does not establish historical identity or authorize a rewrite.
`created_at?`	`integer`	`—`	Earliest request timestamp in the evaluated session

PlaybookDiagnosis

Field	Type	Required/default
`category`	`content_defect`, `application_failure`, `external_failure`, `no_issue`, or `unknown`	Required
`reason`	`string`, 1–4,000 characters	Required
`evidence_interaction_ids`	`list[int]`, at most 20	`[]`

Diagnosis explains the observed problem; it is not replay evidence or proof of improvement. Missing or unknown diagnosis does not disqualify otherwise eligible historical evidence. Out-of-session citations are discarded; only verified, supported diagnoses can influence tuning. See diagnosis and tuning.

GetRetrievedLearningEvaluationResultsRequest

Request model for POST /api/get_retrieved_learning_evaluation_results.

Prop	Type	Default	Description
`user_id?`	`string`	`—`	Filter by session owner
`session_id?`	`string`	`—`	Filter by session
`start_time?`	`datetime`	`—`	Inclusive lower bound on interaction_created_at
`end_time?`	`datetime`	`—`	Inclusive upper bound on interaction_created_at
`limit?`	`integer`	`100`	Maximum rows to return (1-1000), ordered newest first

GetRetrievedLearningEvaluationResultsResponse

Response model for POST /api/get_retrieved_learning_evaluation_results.

Prop	Type	Default	Description
`success`	`boolean`	`—`	Whether the request was successful
`results`	array[RetrievedLearningEvaluationResult]	`—`	Matching per-learning verdicts. Time-filtered reads are grouped newest target interaction first; unfiltered reads retain created_at DESC, result_id DESC ordering.
`msg?`	`string`	`—`	Additional message

EvaluationSourceSetRequest

One labeled request-source cohort for evaluation comparison. Sources match Request.source exactly, including the empty string for requests published without a source.

Prop	Type	Default	Description
`label`	`string`	`—`	Human-readable label for this source set
`sources`	`array[string]`	`—`	One or more exact Request.source values in this set

HeroBucket

One time bucket in the success-rate trend. Rates are fractions from 0 to 1; ts is a Unix timestamp in seconds.

Prop	Type	Default	Description
`ts`	`integer`	`—`	Bucket timestamp.
`regular_rate`	`float`	`—`	Success rate with retrieved learning.
`shadow_rate`	`float \| null`	`—`	Success rate without retrieved learning, when available.
`regular_n`	`integer`	`—`	Number of regular evaluation results in the bucket.
`shadow_n`	`integer`	`—`	Number of shadow evaluation results in the bucket.
`avg_corrections?`	`float`	`0.0`	Mean corrections per evaluation result, including zero-count sessions.
`escalation_rate?`	`float`	`0.0`	Fraction of sessions marked as escalated.

ContextTile

Session-level metrics with changes against the previous seven-day window. Percentage metrics use percentage points; count metrics use absolute values.

Prop	Type	Default	Description
`success`	`{ current: float, delta_pp: float }`	`—`	Task success percentage and its change.
`behavior_success`	`{ current: float \| null, delta_pp: float \| null, eligible_sessions: integer, excluded_system_errors: integer }`	`—`	Success excluding system-error results. The percentage is null when no eligible sessions exist; its change is null if either comparison window has none.
`corrections`	`{ current: float, delta: float }`	`—`	Average corrections per session and its change.
`turns`	`{ current: float, delta: float }`	`—`	Average user turns to resolution among sessions with that measurement, and its change.
`escalation`	`{ current: float, delta_pp: float }`	`—`	Escalation percentage and its change.

ScoreDistribution

Corrections-per-session histogram for the current window and its baseline. The arrays align by index: each label identifies the corresponding count in current_bins and baseline_bins.

Prop	Type	Default	Description
`current_bins`	`array[integer]`	`—`	Session counts per bin in the current window.
`baseline_bins`	`array[integer]`	`—`	Session counts per bin in the baseline window.
`labels`	`array[string]`	`—`	Correction-count labels for the bins.

RuleAttributionRow

Session outcomes attributed to one cited profile or playbook.

Prop	Type	Default	Description
`rule_id`	`string`	`—`	ID of the cited learning.
`kind`	`"playbook" \| "profile" \| "user_playbook" \| "agent_playbook"`	`—`	Learning kind.
`title?`	`string`	`—`	Display title. Defaults to an empty string.
`successes_with`	`integer`	`—`	Successful sessions citing the learning; at least zero.
`failures_with`	`integer`	`—`	Failed sessions citing the learning; at least zero.
`net_sessions`	`integer`	`—`	Successful sessions minus failed sessions.
`cited_session_ids?`	`array[string]`	`—`	Session IDs in the trend window that cited this learning. Defaults to [].

SourceSetEvaluationMetrics

Metrics for one source set in SourceSetComparison.

Prop	Type	Default	Description
`label`	`string`	`—`	Source-set label
`sources`	`array[string]`	`—`	Exact Request.source values included in the set
`session_count`	`integer`	`—`	Evaluated sessions in the selected window assigned to this set
`session_ids?`	`array[string]`	`[]`	Session IDs assigned to this set, used by the Evaluation page detail filter
`success_rate_pp`	`float`	`—`	Success rate for this set, in percentage points
`buckets`	array[HeroBucket]	`—`	Success-rate trend buckets for this set
`context_tiles`	ContextTile	`—`	Success, corrections, turns, and escalation metrics for this set
`score_distribution`	ScoreDistribution	`—`	Corrections-per-session distribution for this set
`rule_attribution`	array[RuleAttributionRow]	`—`	Rule attribution computed only from sessions in this set
`braintrust_tiles?`	`array[BraintrustTileRow]`	`[]`	Imported scorer aggregates matched to this set's sessions

SourceSetComparison

Request-source comparison payload. Each evaluation result is assigned by joining to the session's first request and reading that request's source.

Prop	Type	Default	Description
`available_sources?`	`array[string]`	`[]`	Distinct first-request source values in the selected evaluation window
`source_sessions?`	`array[{ source: string, session_ids: array[string], sessions: array[{ user_id: string, session_id: string }] }]`	`[]`	Evaluated sessions grouped by first-request source. Use sessions for collision-safe filtering across users; session_ids is retained for compatibility. Both lists are de-duplicated and sorted within each source.
`sets?`	array[SourceSetEvaluationMetrics]	`[]`	Per-source-set metrics for the requested source_sets
`unmatched_session_count?`	`integer`	`0`	Evaluated sessions whose first-request source was not included in any requested source set

GetEvaluationOverviewRequest

Input for the Evaluation Overview endpoint (POST /api/get_evaluation_overview).

Prop	Type	Default	Description
`from_ts`	`integer`	`—`	Window start, unix epoch seconds
`to_ts`	`integer`	`—`	Window end, unix epoch seconds
`bucket?`	`string`	`"week"`	Granularity of the hero trend buckets. One of 'day' or 'week'.
`include_shadow?`	`boolean`	`true`	When false, skip the shadow-side aggregations (cheaper) — the hero degrades to shadow_off state.
`source_sets?`	array[EvaluationSourceSetRequest]	`[]`	Optional labeled source sets for comparison. Labels must be unique; source values cannot overlap across sets.

GetEvaluationOverviewResponse

Response model for the Evaluation Overview endpoint. The global fields populate the all-source hero, context, and rule-attribution sections. Source-set metrics are returned separately in source_set_comparison; the web portal uses those source-set metrics to scope the page when a user selects a single source.

Prop	Type	Default	Description
`hero`	`HeroBlock`	`—`	Headline trend block — success-rate trend, delta, and shadow comparison state
`context_tiles`	`ContextTile`	`—`	The four mini-tiles in the context band (success, corrections, turns, escalation)
`rule_attribution`	`array[RuleAttributionRow]`	`—`	Rows in the 'rules that moved the needle' panel
`score_distribution`	`ScoreDistribution`	`—`	Corrections-per-session histogram, current window + baseline
`braintrust_tiles?`	`array[BraintrustTileRow]`	`[]`	Imported-scorer aggregates (Braintrust integration)
`shadow_win_rate_trend?`	ShadowWinRateTrend	`—`	F1 per-turn shadow win-rate trend, daily buckets plus a window total. Filtered to the org's pinned shadow_comparison_judge_prompt_version so prior rubric epochs never mix in. Defaults to an empty payload when no verdicts exist in the window.
`recent_results?`	array[AgentSuccessEvaluationResult]	`[]`	The newest 100 embedding-free evaluation summaries in the requested window, ordered newest first. The fields match AgentSuccessEvaluationResult except that the internal embedding is never exposed.
`source_set_comparison?`	SourceSetComparison	`—`	Request-source set comparison payload. Populated from Request.source on the first request in each evaluated session.

ShadowComparisonOutput

LLM judge verdict for a single per-turn Reflexio-vs-shadow comparison (F1). The position of the two responses shown to the judge is randomized per call to mitigate position bias; the mapping is recorded on ShadowComparisonVerdict.reflexio_is_request_1.

Prop	Type	Default	Description
`better_request`	`string`	`—`	Which side the judge picked. One of "1", "2", or "tie". "1" and "2" are blind labels — the dashboard derives "Reflexio won/lost/tied" by combining this with reflexio_is_request_1.
`is_significantly_better`	`boolean`	`—`	True if the better side is clearly better; False if marginal/close-but-edges-it. Used to filter the "Top 10 disagreements" widget down to actionable cases.
`comparison_reason?`	`string`	`—`	1-2 sentence rationale. Displayed in the drill-down drawer.

ShadowComparisonVerdict

One per-turn comparison verdict, stored per (interaction_id, judge_prompt_version) in the shadow_comparison_verdicts table.

Prop	Type	Default	Description
`verdict_id`	`integer`	`—`	Storage-assigned autoincrement primary key.
`interaction_id`	`string`	`—`	The interaction this verdict grades. Joins to Interaction.interaction_id for drill-down display.
`session_id`	`string`	`—`	The session containing the graded interaction.
`agent_version`	`string`	`—`	Pinned for trend-by-version slicing.
`reflexio_is_request_1`	`boolean`	`—`	Position-randomization record. True when the Reflexio response was shown as Request 1 to the judge. Combine with output.better_request to derive win/loss/tie.
`output`	ShadowComparisonOutput	`—`	The judge's structured verdict.
`judge_prompt_version`	`string`	`—`	Semver of the shadow_comparison prompt used. The dashboard filters to the org's currently pinned Config.shadow_comparison_judge_prompt_version so verdicts from a prior rubric never mix into the headline.
`created_at`	`datetime`	`—`	When the judge call returned. Storage layers assume UTC.

ShadowWinRateTrendPoint

One daily bucket of per-turn shadow comparison verdicts (F1). Buckets are UTC-aligned and surfaced in ascending date order.

Prop	Type	Default	Description
`date`	`string`	`—`	ISO date for the bucket start (YYYY-MM-DD), UTC.
`n`	`integer`	`—`	Total verdicts in this bucket. Non-negative.
`wins`	`integer`	`—`	Reflexio wins in this bucket. Non-negative.
`losses`	`integer`	`—`	Reflexio losses in this bucket. Non-negative.
`ties`	`integer`	`—`	Ties in this bucket. Non-negative.

ShadowWinRateTrendWindowTotal

Aggregate of all shadow verdicts across the trend window (F1). Used to render the headline win-rate tile on the Evaluation page.

Prop	Type	Default	Description
`n`	`integer`	`—`	Total verdicts in the window.
`wins`	`integer`	`—`	Reflexio wins.
`losses`	`integer`	`—`	Reflexio losses.
`ties`	`integer`	`—`	Ties.
`win_rate`	`float`	`—`	wins / n in [0.0, 1.0]; 0.0 when n == 0.
`net_win`	`float`	`—`	(wins - losses) / n in [-1.0, 1.0]; 0.0 when n == 0.

ShadowWinRateTrend

F1 shadow win-rate trend payload. Carried on GetEvaluationOverviewResponse.shadow_win_rate_trend. Verdicts produced under a previous rubric epoch are filtered out at storage time by judge_prompt_version, so the dashboard never silently mixes incompatible rubrics into the headline number.

Prop	Type	Default	Description
`daily?`	array[ShadowWinRateTrendPoint]	`[]`	Daily UTC buckets, sorted ascending. Empty when no verdicts exist in the window.
`window_total?`	ShadowWinRateTrendWindowTotal	`—`	Aggregate over all daily buckets.
`judge_prompt_version?`	`string`	`"v1.0.0"`	Pinned prompt version the verdicts in this payload were graded under. Echoed so the dashboard can show which rubric epoch produced the numbers.

GetRecentShadowComparisonsResponse

Returned by GET /api/evaluations/shadow_comparisons/recent.

Prop	Type	Default	Description
`verdicts?`	array[ShadowComparisonVerdict]	`[]`	Recent verdicts for the org's currently pinned shadow_comparison prompt version, newest first. Capped at the limit query param (default 10, max 100). Empty when the backend does not support the shadow_comparison_verdicts storage feature, when no verdicts exist in the 30-day lookback window, or when no verdicts match the pinned prompt version.

GET /api/evaluations/shadow_comparisons/recent

Returns the N most recent per-turn shadow comparison verdicts (F1). Powers two surfaces on the Evaluation page:

The drawer triggered from the per-turn comparison tile — shows the N most recent verdicts so you can spot-check the judge.
The "Top 10 disagreements" widget — fetches a wider pool and the frontend filters to is_significantly_better=True losses to surface actionable rule-correction candidates.

Verdicts are restricted to the org's currently pinned Config.shadow_comparison_judge_prompt_version so verdicts from an older rubric never mix into the drawer. A 30-day lookback is enforced server-side so the storage layer can use an index range scan instead of a full table read.

Query parameters

Prop	Type	Default	Description
`limit?`	`integer`	`10`	Max verdicts to return. Clamped to [1, 100].

Response

GetRecentShadowComparisonsResponse.

Errors

503 Storage not configured — the server has no storage backend wired.
Returns verdicts: [] (200) when the storage backend does not implement the shadow_comparison_verdicts feature (e.g. the disk backend), or when no verdicts in the 30-day window match the pinned prompt version.

RegenerateRequest

Input for POST /api/evaluations/regenerate — kicks off a replay-the-judge job over a closed window.

Prop	Type	Default	Description
`evaluation_name?`	`string`	`—`	Deprecated compatibility field. Accepted but ignored; regenerate runs the singleton agent success evaluation for the org.
`from_ts`	`integer`	`—`	Inclusive lower bound of the window (Unix epoch seconds).
`to_ts`	`integer`	`—`	Inclusive upper bound of the window (Unix epoch seconds). Must be strictly greater than from_ts.

RegenerateStartResponse

Returned by POST /api/evaluations/regenerate.

Prop	Type	Default	Description
`job_id`	`string`	`—`	Opaque handle used to poll status or cancel.
`total`	`integer`	`—`	Number of distinct (user, session, agent_version, source) tuples the worker will replay.

RegenerateFailure

One failed session in a regen job's failure list.

Prop	Type	Default	Description
`session_id`	`string`	`—`	The session whose replay failed.
`reason`	`string`	`—`	Truncated exception message (worker boundary).

RegenerateStatusResponse

Returned by GET /api/evaluations/regenerate/{job_id}. Carries the live lifecycle state of the regen worker plus the F3 sampling and concurrency counters described in Measuring Reflexio's Impact.

Prop	Type	Default	Description
`job_id`	`string`	`—`	Opaque job handle.
`status`	`string`	`—`	One of "running", "completed", "cancelled", "error". "completed" means the worker loop finished iterating regardless of whether every session succeeded — per-session pass/fail is in completed vs. failed. "error" means the worker itself crashed before or during iteration.
`total`	`integer`	`—`	Total tuples queued at job creation.
`completed`	`integer`	`—`	Number of successfully replayed tuples.
`failed`	`integer`	`—`	Number of tuples that raised in the worker.
`failures`	array[RegenerateFailure]	`—`	Per-session failure rows, capped inside the worker so the response stays small.
`started_at`	`float`	`—`	Unix-seconds wall-clock timestamp at job creation.
`finished_at?`	`float`	`—`	Unix-seconds wall-clock timestamp at worker exit; null while status == "running".
`total_candidates?`	`integer`	`0`	F3: count of distinct (session, agent_version) candidate tuples discovered in the regen window BEFORE stratified sampling. Defaults to 0 for jobs created before F3 shipped.
`sampled_count?`	`integer`	`0`	F3: count of candidates retained after stratified sampling. Equal to total_candidates when no stratum exceeded Config.eval_sample_n_per_stratum.
`concurrency_limit?`	`integer`	`0`	F3: max simultaneous worker threads. Mirrors Config.eval_concurrency_limit at job start; reported so the dashboard can show 'n_sampled / concurrency_limit' status legibly.

GradeOnDemandRequest

Input for POST /api/evaluations/grade_on_demand — single-session click-through grading triggered when a dashboard user opens a session that wasn't in the sampled regen set. See Sampling and freshness.

Prop	Type	Default	Description
`session_id`	`string`	`—`	Target session to grade. Must be non-empty.
`agent_version`	`string`	`—`	Agent version filter. Must be non-empty — eval results are versioned.
`evaluation_name?`	`string`	`—`	Deprecated compatibility field. Accepted but ignored; on-demand grading runs the singleton agent success evaluation.

GradeOnDemandResponse

Returned by POST /api/evaluations/grade_on_demand. Results are cached for 24 hours via the operation-state mechanism — the second call for the same (session, agent_version) within the cache window will return cached: true with the original result_id.

Prop	Type	Default	Description
`session_id`	`string`	`—`	Echo of the requested session.
`result_id?`	`integer`	`—`	The eval result row id, or null if grading was skipped (e.g., session not found, no interactions).
`cached?`	`boolean`	`false`	True when the response came from the 24h cache window. False on a fresh grade.
`skipped_reason?`	`string`	`—`	If grading was skipped, the reason (e.g., "NO_REQUESTS"). null on success.
`retrieved_learning_status?`	`string`	`—`	Outcome of the retrieved-learning evaluation for this session (e.g., "complete", "degraded", "failed", "not_applicable"). null on legacy cache entries and skipped grades.

Client Methods

Interaction Models

Configuration Guide

Searching Learned Context