Client Methods

Retrieval Experiments

Start, stop, inspect, and report user-level retrieval holdout experiments.

Retrieval experiments deterministically assign users to treatment or holdout. Treatment searches retrieve learned context normally; holdout searches succeed with empty learning results. Both arms continue to publish, receive session evaluation, and teach Reflexio.

Only one experiment can be active for an organization. Experiment IDs are unique and stopped experiments remain in history.

`list_retrieval_experiments`

Python

experiments = client.list_retrieval_experiments()
print(experiments.active_experiment)

Calls GET /api/retrieval_experiments and returns the active record, when one exists, plus all historical records in reverse start order.

`start_retrieval_experiment`

Python

experiments = client.start_retrieval_experiment(
    experiment_id="support-agent-2026-08",
    holdout_percentage=10,
)

Calls POST /api/retrieval_experiments. holdout_percentage must be greater than 0 and less than 100. Starting while another experiment is active or reusing an ID returns 409.

`stop_retrieval_experiment`

Python

experiments = client.stop_retrieval_experiment("support-agent-2026-08")

Calls POST /api/retrieval_experiments/stop. Stopping ends new assignments but does not remove stored publish attribution or historical results.

`get_retrieval_experiment_results`

Python

results = client.get_retrieval_experiment_results("support-agent-2026-08")
print(results.treatment.success_rate)
print(results.holdout.success_rate)
print(results.treatment.average_output_tokens)
print(results.holdout.average_output_tokens)
print(results.success_rate_lift_percentage_points)
print(results.confidence_interval_95_percentage_points)

Calls GET /api/retrieval_experiments/{experiment_id}/results. Results group session-level success evaluations by the user-level arm stored on published requests. It also compares stored non-user content-token totals per published session. The confidence interval is clustered by user. Evaluation coverage and the unattributed session count show whether evaluated sessions were published with experiment metadata; output_token_session_count shows how many published sessions have complete persisted token counts.

Response schemas

`RetrievalExperimentListResponse`

Prop	Type	Default	Description
`active_experiment?`	RetrievalExperimentRecord \| None	`—`	The currently active experiment. Omitted when no experiment is running.
`experiments`	list[RetrievalExperimentRecord]	`—`	All experiments in reverse start order, including the active experiment when present.

`RetrievalExperimentRecord`

Prop	Type	Default	Description
`experiment_id`	`str`	`—`	Unique experiment identifier.
`holdout_percentage`	`float`	`—`	Percentage of users assigned to holdout.
`started_at`	`int`	`—`	Experiment start time as a Unix timestamp.
`ended_at?`	`int \| None`	`—`	Experiment stop time as a Unix timestamp. Omitted while the experiment is active.

`RetrievalExperimentResultsResponse`

Prop	Type	Default	Description
`experiment`	RetrievalExperimentRecord	`—`	Lifecycle metadata for the reported experiment.
`treatment`	RetrievalExperimentArmMetrics	`—`	Published and evaluated treatment-arm metrics.
`holdout`	RetrievalExperimentArmMetrics	`—`	Published and evaluated holdout-arm metrics.
`success_rate_lift_percentage_points?`	`float \| None`	`—`	Treatment success rate minus holdout success rate, in percentage points.
`relative_success_lift?`	`float \| None`	`—`	Success-rate lift relative to the holdout success rate.
`confidence_interval_95_percentage_points?`	`tuple[float, float] \| None`	`—`	User-clustered 95% confidence interval for success-rate lift, in percentage points.
`evaluated_session_coverage?`	`float \| None`	`—`	Share of evaluated sessions carrying attribution for this experiment.
`unattributed_evaluated_session_count`	`int`	`—`	Evaluated sessions in the experiment window without matching attribution.

`RetrievalExperimentArmMetrics`

Prop	Type	Default	Description
`arm`	`"treatment" \| "holdout"`	`—`	Experiment arm represented by these metrics.
`assigned_user_count`	`int`	`—`	Distinct users with at least one attributed published session.
`published_session_count`	`int`	`—`	Distinct published sessions attributed to the arm.
`evaluated_user_count`	`int`	`—`	Distinct attributed users with session evaluations.
`evaluated_session_count`	`int`	`—`	Attributed sessions with evaluations.
`success_rate?`	`float \| None`	`—`	Mean session-success value for evaluated sessions.
`average_corrections?`	`float \| None`	`—`	Mean judge-derived corrective user-turn count across evaluated sessions, including sessions with zero corrections.
`average_turns_to_resolution?`	`float \| None`	`—`	Mean turns to resolution when that value is available.
`escalation_rate?`	`float \| None`	`—`	Share of evaluated sessions marked as escalated.
`average_output_tokens?`	`float \| None`	`—`	Mean stored token count for non-user interaction content across fully measured published sessions.
`output_token_session_count`	`int`	`—`	Published sessions included in average_output_tokens because every non-user interaction has a stored token count.

Use the complete integration pattern in Measuring Reflexio's Impact.