Retrieval Experiments

Retrieval experiments deterministically assign users to treatment or holdout. Treatment searches retrieve learned context normally; holdout searches succeed with empty learning results. Both arms continue to publish, receive session evaluation, and teach Reflexio.

Only one experiment can be active for an organization. Experiment IDs are unique and stopped experiments remain in history.

list_retrieval_experiments

experiments = client.list_retrieval_experiments()
print(experiments.active_experiment)

Calls GET /api/retrieval_experiments and returns the active record, when one exists, plus all historical records in reverse start order.

start_retrieval_experiment

experiments = client.start_retrieval_experiment(
    experiment_id="support-agent-2026-08",
    holdout_percentage=10,
)

Calls POST /api/retrieval_experiments. holdout_percentage must be greater than 0 and less than 100. Starting while another experiment is active or reusing an ID returns 409.

stop_retrieval_experiment

experiments = client.stop_retrieval_experiment("support-agent-2026-08")

Calls POST /api/retrieval_experiments/stop. Stopping ends new assignments but does not remove stored publish attribution or historical results.

get_retrieval_experiment_results

results = client.get_retrieval_experiment_results("support-agent-2026-08")
print(results.treatment.success_rate)
print(results.holdout.success_rate)
print(results.treatment.average_output_tokens)
print(results.holdout.average_output_tokens)
print(results.success_rate_lift_percentage_points)
print(results.confidence_interval_95_percentage_points)

Calls GET /api/retrieval_experiments/{experiment_id}/results. Results group session-level success evaluations by the user-level arm stored on published requests. It also compares stored non-user content-token totals per published session. The confidence interval is clustered by user. Evaluation coverage and the unattributed session count show whether evaluated sessions were published with experiment metadata; output_token_session_count shows how many published sessions have complete persisted token counts.

Response schemas

RetrievalExperimentListResponse

RetrievalExperimentRecord

RetrievalExperimentResultsResponse

RetrievalExperimentArmMetrics