Playbook Optimization

Reflexio can optionally optimize playbook wording with a GEPA-backed candidate search loop. The optimizer is disabled by default.

The optimizer compares real assistant behavior, not just playbook text. For each source interaction window, Reflexio calls the configured assistant backend twice with the same user turns:

• incumbent playbook content
• candidate playbook content

Only the injected playbook content changes. A judge model scores the paired rollouts and records actionable feedback such as regressions, missing instructions, and recommended mutations.

By default, optimization holds out up to two source windows for validation and uses all remaining source windows as the GEPA training pool. GEPA samples training minibatches from that pool, while commit thresholds are checked only against the held-out validation windows. If the validation holdout is smaller than min_commit_windows, Reflexio skips GEPA because no winner could satisfy the adoption threshold.

With the default min_commit_windows=2 and max_validation_windows=2, the optimizer needs at least three source windows to run — two are held out for validation and one becomes the training pool. Targets with one or two source windows are skipped under the defaults; lower min_commit_windows to 1 to opt into single-window or two-window optimization. Reflexio also caches GEPA evaluations across iterations so the same candidate/window pair is never scored twice.

Single-window optimization is eligible when min_commit_windows=1. In that mode, the same source window is used for training and validation, and Reflexio asks GEPA to reuse the training set as the validation set.

The default GEPA budget is max_metric_calls=20 and max_turns=4 per run — earlier releases shipped with 40 and 5 respectively. The lower defaults keep nightly optimizer runs cheaper; raise them when you observe under-converged candidates.

Agent playbooks snapshot the source interaction windows that produced them during aggregation. This lets optimization replay the original scenarios even if the contributing user playbooks are later archived, deduplicated, or replaced by reflection.

Safety Model

Agent playbook optimization only targets current PENDING agent playbooks. APPROVED agent playbooks are never modified, archived, or replaced.

When auto_update_pending_agent_playbooks is enabled, a winning candidate replaces only the pending row and the successor remains PENDING. Human approval is still required before the optimized agent playbook can affect production retrieval.

User playbook optimization is separately gated with optimize_user_playbooks and only runs when auto_update_user_playbooks is enabled. With the default auto_update_user_playbooks=false, Reflexio skips user-playbook GEPA runs because there is no configured adoption path.

Assistant Contract

The assistant backend can be an HTTP webhook or a local script. Configure exactly one of webhook_url or assistant_script_path.

Both backends receive the same JSON payload:

{
  "messages": [{ "role": "user", "content": "..." }],
  "playbooks": [{ "id": 123, "content": "...", "trigger": "..." }]
}

It must return:

{ "content": "assistant response" }

For a local script, Reflexio invokes [assistant_script_path, ...assistant_script_args], writes the JSON payload to stdin, and reads the JSON response from stdout.

Configuration

Use playbook_optimizer_config:

{
  "enabled": false,
  "optimize_agent_playbooks": true,
  "optimize_user_playbooks": false,
  "auto_update_pending_agent_playbooks": true,
  "auto_update_user_playbooks": false,
  "webhook_url": "https://assistant.example.com/reflexio-rollout",
  "assistant_script_path": null,
  "assistant_script_args": [],
  "max_metric_calls": 20,
  "max_turns": 4,
  "early_stop_score": 0.9,
  "max_validation_windows": 2
}

The optimizer also exposes budget and threshold controls such as max_metric_calls, max_turns, early_stop_score, reflection_minibatch_size, min_commit_windows, min_commit_score, and min_commit_likert.

Internal architecture notes live in docs_for_coding_agent/reflexio_overview.md.