Playbook Optimization

Playbook optimization is an optional refinement step for playbooks that already exist. It asks a narrow question: would this playbook steer the assistant better if the guidance were worded differently?

It does not replace extraction, aggregation, or human review. Reflexio still learns signals from interactions, aggregates repeated user-playbook patterns into agent playbooks, and requires approval before an agent playbook becomes production guidance. The optimizer sits between aggregation and approval, sharpening pending guidance before a human decides whether to approve it.

Where Optimization Fits

The normal playbook path is:

Optimization is intentionally late in the path. By the time it runs, Reflexio already knows which real interaction windows produced the playbook. That gives the optimizer scenarios it can replay when comparing the original wording against candidate wording.

What It Changes

The optimizer focuses on playbook content: the natural-language instruction injected into the assistant. It can make the guidance clearer, more specific, less ambiguous, or less likely to cause regressions.

It is not a discovery engine for brand-new lessons. If the underlying playbook is about respecting a user's stated budget, optimization should improve how that budget rule is phrased. It should not turn the row into an unrelated recommendation strategy.

For agent playbooks, a winning candidate replaces only a current PENDING row. The successor remains PENDING, so the normal approval workflow still decides whether it can affect production retrieval.

How a Run Works

Each optimizer run targets one playbook.

Only the injected playbook content changes between the paired rollouts. The scenario, messages, and assistant backend stay the same, so the comparison is about whether the candidate guidance produced better behavior.

Source Windows

Source windows are the evidence behind optimization. For agent playbooks, Reflexio snapshots the contributing interaction windows at aggregation time. That makes later optimization replayable even if the original user playbooks are archived or deduplicated.

By default:

The former use_merge flag is accepted for compatibility but omitted from saved configuration. A legacy false value normalizes the merge limit to zero; otherwise the existing limit is retained. New configurations use only max_merge_invocations. Re-enable merging by setting a positive budget. Existing job authority keeps its original recorded snapshot; normalization does not rewrite historical jobs.

With the defaults, an agent playbook needs at least three source windows to run: two validation windows and one training window. If a target has only one or two windows, Reflexio skips it because no candidate could satisfy the default commit threshold.

To allow one-window or two-window optimization, lower min_commit_windows to 1. When only one source window is available, Reflexio uses that same window for training and validation. That is useful for small local workflows, but it is weaker evidence than a held-out validation set.

Reflexio caches evaluations during a run, so the same candidate/window pair is not scored twice.

Safety Model

Optimization is gated at several points:

APPROVED agent playbooks are never modified, archived, or replaced by this optimizer. This keeps approved production guidance stable until a human intentionally changes it.

User-playbook optimization and adoption share user_playbooks_enabled, default false. When it is off, no optimization job or assistant rollout is started. Older configurations using optimize_user_playbooks and auto_update_user_playbooks remain accepted: both must be true to enable the new control, and either omitted value defaults to false. An explicitly supplied user_playbooks_enabled takes precedence; canonical saves use only that field. Historical publication authority retains its recorded policy unchanged.

Retrieved-learning evaluations can diagnose a playbook's content, application, or external failures. Those diagnoses are independent of the enterprise offline tuner and are not a prerequisite for its evidence selection. See playbook diagnosis and tuning.

Enterprise offline-tuner edits use a separate audit path. Rows created by that tuner use source="offline_optimizer" so operators can distinguish offline tuner mutations from extracted, reflected, manual, or GEPA optimizer rows.

Assistant Backend

Optimization needs a way to ask your assistant, "how would you respond with this playbook?" Configure exactly one backend:

Both receive the same JSON payload:

{
  "messages": [{ "role": "user", "content": "..." }],
  "playbooks": [{ "id": 123, "content": "...", "trigger": "..." }]
}

Both must return:

{ "content": "assistant response" }

For a local script, Reflexio invokes [assistant_script_path, ...assistant_script_args], writes the JSON payload to stdin, and reads the JSON response from stdout.

Minimum Configuration

The optimizer is off by default. To optimize pending agent playbooks, enable the master switch, enable the agent-playbook target, and provide one assistant backend.

{
  "playbook_optimizer_config": {
    "enabled": true,
    "optimize_agent_playbooks": true,
    "user_playbooks_enabled": false,
    "auto_update_pending_agent_playbooks": true,
    "webhook_url": "https://assistant.example.com/reflexio-rollout",
    "assistant_script_path": null,
    "assistant_script_args": [],
    "max_metric_calls": 20,
    "max_turns": 4,
    "early_stop_score": 0.9,
    "max_validation_windows": 2,
    "min_commit_windows": 2,
    "min_commit_score": 0.75,
    "min_commit_likert": 4
  }
}

Use assistant_script_path instead of webhook_url for local evaluation. Do not set both; Reflexio rejects configurations with two assistant backends.

When To Use It

Use playbook optimization when you already have pending agent playbooks and want stronger wording before review. It is especially useful when:

Skip it when you do not yet have enough source windows, when the assistant backend is not representative of production, or when the pending playbook needs human product judgment rather than wording refinement.

Enterprise Offline Tuning

Enterprise has an evidence-driven path for current user playbooks, surfaced as Playbook Tuning. See Playbook Tuning for integration and operational details. Availability depends on deployment setup, configuration reset and completion of the tenant's method-schema upgrade; this page does not assert that every hosted deployment has already upgraded.

The method reads owned, reconstructable sessions that used the exact incumbent version. It needs no minimum success/failure counts. A judge records cited evidence across four dimensions: task completion, constraint compliance, evidence grounding and execution effectiveness. It separates an instruction gap from an agent's failure to follow adequate instructions. The tuner selects a supported gap and proposes replacement text for one unique clause; code preserves the surrounding content and playbook metadata. No supported gap means no edit.

This differs from GEPA, which sharpens pending agent playbooks behind a human approval gate. Enterprise tuning can publish a user playbook version through its existing authorized publication path, and keeps that version until another supported revision. There is no extra held-out semantic judge or completion-label gate. Missing historical context remains unknown. Ownership, governance, citations, lease and stale-version checks, search projection and atomic publication still protect the write. Structural checks are not proof of semantic truth; observed production issues inform subsequent experiments.

Enablement is gated separately from that work. An explicit configuration write that sets offline_tuner_config.enabled=true is rejected by a capability check that does not consult Phase 3 at all, so Phase 3 composition is not what makes enablement unavailable. That write returns HTTP 409 with a stable detail.code, using the first cause that matches:

enabled=false is always accepted, and no unavailable outcome creates a tuner job. At startup, configuration loading resets persisted pre-cutover tuner configuration to exactly {"enabled": false} and verifies the write before ordinary configuration parsing continues. In platform mode, a loaded enabled configuration is also normalized to disabled at runtime when no generation model is configured.

Related

Understanding checks that wait

The tuning settings show currently retained completed results for today and their reasons for waiting. A later attempt can replace an earlier result, so these counts are not cumulative and the latest available check may move backwards. No available completed result does not mean tuning has never run. A completed check does not necessarily produce a playbook revision: it revises only when cited session evidence supports a specific instruction gap. Infrastructure failures are not counted as agent mistakes.

Some search records may not connect to a published conversation. This can happen when a turn is cancelled or an integration searches without publishing its response. Keep the same request, user, and session identities across search and publication, and report the playbooks provided to the agent. Claude Smart keeps this request identity in its local buffer across publish retries. New traffic can supply missing evidence; the settings counts do not promise that any particular playbook will qualify or be revised daily.

Explore Related Guides