Two teams tell you their agent learns from experience.
The first retrains model weights every night on its own successful trajectories. The second appends a line to a markdown file when a user corrects it.
Both are telling the truth. Neither description tells you anything useful about the other, and the phrase they share has stopped carrying information.
That is a problem worth fixing, because the choice between those two systems is not a matter of sophistication. They fail differently, cost different amounts, need different evidence to justify themselves, and are appropriate in different situations. A team that picks one because it read a paper about the other has not made a decision — it has made a guess.
This post is the map. The five that follow it walk the territory.
One loop, four parts
Strip away the vocabulary and every method in this space is the same loop.
Something proposes a change. A reflection on a failed attempt, a rewritten instruction, a gradient step, a new entry for a memory store.
Something decides whether to keep it. This is the part that varies most and gets discussed least.
Something stores it. The change has to land somewhere that outlives the current session, or nothing has been learned.
Something measures whether the agent is now better. Not whether the change looks reasonable. Whether the agent's outcomes improved.
The first three are present in every system. The fourth is frequently absent, and its absence is almost never stated — a system with no measurement step still runs, still produces changes, still generates the pleasant sensation of progress. It just cannot tell you whether any of it worked.
Three questions locate any method on this landscape.
What changes
The most visible axis, and the one papers are organized around.
Weights. The model itself is updated. Reinforcement learning from verifiable rewards, self-training on the model's own filtered outputs, and self-edit methods that let a model write its own training data all live here.
Context. The instructions, prompt, or system message the model runs under. Automated prompt optimizers rewrite this text; so, more crudely, does a human editing a system prompt after a bad demo.
Memory. A persistent store of facts, preferences, and past events, retrieved at runtime and injected before the model answers.
Tools and skills. The agent's action space. An agent that writes a function, verifies it runs, and files it in a library for reuse has extended what it is able to do, without any change to the model or the prompt.
The harness. The scaffolding around the model — how it plans, when it spawns subagents, how it manages context, what it does on failure. This is the newest category to be taken seriously, and the one with the most room left in it.
These are not exclusive. Production systems usually touch two or three.
What decides
The axis that actually determines whether a method works.
The model's own judgment. The system generates a change and asks a model whether it is good. Cheap, universally applicable, and structurally the weakest option available.
A verifiable reward. Unit tests pass or fail. The proof checks or it does not. The arithmetic is right or wrong. Where this signal exists it is close to unimpeachable, which is why the most convincing results in the field come from domains where it does.
A human. Someone reads a diff and accepts or rejects it. Expensive, slow, does not scale, and considerably more reliable than the first option.
A measured outcome. Run the old version and the new version against the same held-out cases, compare what each produced, keep the change only if it wins. Expensive in a different way — it requires the ability to re-run your agent faithfully — and the only option on this list that answers the question the loop was built to ask.
When it changes
Within an episode. The agent notices a problem and corrects course mid-task. Nothing persists; the next session starts fresh.
Between episodes. After a task ends, something is written down that the next task can use.
Offline, in batches. A separate process reads accumulated history when the agent is idle and rewrites some persistent state. Consolidation passes, prompt optimizers, and nightly fine-tunes all sit here.
The grid
Placing the well-known methods on all three axes at once makes the shape of the field legible in a way that reading them one at a time does not.
| Method | What changes | What decides | When |
|---|---|---|---|
| Self-Refine | The answer in progress | The model's own critique | Within an episode |
| Reflexion | Reflection text in episodic memory | Task feedback, then self-critique | Between attempts |
| Voyager | A library of executable skills | Whether the code actually runs | Between episodes |
| ExpeL, Agent Workflow Memory | Distilled insights and reusable workflows | The model's own judgment | Offline |
| Agentic Context Engineering | A structured playbook | The model's own judgment | Offline and online |
| Automated prompt optimizers | Instruction text | Score on a held-out set | Offline |
| RL from verifiable rewards | Weights | A checkable ground truth | Offline |
| Self-edit methods | Weights | Downstream task performance | Offline |
| Self-modifying coding agents | The agent's own source code | Benchmark score | Offline |
| Memory consolidation, "dreaming" | The memory store | Review, or recall statistics | Offline |
Two things fall out of the table immediately.
The first is that the "what changes" column is diverse and the "what decides" column is not. Six of these ten methods are decided, wholly or partly, by a model's opinion of its own work. That concentration is not a coincidence — self-judgment is the only decider available in every domain, so it becomes the default anywhere a verifiable reward does not exist.
The second is that the methods with the most convincing published results are the ones with the strongest deciders. Skill libraries gate on whether code executes. Verifiable-reward training gates on ground truth. These are not the most sophisticated proposal mechanisms in the table; they are the ones whose acceptance test cannot be talked out of a rejection.
The axis that gets read, and the axis that decides
Almost all the public conversation runs on the first axis. It is where the interesting engineering is, it is how papers are titled, and it is what a demo shows.
But the failures cluster on the second.
The clearest demonstration of this came from training a model against its own reference-free judgments and watching what happened. The judge's pass rate climbed from 0.72 to 0.94. True accuracy stayed at 0.20. The policy had not learned to be correct; it had learned to be convincing, because a judge shown a candidate answer scores how plausible it looks, not whether it is right. The effect transferred across model families, and a strict panel of three judges still accepted more than half of the bad answers.
That is not a bug in a particular judge, and a better prompt does not fix it. It is what happens when the thing being optimized is also the thing doing the grading.
This is the reason the series is organized the way it is. It is easy to write about self-improvement as a story about increasingly clever ways to change an agent. The more useful story is about evidence: what each method accepts as proof that a change was an improvement, and what that choice costs.
What the rest of this series covers
Learning without touching the weights — reflection, experience reuse, skill libraries, playbooks, and automated prompt optimization. The largest and most practical family, and the one where a small team can get real results this quarter.
Learning in the weights — verifiable rewards, self-training, and self-edits. What becomes possible when you have a checkable ground truth, and what it costs when you do not.
Memory as the substrate — the memory architectures everything else is built on, what their benchmarks actually measure, and the failure modes that arrive with scale.
The signal problem — credit assignment, self-authored verification, reward hacking, and the theoretical limits on how far a system can bootstrap itself. The spine of the series.
What to actually build — a decision procedure, what to measure, and an honest account of which parts of this are solved.
A closing note on posture. This is a young field with a serious replication problem: many reported gains are measured against weak baselines, several headline numbers come from the vendor selling the system, and the benchmarks are new enough that they are still being argued about in public. Where a number is contested in the posts that follow, both sides get named. Where something is unresolved, it is described as unresolved rather than smoothed over.
We build the measurement half of this loop at Reflexio — corrections held as rules a person can read, scoped to where they earned their evidence, tested before they ship. That work is why we read this literature closely, and it is a bias worth knowing about while reading a survey we wrote.
For the implementation boundary, the playbook optimization documentation describes supported candidate search, while the impact measurement guide explains how to inspect results.
Related reading on this blog: Your Agent Remembers Everything and Learns Nothing, on why memory layers and learning layers get mistaken for each other, and Your Agent Dreams. It Still Can't Tell You If It Got Better., on what consolidation passes actually validate. Reflexio is on GitHub, along with our open-source coding-agent plugin claude-smart.