Orchestration Without a Signal From the Room

A conductor who has only ever studied the score is not the same as a conductor who has stood in front of that particular orchestra. The score already knows which entrance belongs to the horns. What it cannot tell anyone is whether those horns will actually arrive on time tonight, or whether the first violin is sitting a half-beat late, or whether the room itself is swallowing the low strings. Those facts only show up in rehearsal.

Most production agent stacks are still closer to the score. A large language model, a system trained to generate text by predicting the next word, gets wrapped in a graph of specialists with names like planner, executor, critic, and coder. Each node is a frozen model plus a prompt. Tools get called. Memory gets appended. The graph looks like architecture. Orchestration, the work of deciding what happens next, which tool, which sub-goal, whether the last result was even usable, often does not receive a learning signal from the trajectory it just caused. The room already answered. The graph did not listen.

From a systems perspective, that missing wire is the whole story. A deployed controller that cannot hear the plant will keep conducting from last week's diagram. Prompt edits after a surprising failure are not the same as a gradient, the training signal that tells a model how to change. They are a human rewriting the score by hand.

A Score With Every Entrance Marked

The conversation frameworks that made agent graphs easy to ship did something useful. They made orchestration programmable as talk among specialists, humans, and tools, without requiring a training loop for that particular collaboration ^[1]. The graph became a product surface. Roles could be named. Handoffs could be coded. What those designs did not attach, by construction, was a learning signal to the talk itself. When a search result was noisy, a code tool returned an exception, or an early sub-goal was the wrong one, the usual fix was another prompt. Prompting problems tend to be rewritten after every surprising failure.

Every Switch Already Has a Label

A room like this looks finished. Every circuit has a nameplate and a home position. What it does not have is a way for the person at the board to get better at using it from the last shift's near misses. Frozen agent graphs have the same finished look. The planner, executor, and verifier all have roles. In the default design, none of those roles takes a gradient from what the last tool call actually returned.

That is orchestration without a signal from the room. It can still run a job. It cannot, on its own, get better at sequencing the job from the job's own outcome.

A Transcript Is Not the Room Either

The obvious patch is to train the orchestrator after all, just not in the live loop. Collect traces from a stronger model, copy them token by token, and call the result learning. The traces look like rehearsal notes. They are still a score. They were written for a different night, a different band, a different set of missed entrances.

Work on in-the-flow agent training put that patch to a direct test. Inside the same modular graph, a planner, an executor, a verifier, and a generator sharing a structured memory, distilling a stronger model's planner traces into a smaller planner produced a collapse, a 19 percent drop in average accuracy relative to leaving that planner frozen ^[2]. The imitation objective was answering the wrong question. It rewarded looking like a good transcript, not finishing the work after a tool had already gone sideways.

The same graph, with the planner updated from the live rollouts it actually induced, recovered that ground. On-policy training, updates from the trajectories the current system produces rather than from a cleaned-up archive, is what puts the orchestrator back in the room. The planner still sees the live state, the query, the toolset, the current memory, because that is the state it will face at inference. After that training, tool choice moved with the task instead of staying at a default search habit. Broad factual questions pulled more web search. A medical question set spent more of the budget inside Wikipedia and page-level retrieval ^[2]. A frozen prompt is supposed to produce that kind of shift and often does not, because the prompt cannot see, in gradient terms, which of those choices ended in a correct answer.

The lesson is narrower than "train a planner." It is about which signal counts. A transcript is a record of how someone else conducted a different night. A live outcome is what this orchestra, in this room, actually did.

Dumping the Outcome Through the Whole Band

Some stacks do send a final correct-or-incorrect score back through the model that issued the tool calls. The search engine sits in the environment. The model learns when to query and how to use what comes back, with retrieved tokens masked so the update does not try to credit the search engine's text as if the model had written it ^[3]. That is a real signal. It is also a signal dumped through every thought, every tool choice, and every wording decision in one long context.

The room is noisy. Tool output is often off the pretrained distribution. Even when those tokens are masked from the loss, the model's next generation inherits the shift, samples increasingly unlikely tokens, and the gradient can explode. One practical response has been to throw out entire trajectories that contain a void turn, a response that produced neither a code block nor a final answer, because those turns were pulling the update in the wrong direction ^[4]. The filter is a way of saying the room spoke, and some of what it said was not a usable learning signal.

A related failure shows up when the only reward is that the episode worked. Reward variability collapses, gradients spike, and the agent starts repeating locally rewarded patterns that do not amount to reasoning. Shallow strategies and hallucinated thoughts are easy to grow if the signal has nothing to say about whether the thoughts were real ^[5]. That sits next to an earlier Thinkata look at temporal credit assignment. A single terminal score is a hard thing to send backward through a long chain. Stuffing it through the entire policy makes the chain even longer.

The Wire Has to Reach the Loop That Chooses

A working plant does not rewire every gauge because one loop is drifting. It retunes the controller that actually sets the next action. The learning signal has to land somewhere specific. Spread it across every token in a growing context and the noise in the room can swamp the update. Leave it unconnected and the labeled panel does not learn from the last shift.

The engineering question is not whether a terminal outcome is too crude. Outcome rewards are often the only score that can be checked. The question is where that crude score is allowed to change the system. One measured answer has been to keep the modular graph and copy a single verifiable result, right or wrong at the end, onto every orchestrator turn, while each turn still conditions on the full memory so far ^[2]. The local decision is not blind. The update still points at whether the job finished. That is a systems choice about which module is allowed to hear the room, not a new theory of credit.

How Much of the Room Needs to Hear It

Training only the conductor is a bet about the rest of the stack. Tools, executors, and verifiers can stay versioned infrastructure. When the toolset or the task mix changes, the thing that gets retrained is the loop that chooses, not the whole orchestra. That matches how a lot of production software is already pinned.

It is also incomplete in a specific way. If the models that actually run the steps stay frozen while only the designer or planner learns, part of the loop is still conducting from a score. Work on automatic multi-agent systems has started treating that gap as a ceiling. Jointly training the side that writes the workflow and the side that executes it produced gains that leaving either side frozen did not, and the two roles appeared to improve on a staggered schedule rather than all at once ^[6].

That does not settle how much of an agent graph should be trainable. For a team whose executor is a deterministic API, a search endpoint, a code runner, a pinned verifier, putting the learning signal on the orchestrator is the operable experiment. For a team whose executors are themselves language models with their own failure modes, freezing them may be the part of the room that later looks deaf. Domain practitioners will know better than a systems argument whether a given executor is a tool or a second conductor.

What does seem worth taking seriously is the missing wire itself. A prompt-orchestrated graph is a practical way to run specialists. It does not, by itself, teach the module that sequences them. Sending the outcome through every token of a monolithic policy can teach that sequencing, and then spend the training budget on not exploding. The option that matches the rest of a production stack is smaller. Pin the tools. Pin the memory format. Attach a learning signal to the loop that is actually conducting, from the trajectory that loop just caused, in the room it is standing in tonight.