When Replay Is Not an Option

A river never pools before it moves on. Water arrives from upstream, passes a given point exactly once, and continues toward the sea without waiting to be collected first. Reinforcement learning, the branch of machine learning where a software agent learns by acting in an environment and receiving rewards, was originally built the same way. Q-learning and SARSA, two classic algorithms that both estimate the future payoff of taking a given action in a given situation, were designed to update from one experience at a time and move on, a mode of operation known as streaming or online learning ^[2]. Both trace back to value iteration, the older dynamic-programming idea of repeatedly correcting a value estimate until it stops changing.

Modern deep reinforcement learning rarely runs this way anymore. Since Deep Q-Networks combined Q-learning with neural networks to reach competitive play on Atari games, most systems have leaned on a replay buffer, a large memory bank that stores past experience so it can be resampled again and again in batches ^[1]. Averaging many samples together before each update smooths out noise and lets the same experience teach the network more than once. It works well, but it assumes there is somewhere to put the water while it waits its turn.

Why the River Stopped Flowing

Not every system can afford that assumption. A small robot with a modest onboard processor, a wearable sensor, or a system bound by strict data-privacy limits may not have the memory, bandwidth, or permission to keep raw experience around for later replay ^[2]. When the water cannot be stored, learning has to happen as it passes through, or not happen at all.

When researchers tried simply removing the replay buffer from standard deep reinforcement learning algorithms, the results were not encouraging. Learning became unstable or collapsed outright across several benchmark tasks, an effect one recent paper terms the stream barrier ^[2]. Something about updating incrementally, without the averaging effect of a batch, exposed weaknesses that batch learning had quietly been covering up. Both Q-learning and SARSA compute what is called a temporal difference error, the gap between what the network predicted and what actually happened one step later, then nudge the estimate a small amount toward closing that gap. In a deep network, that nudge is scaled by a step size, and getting the step size wrong turns out to matter far more without a batch to soften the blow.

One proposed fix combines several older ideas that had fallen out of fashion. Eligibility traces, a short-term memory that decays gradually and lets a single reward influence not just the most recent action but several that came before it, were reintroduced alongside layer normalization, a technique that keeps a neural network's internal activity at a stable scale over time, and a sparse pattern of initial connections that reduces interference between unrelated inputs ^[2]. Bundled into algorithms called stream Q, stream SARSA, and stream actor-critic, these techniques reportedly let a network learn from Atari games, robotic control benchmarks, and a real-world electricity demand forecasting task as each experience arrives, matching or outperforming batch methods on several of them ^[2].

Tuning the Flow Without a Gauge

A valve wheel like this carries no readout of how much water is actually moving through the pipe behind it. Whoever turns it has to judge the right amount by feel, guided mostly by what happened the last time it was turned this far. Step size, the setting that controls how large a correction a learning algorithm makes after each mistake, has traditionally been chosen the same way in streaming reinforcement learning. Too far, and the update destabilizes the model. Too little, and nothing changes fast enough to matter.

Choosing the Step Size on Purpose

A separate line of work asks a different question about that same wheel. Rather than picking a step size measured in the units of a network's internal weights, why not decide first how much the model's actual prediction should change, then solve backward for whichever step size makes that happen, sidestepping the guesswork that made the stream barrier so damaging in the first place ^[3]. Researchers working alongside Richard Sutton, one of the field's founding figures, call the result intentional updates. Instead of fixing how far to turn the wheel, the method fixes how much water should flow and calculates the turn needed to get there ^[3]. Applied to temporal difference learning and to Q-learning, the approach is reported to reach streaming performance that is often comparable to batch and replay-buffer methods across the domains tested so far ^[3]. Whether that holds up on tasks with much longer horizons than the current benchmarks is a question the authors leave open, and one further work will need to settle.

Learning on the Device Itself

A board this size holds only what is happening right now, with no spare memory set aside for a warehouse of stored samples. That is not an edge case for streaming reinforcement learning so much as close to the point of the exercise.

From Simulation to Real Hardware

Researchers recently tested an incremental policy gradient method, a family of algorithms that adjust a decision-making policy directly rather than through an intermediate value estimate, under the name Action Value Gradient. On simulated robotic control benchmarks, it was reportedly the only incremental method among those compared that learned effectively without a replay buffer, target network, or batch update ^[4]. It was later used to train a robotic manipulator arm and a mobile robot with only real-time, incremental updates, which the authors describe as the first demonstration of effective deep reinforcement learning on physical robots restricted to incremental learning ^[4]. The result matters less because of the specific robots involved and more because it suggests the resource savings from dropping batch machinery do not necessarily come at the cost of learning ability, at least on the hardware and tasks tested.

Where the Simpler Algorithms Already Work

None of this implies classical Q-learning and SARSA need deep networks or streaming tricks to be useful today. In a 2025 comparison of microgrid energy management, a setting involving batteries, solar generation, and shifting demand, researchers found that plain tabular Q-learning and SARSA still produced meaningful cost savings compared with having no learning agent at all, even though a deep Q-network outperformed both by a further margin ^[5]. Whether that gap between simple and deep methods narrows or widens as more real energy systems get instrumented is a question best left to people closer to that domain. From a systems perspective, though, it is a useful reminder that the current streaming revival is not solving a problem that never existed before. It is trying to bring the modeling capacity of deep networks into settings where the older, simpler algorithms already had to operate without a replay buffer by design.

What This Suggests

Streaming reinforcement learning was not so much invented recently as recovered. Q-learning and SARSA were streaming algorithms from the outset, descendants of value iteration's habit of repeatedly correcting an estimate against a moving target. Batch learning became dominant afterward, once neural networks entered the picture, partly because batching hid instabilities that incremental updates expose immediately. The recent work on the stream barrier, intentional step sizes, and incremental policy gradients does not eliminate that instability. It builds tools aimed at the constraint that made replay buffers attractive in the first place, limited onboard hardware and experience that can only be used once. Whether these methods extend cleanly to longer, messier tasks than the current benchmarks cover remains an open question, one likely to be answered one small robot and one edge device at a time.