Nothing Gets Deleted, Just Blurred

A strip of developed film holds every frame a photographer shot, but only as a thumbnail. Nothing on the strip is discarded, and nothing on it is sharp enough to read without a loupe or an enlarger. The decision about which frame gets printed at full size happens later, once someone knows which shot actually matters. Large language models handling long documents or long conversations face a version of the same choice, and for the last few years, most systems have chosen the opposite path, throwing frames away entirely rather than shrinking them.

The technical name for the film strip inside a language model is the key-value cache, commonly shortened to KV cache. Every time a transformer, the neural network architecture behind most modern language models, processes a token, it computes a key and a value vector for that token in every attention layer, the mechanism that lets the model weigh how much each earlier token matters to the one being generated now. Those vectors get stored so future tokens can attend back to them without recomputing them from scratch. Research from Google on scaling transformer inference showed early on that this cache grows linearly with how much text the model has read, and that the resulting memory pressure, not raw compute, is often what limits how long a context a deployed model can actually afford to hold ^[1].

Deciding Once and Living With It

Most existing fixes to this problem make an early, permanent call. Token eviction methods watch which tokens receive the least attention early on and drop them from the cache for good. Semantic compression methods group tokens into chunks during the initial read and replace each chunk with a single averaged summary before generation even starts. Both approaches work, and both share the same limitation, according to a team from the University of British Columbia and Microsoft Research studying long-context KV caching. A decision about what to keep gets made once, at the beginning, before the model has any idea which later question will actually depend on which earlier passage ^[2]. Evidence that looked irrelevant in the first paragraph of a legal filing or a codebase might turn out to be the exact clause a later question hinges on, and once it has been evicted or blended into an average, there is no way back.

The researchers propose something they call SeKV, short for semantic KV cache, which keeps every span of text in two forms at once rather than picking one. A lightweight summary vector for each span stays resident on the GPU, cheap to scan and used only to decide whether that span looks relevant to the current decoding step. A separate, more detailed representation of the same span, built from a technique called singular value decomposition that factors a block of numbers into a smaller set of components capturing most of its structure, sits on the CPU instead, waiting to be fetched only if the summary suggests it is worth the trip. Segment boundaries themselves come from a signal already available for free during the model's first pass over the text, a measure of how surprising each token is given what came before it, which tends to spike at genuine topic shifts and drop within a coherent run of text ^[2].

A Coarse View That Can Be Zoomed

A satellite photograph like this one shows enough to locate a shoreline or a mountain range, but not enough to make out a building or a road. Getting that level of detail means requesting a closer pass over one particular area, not over the whole frame. SeKV applies the same logic to context, routing over cheap summaries first and only reconstructing full detail for the handful of spans a given decoding step actually needs.

Whether a given span deserves that closer look gets decided by a small trained component, referred to as a zoom-in mechanism, that scores each summary against the model's current query and expands only the spans that clear a learned threshold. Across four long-context benchmarks, the reported result was a 5.9 percent average improvement over the strongest semantic-compression baseline tested, alongside a 53.3 percent reduction in GPU memory compared with keeping the full cache at a 128,000-token context length, achieved while adding fewer than 0.05 percent additional trainable parameters to a frozen base model ^[2]. The efficiency gain and the accuracy gain showing up together is worth noting mainly because compression methods usually have to trade one for the other.

A Token That Asks for a Refresh

A separate team, working across several Chinese research labs, tackled a closely related question from another angle. Rather than deciding what to keep once and reconstructing detail on demand, their system, called PReM for preserve and refresh memory, lets the model itself decide when its current compressed view of the context has gone stale and needs to be rebuilt ^[3]. PReM trains a special memory token that the model can emit mid-generation. Emitting it triggers a fresh look back over the full context, re-selecting which chunks deserve to be kept at full token-level detail and which get replaced with an averaged stand-in, based on whatever the model is trying to figure out at that specific step rather than on whatever seemed important when the context was first read.

Training a model to condition its own generation on a memory state that changes partway through, rather than staying fixed for the whole response, required the researchers to split each generation step into two separate forward passes, one that handles the refresh decision and one that generates conditioned on the result. On a set of question-answering benchmarks using 32,000-token contexts, PReM reportedly outperformed eight established KV-cache and context-compression baselines under both 16 times and 32 times compression, improving average exact-match and F1 scores by up to 10.23 and 12.55 points respectively over the strongest baseline tested ^[3]. A smaller 3-billion-parameter version of the model was also reported to outperform some larger 7-billion-parameter compression baselines, which the authors take as a sign that learning when to refresh can partly substitute for raw model scale, at least on the benchmarks tried so far.

Refreshing the Board Without Losing the Room

Wiping part of a whiteboard clean does not erase the meeting that produced the notes, only the summary written down at the time. PReM's memory token works on a similar principle, refreshing the compressed view the model is currently attending to while the full source context stays available underneath, ready to be resummarized differently the next time a refresh is triggered.

Memory That Outlives a Single Document

Both SeKV and PReM operate within a single long document or a single extended answer. A different line of work asks what changes when the relevant memory spans many separate conversations with the same user over time, a setting closer to how a deployed assistant actually gets used. A team centered at the University of Electronic Science and Technology of China proposes LightMem, which splits an agent's memory into three tiers, a short-term store for the immediate conversation, a mid-term store of reusable summaries from recent sessions, and a long-term store of consolidated knowledge, each managed by a small, purpose-built language model rather than a large one ^[4]. Retrieval happens in two cheap stages, a coarse vector search followed by a semantic consistency check, before anything gets handed to the main model. Reported results on a long-term dialogue benchmark showed roughly a 2.5-point average gain in F1 score across model scales, with a median end-to-end latency around 581 milliseconds and retrieval alone completing in about 83 milliseconds ^[4]. The appeal of the approach, from a systems perspective, sits less in the accuracy number and more in the latency figure, since memory operations that run on small models rather than the main large model avoid competing for the same expensive compute.

A related but distinct idea comes from a team at the University of Science and Technology of China, who point out that agents which compress a long document into memory through a single linear pass, reading start to finish and updating a running summary as they go, tend to prune evidence early that only turns out to matter once a much later part of the document has been read ^[5]. Their system, ReMemR1, lets an agent issue what the authors call a callback query mid-reasoning, retrieving an earlier memory state instead of only ever moving forward through it, and trains the behavior with a reward signal that combines the final answer's correctness with denser, step-level feedback about whether a given callback was actually useful. On long-context question-answering benchmarks, the reported result was upward of a 20 percent relative reduction in error rate compared with linear memory baselines, with the retrieval mechanism adding less than 0.2 percent to overall computation time ^[5].

What This Suggests

Four independent groups converging on some version of the same idea, that compression decisions should stay reversible and resolution should be allocated on demand rather than fixed up front, is a pattern worth watching rather than a settled conclusion. Every result above comes from a specific benchmark and a specific context length, 32,000 tokens for PReM, 128,000 for SeKV, documents padded to a few thousand entries for ReMemR1, and none of them yet speaks directly to million-token agent contexts or to workloads where the CPU-to-GPU transfer that SeKV and similar systems depend on becomes a bottleneck of its own under real production load. Whether the accuracy gains reported here persist once these methods leave curated benchmarks and meet the messier, longer-horizon contexts that production agents actually accumulate remains an open question, and one that will likely need answering system by system rather than in the abstract.