Mark Williams
Mark Williams
A modern office lobby with a reception desk and elevators behind it

A model that answers "I can't help with that" sounds like a model that can't. Ask a chat assistant how to make a nerve agent or write working ransomware, and the polite refusal reads as a statement about what the model is able to do. Research from the past three years suggests it is usually a statement about what the model has been trained to say first.

The lobby of an office tower is a useful picture. A guard at the front desk checks badges and turns visitors away, and to anyone standing at the entrance the building looks closed. The offices upstairs are still full. Someone who comes in through the loading dock, or who is already past the desk, finds every floor unguarded. Safety training, the stage where a model learns to decline harmful requests, appears to work a lot like that guard. It is very good at the front door and much weaker everywhere else.

An earlier Thinkata piece on machine unlearning found that methods meant to make a model forget mostly make it quieter. Refusal raises the same question from the other side. A model that declines might be unable to help, or it might just be unwilling at that moment. The difference matters a great deal to anyone deciding how much to trust it.

A Guard Who Only Watches the Door

A language model writes its reply one token at a time. A token is a word or a piece of a word, and each new one is chosen based on everything written so far. To find where safety training actually changes that process, a team from Princeton and Google DeepMind compared a safety-trained model with the base model it started from [1]. The differences were concentrated in the first few tokens of the reply. After that, the two models behaved much alike.

One experiment in the paper makes the point vividly. The researchers took a base model that had never had any safety training and simply forced its reply to begin with the words "I cannot." That one change took it from giving harmful answers most of the time to almost never, close to the safety-trained version. A large share of what looks like learned judgment, in other words, can be reproduced by getting the opening words right.

The reverse works too. In a prefilling attack, the attacker writes the first few words of the reply, something like "Sure, here is how," and lets the model continue. Some model providers allow this directly, and anyone running an open model can do it trivially. The more harmful words were placed at the start, the more often the safety-trained model carried on as if it had agreed. The paper received an outstanding paper award at ICLR 2025. It calls the pattern shallow safety alignment, and much of the follow-up work has adopted the term.

Why would training settle on the doorway? A 2026 preprint from the University of Southampton offers an explanation rooted in how these models generate text [2]. Once a reply has started down a path, the model strongly tends to keep going along it, a property the authors call autoregressive consistency. A refusal that opens with "I'm sorry, but" almost writes itself from there. Training has little left to correct in the later words, so its changes land on the opening ones. On this account, the shallowness is less an accident of particular datasets than a side effect of next-word prediction itself. The argument rests on a theorem with stated assumptions and on tests of a few open models, and it has not yet been peer reviewed.

One Switch on the Wall

If refusal mostly lives in the opening words, what decides which opening the model picks? Researchers looking inside thirteen open chat models, the largest with 72 billion parameters, found what looks like a single control [3]. Deep in a model's internal activations, the running numerical state it carries while processing a prompt, there was one direction associated with refusing. Erasing that direction left the models answering harmful requests they had previously declined. Adding it made them refuse perfectly harmless ones. Their other abilities came through largely intact.

A separate study arrives at a compatible picture by removing pieces of a model rather than steering it [4]. The parts that matter most for safety turned out to be small and separate from the parts that make the model useful. They made up around 3 percent of its parameters, the adjustable numbers a model learns during training. Remove them and safety breaks down while the model's usefulness barely changes.

In the lobby picture, the guard answers to a single switch on the wall, and the switch is not wired to the offices upstairs. That separation is what makes the arrangement fragile. A capability and the refusal to use it are stored in different places, so it is possible to touch one without disturbing the other.

Retraining the Guard

Most people will never edit a model's internal activations. Many more can fine-tune one, which means giving it a short round of extra training on examples of their choosing, a service several model providers sell. In 2023, researchers made a safety-trained model comply with nearly any harmful instruction using just ten carefully written examples [5]. Fine-tuning on them through a commercial service cost under twenty cents. More troubling for anyone offering fine-tuning as a product, training on ordinary, harmless datasets also eroded safety, less dramatically but noticeably, even when nobody intended it. The shallow alignment paper suggests a reason. Ordinary training on benign data tends to move the opening tokens most, which is exactly where the guard stands [1].

An earlier analysis framed the deeper issue as mismatched generalization [6]. A model's abilities reach widely, to unfamiliar languages, encoded text, and odd phrasings, because pretraining on vast amounts of text taught them broadly. Safety training covers a much narrower range of examples. Wherever the two diverge, the ability is present and the refusal is not. The authors argued that safety measures would need to be roughly as capable as the model they protect, and that scale alone would not close the gap.

A long empty hallway with doors on both sides

Guards on Every Floor

Every door along a corridor is another place to slip in, and a guard posted only in the lobby never sees most of them. The obvious repair is to train the model to recover partway through a reply, not just at the start. The shallow alignment paper did this with what it calls safety recovery examples. These are replies that begin harmfully and then turn back to a refusal. Training on them made prefilling and several other attacks far less effective [1]. The same paper proposed a training rule that limits how far later fine-tuning can move the first few tokens, guarding the doorway itself.

Follow-up work suggests the corridor is longer than it first appeared. The Southampton group's random insertion attack drops a short harmful passage into the middle of an otherwise firm refusal. Models often pick up the harmful thread and continue it, even after refusing for a long stretch [2]. A second 2026 preprint, from researchers in Seoul, found that injecting a few words at almost any point in a reply could change what the model did next [7]. The effect was strongest early on, but no point in the reply was fully safe. Both groups trained models on deliberately interrupted replies and reported large drops in attack success.

The Seoul study adds a complication. Models whose internal state lined up strongly with the refusal direction were not, as a result, any harder to push off course. A model can look internally committed to refusing and still give in. The single switch may be a handy lever for attackers, then, without being a reliable gauge for defenders. The deeper training also made two of the models turn down harmless requests noticeably more often, a reminder that a guard on every floor costs something in friction.

Locking the Rooms

A different line of defense leaves the refusal alone and acts on the capability. Circuit breakers, introduced in 2024, watch for the internal patterns a model produces when it starts generating harmful content, and scramble them as they appear [8]. The model loses its train of thought instead of declining politely. Because the method targets what the model is doing rather than how its reply begins, a prefilled "Sure, here is" does not get around it in the same way. The authors report strong resistance to attacks the system was never trained against, without much loss in usefulness. In the lobby picture, this is closer to locking the rooms upstairs than to adding guards.

Locked rooms are still full rooms. The knowledge remains in the model, and someone with access to the weights could in principle retrain the lock away. That leads back to the unlearning question. A model that truly cannot help would have to lack the ability, and current methods struggle to show they have removed anything at all.

What a Refusal Can Honestly Claim

From a systems perspective, the evidence points toward treating a refusal as a behavior observed under particular conditions, rather than as a fixed property of the model. A claim that a model declines harmful requests is only as strong as the list of access routes it was tested against. A closed model behind an interface that blocks prefilled replies and custom fine-tuning faces far fewer attacks than the same model with those features turned on. An open model, whose weights anyone can download, faces all of them, including direct removal of the refusal direction. Decisions about which features to expose are, in effect, safety decisions. Earlier Thinkata pieces on enforcement that sits outside the model and on isolation between customers make a related case for controls that do not depend on the model's own good behavior.

The harder question is how anyone could tell a model that will not from a model that cannot. The closest available test is to deliberately try to undo a model's safety training and then check what it can still do. Like the relearning attacks in unlearning research, that only sets a floor, since a more determined attacker might recover more. Specialists in biosecurity or cybersecurity would know better than a systems reading can how much leftover ability actually matters in their fields. The research so far does make two things fairly clear. The guard at the front desk is doing real work, and the building behind the guard is still full.

References

  1. X. Qi et al., "Safety Alignment Should Be Made More Than Just a Few Tokens Deep," in Proc. International Conference on Learning Representations (ICLR'25), 2025, [Online]
  2. B. Lyu et al., "When Autoregressive Consistency Hurts Safety Alignment," arXiv, 2026, [Online]
  3. A. Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction," in Advances in Neural Information Processing Systems, vol. 37, 2024, [Online]
  4. B. Wei et al., "Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications," in Proc. 41st International Conference on Machine Learning (ICML'24), 2024, [Online]
  5. X. Qi et al., "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!," in Proc. International Conference on Learning Representations (ICLR'24), 2024, [Online]
  6. A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How Does LLM Safety Training Fail?," in Advances in Neural Information Processing Systems, vol. 36, 2023, [Online]
  7. K. Park and T. Kim, "Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories," arXiv, 2026, [Online]
  8. A. Zou et al., "Improving Alignment and Robustness with Circuit Breakers," in Advances in Neural Information Processing Systems, vol. 37, 2024, [Online]

Working on something related?

Thinkata builds production AI systems of the kind these articles discuss. A 20-minute call is the fastest way to find out whether it fits.

Or Send Message