learn · advanced
What safety training changes, and what it leaves in place
After this page: You can describe a refusal as a computation, say why a model that refuses still contains what it refused to produce, and explain why jailbreaks and ordinary fine-tuning defeat safety training through the same weakness.
Assumes How models are trained, and how they are changed afterwards
A refusal is not a gate. No component inside the network inspects a request and decides. The sentence declining to help is generated by the same forward pass, one token at a time, as any other sentence: it is the continuation the model was trained to find likely in that situation.
Filtering outside the model does exist — classifiers that block or rewrite either side of the exchange — and it is a different layer, changeable in an afternoon. This page is about what training did to the weights, which is a slower and considerably stranger thing.
Training adds behaviour; it does not subtract capability
Post-training moves probability mass. Supervised examples and preference optimisation — Direct Preference Optimization and its relatives — both work by making some continuations more likely and others less. Neither contains an operation that removes a representation.
So the accurate description of a safety-trained model is not "it can no longer do this". It is "under inputs resembling the ones it was trained on, it is now unlikely to". That distinction is testable, and the tests are the rest of this page.
Refusal is mediated by a single direction
The most useful result here is mechanistic, and it needs two of the field's words first. The activations are the numbers flowing through the network on one particular input, as against the weights, which are fixed and the same whatever you ask. Those numbers travel upward through the stack of blocks as a running total that each block reads from and adds back into, and that running total is the residual stream.
In Refusal in Language Models Is Mediated by a Single Direction, the authors found across thirteen open-weight chat models that refusal is carried by one direction in that stream: erase that direction from the activations and the model stops refusing harmful instructions, while its other capabilities stay largely intact. Add the direction on a harmless request and it refuses that instead. The same paper analyses adversarial suffixes, the odd-looking strings appended to a request to force it through, and finds them working by suppressing the propagation of that same signal. A very different looking attack, with an identical target.
Read structurally, this says safety training did not distribute the behaviour throughout the network. It installed something with a location, and a behaviour with a location can be found and moved. That is why stripping refusals out of a released set of open weights is a small, cheap edit rather than a retraining run, and why open weights and guaranteed refusals are in tension by construction rather than through anyone's negligence.
Alignment is shallow, and that explains a whole family of attacks
The second result: the difference between an aligned model and its base is concentrated in the first few tokens of the response. Safety Alignment Should Be Made More Than Just a Few Tokens Deep names the phenomenon and shows how much rides on it. Once a compliant opening exists in the sequence, the continuation follows something much closer to the base distribution. And no operation in the architecture revises an emitted token, so the model cannot back out of an opening it has already produced.
That one fact accounts for techniques that otherwise look like a list. Prefilling the assistant's turn with the first words of a compliant answer. Requiring the reply to begin with a fixed phrase. Filling the input with many examples of compliant exchanges, so that the pattern in progress is compliance. Anything else at all that gets past the first sentence.
They are not four techniques. They are one technique — get the model past the shallow region — approached from four directions.
The same shallowness explains benign fine-tuning
If the safety behaviour lives in the early-token distribution, anything that moves that distribution moves the safety behaviour, intent notwithstanding. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! measured both halves: a handful of adversarial examples was enough to strip the guardrails from a hosted fine-tuning endpoint, at a cost the authors report in cents, and fine-tuning on ordinary harmless task data degraded them too, less severely and with nobody trying.
This is the practically important one, because it is not an attack. It is the default outcome of adapting an aligned model to your own data. A team that fine-tunes for tone or format and ships the result has probably shipped a model whose refusal behaviour no longer matches the one they evaluated.
What the training signal actually optimises
Preference optimisation maximises agreement with raters' choices between responses. That is a proxy for a policy, and the gap between the two shows up in both directions.
In one direction it over-refuses. Requests sharing surface features with disallowed ones get declined, because the model learned the features that co-occurred with refusal during training, not a rule anybody wrote down.
In the other it under-refuses on a rephrasing. The same request in a different frame, register or language passes, because the features that triggered the learned behaviour are absent even though the request is not.
A model that declines to explain how a lock works and complies when the request is framed as fiction has not applied a policy inconsistently. It never had a policy. It has a learned response to a distribution of inputs, and the input moved.
What this does and does not mean
It does not mean safety training is theatre. Shifting the default behaviour of a system whose use is overwhelmingly ordinary is a large effect on the outcomes that actually occur, and most users never leave the distribution where the training holds.
It means the guarantee is behavioural and distributional rather than structural. When you read that a model "cannot" produce something, three distinct claims are being blurred: a classifier outside the model blocked it, which is inspectable and is what is most often meant; the weights make it unlikely under normal input, which is true and is what was trained; or the capability is absent, which is not what happened, because nothing in the pipeline removes a capability. Only the third would be a guarantee, and it is the one nobody has.