Lenses Have a Ceiling: Why Native Reference Frames Matter

Given “…A review of Boulter’s performance in The” from a WikiText article, OLMo3-7B predicts “ Guardian” (probability 0.29), ahead of “ Times” (probability 0.22). The table lists the top token that each of three readings finds in the hidden state at this position, after every fourth layer. Model layers are numbered from 1, and layer 0 is the embedding:

Layer (of 32)	Logit lens	Jacobian lens	LexFrame
0	“oretical”	–	“ New”
4	“oretical”	“ flavours”	“ Bill”
8	“ nto”	“urga”	“ Bill”
12	“ comparison”	“...\n\n”	“ Times”
16	“orem”	“ Illustrated”	“ Guardian”
20	“odor”	“ Guardian”	“ Guardian”
24	“ Guardian”	“ Guardian”	“ Guardian”
28	“ Guardian”	“ Guardian”	“ Guardian”
32	“ Guardian”	“ Guardian”	“ Guardian”

Through both lenses, the middle layers look like noise. The last column names the runner-up “ Times” at layer 12 and “ Guardian” from layer 16 on. Either the middle layers hold noise, or the model is already choosing between two newspapers. The answer matters beyond interpretability, since early exit, attribution, steering, editing and distillation all act on readings of the middle layers. We argue that lenses fall short because they read states in a frame with a ceiling.

Every hidden state has a native frame

The model’s vocabulary labels, its tokens, are the smallest units of meaning that it receives and produces. At the output, the model expresses a state exactly in these units. The coefficient of token v is its logit, the inner product of v’s unembedding row with the normalised final state, and the lexical basis of v is the state whose logits come closest to “only v”. The lexical basis, weighted by the coefficients and summed, rebuilds the normalised state, so decomposition and composition are both exact and together make up the output frame. At the other end, the embedding fixes the input frame.

Between the two fixed frames the model defines none, so interpretability borrows one. The logit lens, from nostalgebraist’s “interpreting GPT: the logit lens”, borrows the output frame, and the Jacobian lens fits one translation per layer, the same for every input. Such a translation reads every state of a layer identically, regardless of its prefix, and fails silently wherever the model does not. Yet a state at layer 16 is read only by layers 17 to 32, in its context. A hidden state means what these layers make of it. That reading is the state’s native reference frame, and it is the piece that mechanistic interpretability has been missing.¹ It is the frame in which the model itself expresses the state in inference.

The native frame is defined through native transport, which carries the state and its prefix through the remaining layers. The coefficient of a token is the output’s inner product taken after the transport, and its lexical basis is the smallest change to the state that raises that token at the output and little else. Each token also has a readout, the gradient of its coefficient at this state. Like “up”, which points in nearly opposite directions for a Londoner and a Sydneysider, the readout depends on where the state stands. Only the input and output frames are fixed, and between them the native frame shifts layer by layer. Since the transport attends to the prefix, the native frame is also context-aware by construction.

Native frame with five lexical bases (coloured) and one state (black). It is fixed at the input and output and shifts layer by layer in between.

Native frames break the lens ceiling

A lens offers every context the same direction, as if everyone on Earth were told to point the same way. The logit lens offers London’s up, exact at the output and less accurate further away. A fitted lens offers at best the average up. Let κ be the mean cosine between the unit readouts of one token at states of a layer from different contexts. No fixed direction, however it is fitted, reaches a mean cosine above √κ with these readouts. At mid-depth, √κ is 0.35 in OLMo3-7B and 0.30 in SmolLM3-3B. Every fixed reading that we tested stays below this ceiling, including the Jacobian lens.

The native frame has no such ceiling, but computed exactly it requires the whole model. LexFrame approximates it by reading a state through the model’s own remaining layers with one round of attention between positions and replacing the rest of the attention by a stand-in fitted once on unlabelled text, never on the model’s output. LexFrame, the last column of the opening table, exceeds the ceiling at every layer and reaches a cosine of 0.97 with the model’s readouts at mid-depth.

Cosine with the model’s readout in OLMo3-7B (left) and SmolLM3-3B (right). Every fixed method stays below the √κ ceiling (dotted), whereas LexFrame rises above it.

A matching direction alone does not make a frame. A frame for a middle layer must also keep the two properties of the output frame: moving along the lexical basis of a token must compose that token, and inner products must decompose the state into the model’s logits. For composition, we move a state by a tenth of its norm along the lexical basis of a token and check whether the logit of that token then gains most. This holds in 98–100% of cases for the native frame and in 84–97% for LexFrame, and less often for the input or output frame. For decomposition, we take the inner products after the transport. The coefficients of LexFrame reproduce the model’s output distribution within 0.15–0.16 nats, whereas those taken on the state itself or through a lens deviate by more than 5 nats.

Every direction read through a lens inherits the √κ ceiling. The native frame is not a ground truth, but it is a reference against which any interpretation of a state can be scored, and LexFrame shows where this matters. Lenses suggest that predictions form near the output, but LexFrame retains the model’s top token over the last 81–82% of the layers, the lenses over at most 11%. An exit at a probability threshold thus skips 42–45% of the layers with LexFrame but almost none with a lens. LexFrame saves no computation, but it shows how much depth a lens-based exit forgoes. When a component is removed, the change in the coefficient of the predicted token in LexFrame correlates with the measured effect at 0.88–0.90, against 0.62–0.68 for attribution patching.

Convergence follows least dissipation

Mean angle to the model’s readout (grey) after every fourth layer of OLMo3-7B. LexFrame closes the angle steadily, whereas the lenses stay far off until the output.

In OLMo3-7B, the mean angle of LexFrame shrinks by 10.5% of what remains at each layer before the output (exponential fit, R² = 0.99). A constant share per step is exponential relaxation, the way an overdamped system settles towards equilibrium, and the principle of least dissipation selects this path. The model thus refines a state gradually across its layers, rather than in a final jump as the lenses suggest.

In the opening example, LexFrame names the runner-up at layer 12 and the answer from layer 16 on, while both lenses still show unrelated tokens. The middle layers did not hold noise, and the model was already weighing the two newspapers. Interpretability does not need a better lens. It needs the frame that the model already uses.