Rotary Fields: Position as Angle in Attention and in the Brain

Abstract

Rotary position embedding (RoPE) encodes a token's position by rotating its query and key vectors, so that the attention score between two tokens depends on their relative offset rather than on their absolute indices. Hippocampal place cells encode an animal's position within a place field by the phase at which they fire relative to the ongoing theta rhythm, so that relative position becomes a phase difference. These two systems have nothing in common at the level of mechanism, one is a matrix multiplication inside a transformer and the other is a biophysical spiking process, yet they make the same geometric move: they represent a displacement as an angle. This playground implements the RoPE rotation exactly, computes a small toy attention matrix from it, and places the construction next to the neural phase code so the shared geometry is visible. This companion separates what the model computes exactly from what it asserts only by analogy, and it states the limits honestly.

What the playground computes

The core object is the RoPE construction from Su et al. (2021). A query or key vector of dimension d = 2 * pairs is split into pairs two-dimensional blocks. The block at index m is rotated by an angle theta = p * omega_m, where p is the token position and omega_m = 1 / base^(2m / d) is the per-pair frequency. The first pairs rotate fast, the last pairs rotate slowly, which is the frequency ladder. After rotating the query at position i and the key at position j, the playground takes their dot product. Because a planar rotation by i followed by the transpose of a rotation by j equals a single rotation by j - i, the rotated dot product is mathematically a function of the offset j - i. This relative-only property is exact for shared content vectors and is the reason RoPE gives relative-position attention without ever storing a relative-position table.

From the full N by N rotated score matrix the playground derives a handful of metrics. Peak and mean scores summarise the overall sharpness. Nearby mass is the mean score for offsets of one to three tokens; distant mass is the mean score for offsets of eight tokens or more. The headline calibrated metric is concentration, defined as (peak - distant) / 2, clamped to the interval from zero to one. Because unit query and key vectors give dot products in the range from minus one to one, the quantity peak - distant lies between zero and two, so dividing by two maps concentration into a clean zero-to-one scale. Context width is the offset at which a query row's score first falls below half its peak. Translation drift is the mean absolute difference between score(i, j) and score(i + 1, j + 1) across the matrix, a direct measurement of how far the model departs from perfect translation invariance.

What is exact and what is interpretive

The rotation arithmetic is exact. The frequency ladder, the per-block rotation, and the algebraic fact that the rotated dot product collapses to a function of j - i are faithful to the original RoPE paper and to standard implementations. Anyone re-deriving them gets the same formulas.

The attention matrix is a deliberate simplification. There is no softmax, no value projection, no multi-head averaging, and no training. The score shown is the raw rotated dot product, not a normalised attention weight. Showing the raw score is a teaching choice, because softmax would compress the values and hide the smooth position dependence that is the whole point. The content vectors are random rather than meaningful. Each position is assigned a unit-norm random query and a key that shares a tunable similarity with it, plus a small per-position jitter so that changing the seed produces visibly different patterns. Real transformer content comes from token embeddings and the preceding layers; this playground models only the rotation step and lets random content stand in for the rest.

One consequence of using position-specific random content is that translation drift is not exactly zero, even though true RoPE on shared content vectors would make it zero. The nonzero drift the playground reports is therefore a direct readout of how much residual structure is tied to absolute position by the random content, rather than a flaw in the rotation. This is a useful illustration in its own right: it shows that the relative-only guarantee belongs to the rotation, not to the content.

The five presets sample the design space. A LLaMA-style baseline uses base ten thousand and four pairs over a sixteen-token window. A long-range head raises the base and the pair count so the attention mass spreads across the whole window. A short-context head lowers the base and the pair count so fast rotations destroy far-token alignment and sharpen attention onto the diagonal. An extrapolation preset makes the sequence longer than the longest rotation period, the regime where naive RoPE wraps around and fails to extrapolate. A neural-phase preset pushes the phase slope to three hundred sixty degrees across a field, matching the canonical place-cell measurement. The calibration table checks whether the toy attention concentration lands in the regime a reader expects for each preset. The expected concentration values are reader-assigned targets, not measurements from a trained model, so they calibrate the qualitative ordering rather than any absolute number.

The neural cousin

The neural side of the playground does not claim to implement RoPE. It implements the closest biological analogue to position-as-rotation. As an animal crosses a place field, a place cell fires at progressively earlier phases of the theta rhythm, advancing by about three hundred sixty degrees across the field. This is theta phase precession, reported by O'Keefe and Recce (1993) and replicated widely since. The playground renders the rate-by-phase curve and a stochastic spike scatter for a given phase slope and noise level, so that position within the field reads off as firing phase, and relative position reads off as phase difference. This is structurally the same move as the RoPE plane, where relative position reads off as relative rotation angle.

A second neural element is a grid-like interference field, built by summing three planar oscillations at sixty-degree offsets to produce a hexagonal pattern. This follows the oscillatory-interference model of grid cells associated with Burgess and colleagues (2007). It is included because it makes the third face of the same idea visible, that a small number of phases composed at different scales can carry a large amount of spatial structure, which is exactly what the RoPE frequency ladder does by design. The model flags this part as the most contested: recent evidence favours continuous-attractor accounts of grid cells over pure interference, so the interference field is best read as a clean illustration of the phase-composition principle rather than as a settled mechanism.

Where the analogy is strong and where it is only an analogy

The strong claim is geometric and well supported. Both RoPE and theta phase precession encode an absolute position as an angle, and both let a relative position be recovered from a difference of angles. That correspondence is precise and is the reason the two phenomena are worth placing side by side.

The weak claim, which the playground does not make, would be that neurons compute rotation matrices on stored vectors. They do not. The phase code arises from membrane dynamics, oscillatory inputs, and network interactions, not from a linear-algebra operation on a query and a key. The honest formulation is that the two systems are analogous at the level of representational geometry and unrelated at the level of implementation. The playground is a sketch that makes the geometric move legible, not a simulator of either a transformer or a hippocampus.

Limitations

The model captures one step of attention and nothing else. With no softmax, no value vectors, no multiple heads, and no learned content, it cannot say anything quantitative about a real transformer's behaviour. The concentration metric is one reasonable summary of locality versus dispersion, but alternatives such as row entropy or kurtosis could rank the presets differently; only the qualitative ordering is expected to be robust. The neural panels are illustrative renderings of established phenomenology rather than fits to recorded data, and the grid-interference mechanism in particular is contested. The calibration targets are reader-assigned regimes, so the calibration checks self-consistency of the toy, not agreement with measured attention or measured firing. Within those bounds the playground does its one job, which is to make the shared idea, position as angle, tangible across two very different substrates.

References

J. Su, Y. Lu, S. Pan, B. Wen, Y. Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", 2021, arXiv:2104.09864.
J. O'Keefe and M. L. Recce, "Phase relationship between hippocampal place units and the EEG theta rhythm", Hippocampus, 1993.
N. Burgess, C. Barry, J. O'Keefe, "An oscillatory interference model of grid cell firing", Hippocampus, 2007.
S. Chen, S. Wong, L. Chen, Y. Tian, "Extending Context Window of Large Language Models via Positional Interpolation", 2023, arXiv:2306.15595.
O. Press, N. A. Smith, M. Lewis, "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ALiBi), 2022, arXiv:2108.12409.
E. Voita, D. Talbot, F. Moiseev, R. Sennrich, I. Titov, "Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting", 2019, arXiv:1905.09418.