PIATRA . INSTITUTE

kernel smoothing

Attention is a reinvention of kernel smoothing

based on Nadaraya-Watson kernel regression

Gaussian kernel

Data Points & Prediction

Normalized weights α_i

by x_i

Attention as Kernel Smoothing

The attention mechanism in Transformers is mathematically equivalent to Nadaraya-Watson kernel regression. Both compute a weighted average where weights come from a similarity function.

Kernel Smoothing

$K(x_q, x_i)$ = similarity-to-weight mapping

$\alpha_i = \frac{K(x_q, x_i)}{\sum_j K(x_q, x_j)}$

$\hat{y}(x_q) = \sum_i \alpha_i y_i$

Attention (scaled dot-product)

$\text{score}_i = \frac{q \cdot k_i}{\sqrt{d_k}}$

$\alpha_i = \text{softmax}(\text{score})_i$

$\text{out} = \sum_i \alpha_i v_i$

Component Mapping

Transformer Attention	Kernel Regression
query $q$	evaluation point $x_q$
keys $\{k_i\}$	sample locations $\{x_i\}$
values $\{v_i\}$	sample responses $\{y_i\}$
score $q \cdot k_i / \sqrt{d_k}$	similarity / negative distance
softmax weights $\alpha_i$	normalized kernel weights
output $\sum_i \alpha_i v_i$	locally weighted average

Key Insights

Learned kernel: In Transformers, the projections $q = xW_Q$ , $k = xW_K$ , $v = xW_V$ are learned, so the model learns the similarity geometry.
Multiple heads: Multi-head attention uses multiple learned kernels in parallel.
Bandwidth vs temperature: The Gaussian bandwidth $h$ controls locality; the softmax temperature $\tau$ (or $\sqrt{d_k}$ ) controls how peaky the weights become.
Efficient attention: Because softmax attention is a kernel operation, methods like Performer approximate it with random feature maps for linear complexity.

What to Try

Set $h$ small (e.g., 0.2) and move $x_q$ : weights become very local.
Set $\tau$ small in Softmax-dot: weights become peaky (attention focuses on few keys).
Change $x_i$ positions to cluster points: observe how $\alpha$ redistributes.
Compare Gaussian vs Softmax-dot: note how softmax-dot prefers points with same sign as $x_q$ .