Click to Photon

One painted pixel from the mouse to the retina, and the clocks it waits for on the way

Abstract

Between a hand moving a mouse and light leaving a subpixel, a stroke passes through some thirty representations: sensor counts, a USB report, a desktop coordinate, an image coordinate, four bytes, a shader output, a position in the raster scan, link symbols, a row and a column, a voltage or a current, and finally a spectrum. This companion times that chain. Each stage is either work, which costs a fixed time, or a wait for a clock that runs independently of the hand: the USB poll, the application's frame, the display's vertical blank, and in a window the compositor's refresh after it. Waits for independent clocks are uniform between zero and the clock's period, so latency is a constant plus a sum of uniform variables, and its exact distribution is a convolution. With no fitted constant, the model reproduces photodiode measurements of a windowed application by Casiez et al. (2017) to within 2% at 60 Hz and 9% at 120 Hz, and the 15.5 and 15.9 ms they saved by doubling the refresh rate to within 8%. It falls 5 ms short on Windows at 144 Hz, which locates something the model does not contain.

---

1. Work and waiting

1.1 The two kinds of stage

Every stage on the path either does something or waits for something.

Work takes a time set by the work itself:

• the mouse's sensor frame and firmware;
• the host controller and the kernel's HID driver;
• the application handling the event and drawing;
• a GPU frame;
• the monitor's receiver, scaler and timing controller;
• the pixel's transition.

Waiting is for a periodic clock that is not synchronised with the hand:

• USB polling. Interrupt transfers are polled by the host, so a report ready in the mouse waits for the next poll, every 1/f (Wimmer et al. 2019).
• A radio, if the mouse is wireless: a connection event every 11.25 ms is common for Bluetooth Low Energy HID.
• A frame loop, which reads input once per frame.
• Vsync: the display latches a new buffer only at the vertical blank.
• A compositor, which builds the desktop on one refresh to be shown on the next.

1.2 The distribution

A click arrives at a random phase of each clock, so it waits a time uniform on $[0, P_i]$. With fixed costs $c_j$:

$$L = \sum_j c_j + \sum_i U_i,\qquad U_i \sim \mathcal{U}(0, P_i),$$

$$\mathbb{E}[L] = \sum_j c_j + \sum_i \frac{P_i}{2},\qquad \operatorname{Var}[L] = \sum_i \frac{P_i^2}{12}.$$

The fixed costs move the distribution and do not widen it; all of the spread comes from the clocks. The exact density is the convolution of the uniform boxes, shifted by the fixed costs. The playground computes it on a 10 µs grid, in linear time per box from running sums, and checks it against a Monte Carlo sampler of the same stages. For every preset the two agree to within 0.1% in the mean and at the 95th percentile, over 200 000 sampled clicks.

1.3 The stages

---

2. The display side

2.1 A window costs two refreshes and a half

An application drawing in a composited window waits half a refresh, on average, for the vertical blank at which its buffer is latched. The compositor composes the desktop from it on that refresh, and the result is scanned out on the next. The scan then reaches the row some fraction of a refresh later. At 60 Hz, for a row in the middle of the screen, that is

$$\tfrac12 \cdot 16.7 + 16.7 + \left(0.46 + \tfrac12 (16.7 - 0.46)\right) = 33.6 \text{ ms}$$

before the monitor has processed anything or the pixel has started to change. The 0.46 ms is the minimum vertical blanking time of VESA's reduced-blanking timings.

2.2 Against a photodiode

Casiez, Pietrzak, Marchal, Poulmane, Falce and Roussel (2017) measured end-to-end latency with a vibration sensor under the mouse button and a photodiode on the screen, with software probes at the HID report, the system event, the toolkit event and the repaint.

The setup. A Logitech G9 mouse polled at 1000 Hz, a C++/GLUT application in a window on macOS, and a Dell S2716DG monitor.

The model's inputs. The mouse was set from their probes: 0.6 ms to the HID report, 1.4 ms more to the repaint. The monitor's G-SYNC module is taken to add no processing, and its TN panel's transition is taken as 1 ms.

The difference between refresh rates is the sharpest test, because every fixed cost cancels in it. What remains is the number of refresh periods a click spends on the display side: two, in the model, for a composited window and a mid-screen row. The measurements imply 1.86 (GLUT) and 1.91 (Qt).

On Windows the model is 5 ms short at 144 Hz, about three-quarters of a refresh. That compositor adds more than the model's one refresh, or more fixed time. The model does not hide this: it is a calibration case.

2.3 How the refresh rate pays off

Every wait on the display side is a fraction of a refresh, so latency falls as one over the refresh rate. For the office configuration in the playground (a 125 Hz mouse, a composited window, 2 ms of monitor processing):

The first doubling saves 16.7 ms and the last saves 4.2 ms; from 240 Hz on, the fixed costs are most of what is left. In the same configuration, the present mode changes the mean as follows:

• leaving the compositor (fullscreen) saves 16.7 ms, one refresh at 60 Hz;
• variable refresh, which flips as soon as the frame is ready, brings it to 20.1 ms;
• turning vsync off brings it to 19.9 ms.

2.4 Where on the screen

With vsync, the display draws top to bottom: a row is reached at blank + row × (period − blank) after the flip. In the office configuration, a stroke at the top of the screen reaches the light at 37.0 ms on average, and one at the bottom at 53.2 ms. The playground reads the row from where the stroke is on the physical screen.

With vsync off, the buffer is flipped mid-scan and the row updates on the scan's next pass. That is a uniform wait of up to one refresh, whatever the row, and the frame shows a tear where the two images meet.

2.5 The strobe

A strobed backlight stays dark while the panel is scanned and the crystal turns, then flashes once; the transition is hidden and motion is sharp. The light waits for the whole panel to be scanned and settle, so the row stops mattering and the full scan is always paid. In the strobe preset at 144 Hz:

---

3. The mouse

3.1 Polling

A USB mouse does not send its report; the host polls for it. A report that is ready waits for the next poll, uniformly within one polling interval. That interval is in the device's descriptor, and in practice it is rounded down to a power of two (Wimmer et al. 2019).

At 125 Hz the wait has a mean of 4 ms and a standard deviation of $8/\sqrt{12} = 2.31$ ms. Casiez et al. measured the time from the button press to the HID report of a Logitech G9 at each polling rate:

At 500 and 1000 Hz the measured spread stops falling near the method's stated precision of about a millisecond, so only the first two rates are calibration cases.

The mean. Going from 125 to 1000 Hz saved 3.4 ms end to end, against the 3.5 ms difference of the mean waits.

3.2 How little it matters on a desktop

For the office configuration, the polling rate moves the mean from 45.1 ms at 125 Hz to 41.6 ms at 1000 Hz and 41.2 ms at 8000 Hz. In the sensitivity analysis, one step of polling rate is worth 2 ms. Multiplying or dividing the refresh rate by 1.5 is worth 27.8 ms, and moving the stroke from the top of the screen to the bottom is worth 16.2 ms.

---

4. Frames in flight

4.1 The frame loop

A game reads input once per frame, so the click waits up to a frame interval to be read, then for the CPU and GPU to render. With vsync, a frame that starts on a vertical blank is shown on the first blank at least one render time later. If the render takes longer than a refresh, the frame interval is quantised to whole refreshes.

4.2 The queue

If the CPU can prepare frames faster than they are consumed, by the GPU or by the display, the queue between them fills to its limit. Every input then waits behind the frames already in it.

The GPU-bound preset needs 12 ms of GPU time per frame at 144 Hz, so it shows a frame every 13.9 ms: two refreshes, 72 frames a second. With three frames allowed in flight:

• the mean latency is 63.0 ms;
• 27.8 ms of it is two queued frames;
• limiting the queue to one frame brings the mean to 35.2 ms without changing the frame rate, because the CPU now waits for the GPU instead of running ahead of it.

Driver settings for maximum pre-rendered frames, and low-latency modes, exist for this reason.

Swept over GPU time with the same queue of three, latency rises in steps as the frame interval crosses each multiple of the refresh period. It is 38.7 ms up to 6.9 ms of GPU time, 56.1 to 63.0 ms up to 13.9 ms, and 80.4 ms beyond. This is the threshold the playground's classification refers to.

4.3 The extremes

The television's game mode, modelled as cutting processing from 50 to 5 ms, takes the last case to 78.4 ms.

---

5. What people notice

Perception thresholds for latency depend on the task and the device:

• Dragging under a finger on a touchscreen: latencies down to about 6 ms are noticed (Ng et al. 2012).
• Tapping: about 24 ms (Jota et al. 2013).
• Indirect touch: Deber et al. (2015) found just-noticeable differences of 55 ms for dragging and 96 ms for tapping.
• A mouse: Forch et al. (2017) found the median observer notices about 60 ms in total. Observers with more experience of fast games notice less.
• Performance: MacKenzie and Ware (1993) measured pointing with lags from 8.3 to 225 ms, and found that the cost of lag grows with the difficulty of the target: a lag-times-difficulty term in Fitts' law.

These thresholds are drawn on the playground's distribution, labelled with their task and device. They are not interchangeable, and none of them is a property of the display alone.

---

6. What a page can see

A web page cannot see the mouse, the cable, the monitor or the light. It does see three clocks:

• Animation frames, which fire once per refresh, so their median interval gives the refresh rate.
• Input event timestamps, taken when the browser receives the event from the operating system. The time from a timestamp to the start of the next animation frame is the model's frame wait for the browser, uniform on one refresh.
• The Event Timing API, where it exists. For discrete events whose processing and next paint take more than 16 ms, it reports the time from the event to the next paint, rounded to 8 ms (W3C Event Timing).

The playground's probe records all three while the canvas is drawn on. It plots the frame wait in units of the measured refresh period against a uniform distribution, and gives the Kolmogorov-Smirnov distance with its 95% band. A page that is busy with other work, or a browser that coalesces input differently, will show it there.

---

7. What this does not do

• Compositors differ. One refresh fits macOS in the 2017 measurements and does not fit Windows 10.
• Pixel transitions are exponential. Here they are; in real panels they are shaped by overdrive and differ between grey levels. The photon is counted at 50%.
• Wireless intervals are typical, not measured for any device.
• Phases are independent. A game that samples input in step with vsync, or a mouse whose polling has locked to the frame rate, would correlate waits the model treats as independent.
• The layer stack mixes measured and illustrative values. The screen, viewport and canvas coordinates and the colour written are measured. Addresses, voltages and packet contents are illustrative, and marked so in each layer.
• Neural latency is outside the chain. The vision layers come after the photon, and no time is attached to them.

---

References

• Casiez, G., Pietrzak, T., Marchal, D., Poulmane, S., Falce, M. and Roussel, N. (2017). Characterizing latency in touch and button-equipped interactive systems. Proceedings of the 30th ACM Symposium on User Interface Software and Technology, 29. https://doi.org/10.1145/3126594.3126606
• Deber, J., Jota, R., Forlines, C. and Wigdor, D. (2015). How much faster is fast enough? User perception of latency and latency improvements in direct and indirect touch. Proceedings of CHI 2015, 1827. https://doi.org/10.1145/2702123.2702300
• Forch, V., Franke, T., Rauh, N. and Krems, J. F. (2017). Are 100 ms fast enough? Characterizing latency perception thresholds in mouse-based interaction. Engineering Psychology and Cognitive Ergonomics, Lecture Notes in Computer Science, 45. https://doi.org/10.1007/978-3-319-58475-1_4
• Jota, R., Ng, A., Dietz, P. and Wigdor, D. (2013). How fast is fast enough? A study of the effects of latency in direct-touch pointing tasks. Proceedings of CHI 2013, 2291. https://doi.org/10.1145/2470654.2481317
• MacKenzie, I. S. and Ware, C. (1993). Lag as a determinant of human performance in interactive systems. Proceedings of CHI 1993, 488. https://doi.org/10.1145/169059.169431
• Ng, A., Lepinski, J., Wigdor, D., Sanders, S. and Dietz, P. (2012). Designing for low-latency direct-touch input. Proceedings of the 25th ACM Symposium on User Interface Software and Technology, 453. https://doi.org/10.1145/2380116.2380174
• Wimmer, R., Schmid, A. and Bockes, F. (2019). On the latency of USB-connected input devices. Proceedings of CHI 2019, 1. https://doi.org/10.1145/3290605.3300650
• VESA. Coordinated Video Timings Standard, reduced blanking timing.
• W3C. Event Timing API. https://www.w3.org/TR/event-timing/
• USB Implementers Forum. Device Class Definition for Human Interface Devices (HID).