Measuring latency

Posted by Jon Peddie on February 7th 2017 | Discuss
Categories: Hardware Review,

Basemark has a new tool to tell how sick you’re going to get

Latency is one of the biggest challenges in the burgeoning Virtual Reality (VR) industry. While wearing a VR Head Mounted Display (HMD), people notice even the smallest of delays between what’s displayed at the screen and what the brain expects.

Motion-to-photon latency is exactly what it says. It measures time from when the user physically makes a motion (say, begins to turn their head) to when the photons physically coming out of the display change to reflect that motion. A subset of motion-to-photon latency is Application-to-photon latency. Basemark has introduced the VRTrek a solution for application-to-photon latency measurements an aptly named device that can physically measure latency and be quantified as the VRScore. Basemark says they will support motion-to-photon measurements in the near future.

In order to run the tests correctly and come to conclusions there has to be in agreement on the definition of latency. The complication leading to different definitions occurs because many VR headsets have rolling displays, where the physical pixels on the display light up progressively, one row at a time, as the image is scanned out from the host PC. So, when you define latency, you have to say which photons you're talking about. The very first ones to scan out? The very last ones? Somewhere in the middle?

On the other hand, if a headset has a global display instead of a rolling one, then the pixels are kept dark while being updated and then the whole display is flashed on and off at the end of scanout. In this case, it's clear that the appropriate reference point to measure is the beginning of the flash, since all the photons come out at the same time.

Until now, there has not been a professional latency measurement tool that works across different HMDs and applications. Now there is, and we tested it.

For our testing, we used an Oculus Rift, driven by an EVGA GTX1080 in an Alienware Aura gaming PC, and Basemark’s VRScore benchmark.

The VRScore is more than a traditional graphics benchmark because it works together with Basemark’s VRTrek latency measurement device, and enables the VR industry to analyze and provide a better user experience for end users. VRScore is aimed at PC and VR HMD vendors and software developers to measure the performance of the PC and the latency of an HMD.

The tool’s main measurement output is a composite score from the PC’s raw performance and the ability to keep up a steady framerate. The VRTrek section outputs the latency results, or the time in milliseconds it takes for the required image to be displayed on the HMD. VRTrek scans the left and right eye screens at the same time. Also, VRTrek outputs granular data such as dropped frames, duplicate frames and screen persistence.

VRScore measures the experience in framerate and latency a user will get when using a given VR headset and associated PC hardware. HMDs add another layer of complexity to the measurements as each HMD has its own SDK and the associated overhead. Thus it’s not only the PC and its drivers that are the variable in the equation, but it’s also the delay in the HMD itself due to version of the software (SDK) being used which handle all the distortion correction and other projection calculations.

To record latency Basemark provides its VRTrek, a measurement apparatus which consists of two sensors in a plastic oblong plate that can slide up and down on a plastic stand so that the sensors are able to get as coaxial as possible with the HMD’s fisheye lenses. The overall shape is decidedly StarTrekian. 

The cable from the sensor plate plugs into the Mic input of the PC. The HMD’s optical (person) sensor has to be covered to trick the HMD into thinking its up against someone’s forehead (see blue tape in photo). 
Then the software scans both left and right eyes to generates data on an app and headset performance, providing information about the latency from the point of application frame submit to the point when the frame appears on given display. 

Assuming Basemark has a fast loading app this score could then be the best your system will get. That could then be a benchmark goal for app developers.

Basemark says they worked in closely with key industry players, including AMD, Intel and Nvidia. The test uses Crytek’s Cryengine game engine, and a game-like workload developed by Crytek called Codename: Skyharbor.
VRTrek is a patent pending solution and Basemark says it’s is compatible with Rift, Vive, OSVR and other PC-based VR HMDs. Basemark is licensing the VRTrek library to OEMs and labs. The license includes 1x VRTrek Device, and a SDK including Examples, API Documentation and User Guide. Price varies by customer size and usage model. Basemark also offers custom (NRE) services to OEMs if needed or desired.

What do we think
Basemark’s system now measures Application-to-photon latency. Later this year the company will enable motion-to-photon measurements as well. To do that they will have to build a motorized HMD holder that can trigger the accelerometers. Such devices have been built in university test fixtures. 

Essentially testing latency is testing how good a VR system is. If you get sick, you’re not going to like it. Also, longer forms of content won’t be possible if people only have a limited time they can stand in the headset. We’re pretty excited to finally receive a tool for measuring latency. This should enable the industry to set standards and achieve ever better functioning VR systems.—J.P.

Discuss this entry