A new semi-regular series
Having just finished a book on the history of computer graphics I’ve decided to be Janus and also look at the future, which seems appropriate because that’s where we’re all going to end up.
Over the next few months I’m going to post essays and thoughts on what I see as the developing trends, and some far out ideas (like a totally voxel-based game and rendering engine). I’d also like to hear from you. What do you think is an interesting, or scary trend, and what would you like to see? Of course, I don’t have to ask for your criticisms; I know you’ll share them with me.
The move to mobile
I think we’ve all been fascinated with the way mobile devices have taken off in popularity. No small portion of that surge in interest is the display—the place where the user encounters the device. I think it’s safe and fair to say Apple gets credit for lighting up our interest and imagination in mobile personal devices with big displays. Not so much to fawn over Apple, but rather to pick a point in time when the revolution started June 29, 2007. That was when all the things that were being experimented with in various products from various manufactures came together at once: touch, high-resolution display (for the times), large display, intuitive GPU accelerated UI, and dozens of applications that worked together smoothly and efficiently.
The impact of the API
The most important things in any system are the interfaces. How does a signal, or a word, or pulse get from one component to another? When data is moved, an API is usually involved. At Siggraph this year, I will give a brief talk at the Khronos BOFS Party (24 July, Hilton California Ballroom, around 19:00 – the party starts at 17:00 or earlier) on the history of APIs, and discuss why they are so important, contentious, and difficult.
The first computer graphics (CG) API was for the IBM 2250 graphics terminal in 1965. Forty-seven years later we got the first really useful, and (so far) stable API for mobile devices—Open GL ES 3.0.
OpenGL ES 3.0 is now available to developers in Android Jellybean (MR2) and that game engine companies are also in the process of enabling developers to benefit from the new OpenGL ES 3.0 API features.
The Khronos organization develops and manages API used in CG applications. Open GL ES 1.1, released August 2004 from Khronos, was first used in the Samsung SE P850 in 2005, and the first iPhone with PowerVR MBX GPU supported ES 1.1 two years later. So Open GL ES really was the enabler of the revolution to smartphones.
Full-precision
OpenGL ES 2.0, announced in March 2007, introduced the OpenGL Shading Language for programming vertex and fragment shaders. It removed any fixed functionality that a shader program could replace. That minimized the cost and power consumption of advanced programmable graphics subsystems.
Open GL (OGL) ES 3.0, introduced about a year ago at Siggraph in August, was a major breakthrough in bringing unified shaders, which enabled writing programs for shader effects.
Most importantly, OGL ES 3.0 also added support for 32-bit integer and 32-bit floating-point (i.e. full precision) data types to the pixel shaders (AKA fragment shaders). This was significant because as shader complexity increases, as it has been, the potential for errors (without full precision) becomes even larger. In addition, bringing unified, bringing full-precision shader capability to a mobile device gives that device the potential to run computer graphics programs that are of workstation quality and class. This a major tipping point.
Also worth mentioning is the Real int32 support, which is new in OGL ES 3.0 and applies to both Fragment and Vertex Shader.
Occlusion and stamping
OGL ES 3.0 doesn’t have specific geometry shaders; however, it does have several features that help with geometry. Two of the most useful features are occlusion queries and geometry instancing. Occlusion queries allows for fast hardware testing of whether an object’s pixels are blocking (occluding) another object. That’s essential in determining if something doesn’t have to be rendered (because it’s occluded) and thereby saving processing time which in turn gives higher performance with less battery usage.
Geometry instancing (AKA “stamping”) enables the GPU to draw the same object multiple times but only requires the object to be sent to the rendering pipeline once.
This makes objects such as birds, foliage, grass, surface rocks, or phony smoke easier for the CPU to set up because the objet doesn’t have to be sent repeatedly. That in turn also saves battery.
Textures
Ever since Jim Blinn invented texture mapping in 1977, it has been used to create amazingly realistic scenes. However, textures can be big and there can be multiple (smaller) copies of them for mip-mapping. Therefore, it is highly desirable to be able to compress the textures, which is relatively easy to do because there is usually a lot of redundancy in a texture image. However, texture compression has been a contentious subject with claims and counter claims about IP and patents.
The OGL committee, composed of CG experts from all over the world and all the leading CG companies, spent years developing a royalty free texture compression algorithm. In early 2012, Ericsson offered their ETC family of texture compression algorithms on a royalty free basis and allowed Khronos to implement it as a standard texture compression format. It was gratefully accepted and deployed for the first time in OGL ES 3.0—another break through.
ETC was introduced in OpenGL ES 2.0, but did not have Alpha support hence was consequently not adopted by developers. Developers instead had to support hardware specific texture compression formats like ATC, PVRTC, DXTC (This was the biggest pain for developers on Android today). OpenGL ES 3.0 brings in newer and better, royalty free, ETC2 texture compression format (with alpha support) all hardware suppliers (IHVs) are required to support it. This makes it simple for developers, as going forward they do not have to compress their textures in separately for each device.
In addition to texture compression, 3.0 added support for multiple render targets enabling the GPU to render to multiple textures at once. This is significant in that it enables real-time deferred rendering, which saves enormous GPU cycles and therefore more battery power.
Textures, as mentioned, can get large, and therefore more accuracy is needed in rendering them. ES 3.0 includes support for floating-point textures as well as 3D textures, depth textures, non-power-of-two textures, and overlay (alpha plane) 1 & 2 channel textures (R & R/G).
However, the GPU needs to support an extension to access this functionality. Several GPUs such as Quallcomm’s Adreno, and Imagination Technologies support this, but it is not part of of the OGL ES 3.0 requirement.
Shadows and jaggies
Shadow mapping or projective shadowing where shadows are added to an image is a concept introduced by Lance Williams in 1978, and it’s been used extensively in CG ever since. However, a shadow map, like any other object, has edges, and those edges if looked at closely can have jaggies. An edge filtering technique (like AA is some sense) can be used, and in OGL ES 3.0 there a technique known as “percentage closer filtering” (PCF) available.
PCF works by filtering the result of the depth comparison. So when comparing a depth, some depths around should also be compared and the result should be averaged. This will give a softer look on the shadow edges.
PCF is a newer technique for making softer shadows when using shadow mapping. PCF takes advantage of a dedicated hardware block that does texture filtering after the depth test. The standard texture filtering does filtering as soon as the texel is fetched and it does not wait for the depth test.
It is an almost ideal solution for shadows because its fast and just accurate enough that it doesn’t need a lot of processing.
OGL ES 3.0 also has quality anti-aliasing features like multi-sample antialiasing (MSAA).
Open GL ES 3.0 has incorporated many features from its big brother the workstation version of Open GL. In many respects, OGL ES has advanced features not yet found in Microsoft’s Direct X API. That is an extraordinary situation, and creates an environment where mobile graphics can actually exceed PC graphics—unthinkable a few years ago.
The ecosystem has also been developed. Benchmark developers like, Kishonti, and Rightware have already announced their benchmarks based on OpenGL ES 3.0, and Futuremark is working on one. With game engine developer Unity and Google’s official support for OpenGL ES 3.0 on its way, there will be plenty of games and applications in Google Playstore this year.
Because the API is so buried in the system, and perhaps so esoteric, consumers will never be aware of its features or the power it unleashes in todays (and tomorrows) SoCs. However, it should be used in the marketing of mobile device and some educational work done to sensitize the users. I think mobile devices should have an “ES 3.0 enabled” logo or at least a line in the specs.
Open GL ES 3.0 now makes AAA games found on a PC possible on a mobile device—I didn’t think I’d be saying something like that for a few more years.