Upscaling: The Science of Bigger

Have you ever tried watching a VHS tape on your HD TV? How about a standard-def cable channel? If so, you've probably noticed just how awful video looks when it's not played back in its native resolution.

It's not a problem that will go away any time soon. Even the latest disc-based technology, Blu-ray, can only provide up to Full-HD resolution (1920 x 1080 pixels). That's only a quarter of the pixels the latest generation of 4K UHD televisions are capable of displaying. 4K TVs are likely to hit the mainstream before native 4K content is widely available, so what can you do to prevent your collection of full-HD content from looking like crap on your shiny new UHD set?

Well, you have two options. You can display the content at its native resolution (i.e., only using a quarter of the screen), or you can upscale the image from HD to UHD. No one in their right mind would choose the first option—it simply defeats the purpose of buying a big, high-res TV. Upscaling is the short-term future of UHD, so here's an in-depth look into the science behind it.

At its most basic, upscaling is the process of making up data—and its quite a bit of data, too.

Consider what it means to upscale from HD to 4K: This process turns an image containing about 2 million pixels to an image containing over 8.2 million pixels. That means over 75% of the upscaled image is non-original data—it doesn't directly reflect what the camera captured. That new data has to be produced somehow, and that process better be really good. Otherwise, the upscaled image will look terrible.

upscaled prcoess.png
Upscaling means turning 1 pixel into 4. Those extra 3 pixels start out empty, but need to be filled in somehow.

Luckily, image upscaling has been improving for decades. Almost every scientific field that involves imaging—everything from astronomy to medicine—benefits from increased resolution. As a result, quite a few interpolation methodologies have been developed to improve low-resolution images. Some of these have been adapted for use in televisions, blu-ray players, and even HDMI cables to upscale consumer video content.

Of course, some of these methodologies are better than others.

Nearest-Neighbor Filling

nearest neighbor upsclaed.png
Nearest Neighbor Filling is pretty simplistic and usually results in sub-par upscaled images.

Due to its simplicity, one of the first approaches to image upscaling was nearest-neighbor filling. It goes like this:

There are now four pixels instead of one in each part of the image, so you simply fill the three empty pixels with copies of the real pixel data. The resulting images preserve sharp edges extremely well, but also seem blocky and pixilated. If you spend a lot of time on 8- and 16-bit gaming, this sort of upscaling would be perfect, but it translates into very poor performance for higher-res content.

Interpolation

Interpolation, in this context, refers to making an intelligent guess at an unknown value, based on two or more known values.

In the nearest neighbor example, no intelligent guesses were made as to what the pixels might actually be; they were simply filled with the same values as their neighbors. Realistically, most nearest-neighbor guesses are completely wrong. Real-life images are rarely full of solid colors—even those that appear to contain solid colors really show subtle gradients.

To do a better job, you can calculate pixel levels for the three made-up pixels that are based on the pixel levels of multiple known pixels nearby. Below are the three most commonly used interpolation methods in imaging and displays.

Linear Interpolation

linear interpolation full figure.png
Linear interpolation fills the empty pixels by only considering the nearest vertical or horizontal pixels.

Linear interpolation is the simplest of the interpolations and has the benefit of being very processor-friendly. This method simply takes the pixels to the left and the right of the empty pixel (or above and below) and fills it with the average. As with most quick and dirty solutions, the results leave much to be desired. Edges and details become blurry and blocky, so this approach is seldom used in modern upscaling.

Bilinear Interpolation

bilinear interpolation full figure.png
Bilinear interpolation takes into account data from adjacent vertical and horizontal pixels.

The next step up from linear interpolation is bilinear interpolation. As the name suggests, bilinear interpolation takes into account two sets of real pixels when filling in upscaled pixels: the nearest vertical and the nearest horizontal.

The resulting fill color and brightness will be slightly more accurate due to the more larger set of real values used in the interpolation. Images produced with bilinear interpolation are slightly sharper than those using simple linear interpolation, but only slightly. Upscaling using this method will still result in very blurry and blocky images.

Bicubic Interpolation

986px-BicubicInterpolationExample.png
Bicubic interpolation takes into account a much larger portion of the screen to interpolate individual pixels. In this example, the known pixels live under the black dots, and the rest of the colors are interpolated.

Since more data is better, bicubic interpolation takes into account four times as many pixels (16 in total) as bilinear interpolation.

Essentially, you set up an array of 4x4 grids of known pixels, until all pixels in the image are covered. Then all empty pixels within each 4x4 grid are filled with color values based on the values of the known pixels within that square, using continuous curves.

This approach leads to some very smooth transitions between pixels and makes for a pretty impressive upscaled image, preserving much of the fine detail. It does, however, cause haloing at high-contrast edges, which is an often-undesirable artifact.

The Future of Upscaling

Interpolation works well and is relatively easy on your TV's CPU, but as more and more data gets routed to our displays, more horsepower is needed to keep up. Unfortunately, the amount of data required to further improve an image increases exponentially, which means that eventually even the most advanced processor won't be able to keep up with the demand.

If you view a TV as a content delivery device, then upscaling is a huge deal. After all, that pile of DVDs still has to be watchable on the latest 4K OLEDs, unless you want to buy those movies all over again. Upscaling performance is a distinguishing factor between televisions, and we'll soon be introducing an upscaling test that will let you know which TV will be the best nostalgia machine.

Photos: Berland (CC-BY-SA-3.0)

TAGS: tvs televisions television display quality upscaling

What's Your Take?

All Comments