New DigitalCameraInfo Color Test

Dear DigitalCameraInfo readers old and new: As of March 28, 2013, we are changing the way we test and score color performance. Our color performance metric is a combination of how accurately the camera captures color and how accurately those colors are rendered by the camera's internal JPEG processing. Color is a surprisingly complex aspect of camera performance, balancing true-to-life accuracy with what manufacturers think you want to see.

The changes to our color test are just a small step in our constant quest to provide readers with the most accurate and most relevant data possible. We will apply this new methodology to all reviews going forward and have also retroactively updated the color scores for all cameras tested since 2009. Whether you're buying one of your first cameras or want to compare new cameras to previous models, we want to make sure that you have the information you need to choose the perfect camera for you.

What Hasn't Changed

We will continue to shoot our color performance test using the classic X-Rite ColorChecker chart, evenly illuminated in bright lab conditions. As before, we shoot maximum resolution JPEGs in all of the camera's available color modes, using a custom white balance.

Our methodology for analyzing the photos also remains unchanged. We run all JPEGs through Imatest using the program's Colorcheck module. Imatest analyzes the 24 color patches on the X-Rite chart and compares the known color values to what was captured. These values are run through a complex series of equations called the CIEDE2000 color error formula, which calculates the ‘color error’ and saturation percentage of the image.

What's New: The Color Test

DigitalCameraInfo is making one small change to its testing methodology in order to achieve more accurate, consistent results. In the past, we have always made sure that the images we use for color testing fall within a certain tolerance for exposure. Our tolerance for white balance, however, was not as strict.

We have adapted our testing procedure to include a rigorous check for both exposure and white balance to ensure that color error data is as accurate as possible.

For the true photo enthusiasts and science nerds among our readers, there's one other change you'll want to know about: We are now using Imatest's corrected color error, which is a result of the same CIEDE2000 color difference formula, but is adjusted for ideal saturation. This allows us to apply our own over- or undersaturation penalty, which you can read about below.

What's New: The Color Score

The most important change in our color performance test lies not in the testing procedure itself, but in our newly developed scoring algorithm. As you may know, Reviewed.com believes in an infinite scoring system, which allows us to rate products in a constantly improving technological landscape.

If we review a camera in January that has the best color score we've ever seen, we would naturally give that camera a color score of '10.' But what happens when an even better camera comes along in July? Is that also a '10'? Do we go back and retroactively give the January camera a color score of '9.5'?

Our solution for the past several years was to avoid the concept of a maximum score of '10.' If the January camera scored a '10,' the superior July camera might get a '13.5.' However, we recognized that consumers were sometimes confused by a color score of '13.5.' Is that good? Average? This week, we're introducing a new approach.

The new scoring formula can be broken into two parts. In the first part, we take the corrected color error (see previous section) and apply it to the following algorithm:

The new algorithm emphasizes the most salient differences in color performance, while de-emphasizing the difference between cameras with virtually perfect (or terrible) color performance. Once we have the results of the color error formula, we take a look at the camera's saturation data. For cameras with particularly over- or undersaturated images, we apply a saturation penalty using a formula like this one:

The saturation penalty is applied as a percentage of the color error results. We run these two formulae for each of the tested color modes, giving the camera a final color score based on the color mode that turned in the best overall results.

What You'll See

So, what changes will you see on the site?

For starters, all color performance scores for all cameras have been adjusted using the new formulae. That means that all color scores now fall between 0 and 10. Comparing these new color scores, you should be able to tell easily which camera is better.

You'll also see that the ratings have shifted slightly to accommodate the new results. Cameras that were previously receiving an advantage from color score inflation are now on an even playing field with the others cameras in their class. Of course, we also have a new camera review to accompany the launch of our new color test. The excellent Sony NEX-6 also had a small, but expected impact on our ratings.

And, of course, this all goes hand-in-hand with our brand new website and review design!

Stay tuned for even more changes coming in 2013. The more-accurate-than-ever color score and the new site design are just the beginning of the improvements you'll see on DigitalCameraInfo this year, as we continue to make our site more accurate, more informative, and more enjoyable to visit.