Mitsubishi Electric Develops Camera that Refocuses Photos

April 18, 2007 – Every photographer has experienced the letdown of an out-of-focus

picture that is otherwise perfect. Blame it on the photographer or blame it on the auto focus, but call the blurry photo lost for good – until now. Scientists at Mitsubishi Electric Research Labs (MERL) last week announced the heterodyne light field camera that makes refocusing after capture possible. Using coded aperture that can increase depth of field by 10 times, the camera essentially deblurs an image after it is captured, according to the researchers.

"[Out-of-focus pictures are] the Holy Grail problem," said MERL Senior Researcher Scientist Ramesh Raskar in an interview with DigitalCameraInfo.com. "But we can change the original image which is extremely beneficial."

**Changing 2D into 4D with a Crossword Puzzle:

How Optical Heterodyning Works**

After two years of development, the MERL project was accepted last week into the Siggraph Conference, which is the international computer graphics convention that will be held this August in San Diego. MERL has already caught the attention of the imaging tech industry. At Siggraph 2006, MERL scientists addressed the problem of motion blur (blur caused by moving subjects during exposure) with what they call the "flutter shutter camera." Now, the researchers have tackled blurry images caused by the lens not being properly focused.

The MERL heterodyne light field camera is a modified standard camera that has a mask inserted in the optical path between the lens and the sensor. The mask is a transparency slide with a strategic pattern printed on it. The prototype mask was made for a 2.8 aperture. The pattern on the prototype mask is made up of seven rows and seven columns that resemble a "crossword puzzle," according to Raskar. Some boxes are blacked out while other blocks are transparent to let light pass through the masked or coded aperture.

Users can take photos as normal when the mask is inserted; when images are in focus, the post-processed photos will look the same to the human eye. The point of significance is when a user mistakenly takes an out-of-focus photo. For example, a user may want to focus on a face, but the camera’s auto focus detects the area behind the face instead. The mask allows the user to recapture focus during post-processing so that the face is no longer blurry.

"We figured out what it means for an image to be out of focus," said Raskar about MERL’s masked-based approach.

Using the combination of the mask and the post-processing software, MERL researchers were able to reconstruct a 4D light field from the standard 2D camera, explained MERL Visiting Scientist Amit Agrawal. Instead of bending light rays, the patterned mask attenuates the rays inside the camera. The post-processing software reconstructs the light field using an inverse computation of the Fourier transform equation, allowing the user to refocus the image.

Inspiration

MERL’s research is partly inspired by the Stanford method, a venture that sought to refocus an image post-capture with a handheld plenoptic camera, or rather a 4D light field camera that can adjust focus after capture. Announced in 2005, the handheld plenoptic camera is on its way to being commercialized.

One of the drawbacks with the handheld plenoptic camera is that although the software successfully recreates depth of field, it degrades an image’s quality to from its original resolution. In other words, a 16MP image resolution would be reduced to an output of a 300x300 pixel image.

The MERL heterodyne light field camera, on the other hand, preserves the original resolution of the image, so they’re still suitable for printing, they said.

Coded Aperture Versus Photoshop

When asked how using the coded exposure compares to Photoshop tools such as the Sharpen function, MERL researchers asserted that the process doesn’t simply sharpen a photo but actually reduces blur by increasing the depth of field by a factor of 10.

The heterodyne light field camera user does "something proactive at the time of capture," said Raskar. Using the coded aperture requires the same amount of time as editing a photo using Photoshop’s sharpening tools, he added.

**Limitations
**There are a few drawbacks to the heterodyne light field camera. Although it can preserve image quality and increase depth of field, the technology only works in select situations. The camera cannot account for complex scenes such as those with obstructions, such a landscape with a lot of trees. Transparent obstructions, such as glass door in front of a person, may also impede results.

For the majority of situations though, the heterodyne light field camera could be beneficial for consumers, according to the researchers.

Consumer Applications 

Coded aperture will give users more leeway when taking a photo.

"Coded aperture, or masked aperture, gives [people] extra protection," said Raskar. "People don’t have to worry about focusing," added Agrawal.

Although the prototype is a modified Canon Digital Rebel XT DSLR camera with a Canon lens, the MERL scientists said the adjustments can be made to any standard camera and the technology is "ideal for consumer photography." They foresee camera manufacturers or camera hobbyists implementing coded aperture.

The mask would only take pennies to make and could even be printed out on ordinary transparent paper, according to Agrawal.

If camera manufacturers decided to implement the technology, adding the masked aperture would be a simple, minor modification and would not require additional optical components. Raskar envisions coded exposure as an additional aperture setting on the camera dial. Along with the standard aperture settings f/2.8 to f/22, the last setting could be labeled something along the lines of "C" for coded exposure. Users could then bracket by focus, according to Agrawal.

As far as the post-capture software goes, Agrawal said that it could easily be implemented as a Photoshop plug-in.

Described as "ready to go," the technology could be used for traditional photography, microscopy, security imaging, and in the scientific fields.

"This breaks new ground in understanding the issue of focus," said Raskar. "It takes it to the next level of understanding the whole problem."

The paper will be published in July and was authored by MERL senior research scientist Ramesh Raskar, MERL visiting scientist Amit Agrawal, Northwestern computer scientist assistant professor Jack Tumblin, and students Ashok Veeraraghavah and Ankit Mohan, from the University of Maryland and Northwestern University, respectively. MERL will present the publication at Siggraph this coming summer.

For more information, visit the following MERL websites:
www.merl.com/people/raskar/Mask/index.html
www.merl.com/people/raskar/raskar.html 
www.merl.com/people/agrawal/index.html

Images courtesy of Mitsubishi Electric Research Labs.