Carnegie Mellon Develops Algorithm to Transform 2D Photos into 3D Renderings

November 8, 2006 - Researchers at the Carnegie Mellon University School of Computer Science have developed a computer vision algorithm, the Automatic Photo Pop-up program that essentially can turn a two-dimensional outdoor photo into a 3D model. 

With two years of research behind it, the Automatic Photo Pop-up draws on geometry to compile a three dimensional scene from a single photo.The Automatic Photo Pop-up could have a host of possibilities, most notably to create better navigation systems in robotic technology.

We are already familiar with computer vision in existing sensor technology. Sensors can detect faces in facial recognition digital cameras.Robots can avoid obstacles while vacuuming our carpets, and computers can even parallel park our cars. 

Even though computers can detect faces, pedestrians, and room parameters, existing programs lack a sense of the whole picture and lack consideration for the geometry of the entire scene.As Derek Hoiem, Carnegie Mellon Robotics Institute graduate student points out, "One thing we need before robotics become household objects are better sensors."   

The Automatic Photo Pop-up program is closer to reaching that goal.It works in ways similar to that of a children’s pop-up book, from which the program’s name is derived.A 3D model depends on three factors: the objects in the scene, orientation of surfaces, and camera viewpoint, according to the abstract "Putting Objects into Perspective" by Hoiem, Carnegie Mellon Assistant Professor Alexei Efros and Professor Martial Hebert.To get an estimate of where surfaces are in a scene, the algorithm computes different factors including ground, vertical, and sky segments.Based on statistical framework, the program produces a rough estimate of the geometry of the scene.Like its name, the Automatic Photo Pop-up "folds up" vertical objects into a scaled model to produce the 3D scene.

The Automatic Photo Pop-up could be applied to a broad range of interests.Some suggestions from possible long-term investors range from everyday home use to vital defense systems, according to Hoiem.Users at home could one day make 3D models from their online photos or camera phone pictures.Businesses with real estate agents or architectures could use the Pop-up for client models, detailing the insides of building.Then there are possibilities for defense, including automatic vehicles and weapons, said Hoiem. 

The Automatic Photo Pop-up would help average users to "move around in a scene," said Hoiem, "a virtual world."