It takes many people to make a movie. Everybody sees the actors, while others operate the cameras, audio equipment and lighting. Many others work behind the scenes to build the imaginary world, from props to costumes to set design.
A whole other category of people work in digital effects during post-production. They often have to remove the background and replace it with digital assets. One method to remove the background is with green screens, but this takes a lot of time during photography. Another method is called rotoscoping, where nothing is done during photography but the video-effects team has to painstakingly trace the outlines for every single frame later.
What if there were a better way – one with little effort during photography that still makes life easier in post-production?
In the VoLARe project (Video Light field Acquisition and Restoration), funded by MCST, we worked on just such a technology based on a technique called light field photography. The project combines the University of Malta’s academic expertise in signal and image processing with the practical experience in cinematic photography and post-production of Stargate Studios Malta.
The project combines the University of Malta’s academic expertise in signal and image processing with the practical experience in cinematic photography and post-production of Stargate Studios Malta
In traditional light field photography, a large array of identical cameras capture a scene from slightly different angles. Later, software can combine these photographs in different ways. For example, we can obtain a depth map of the scene: a detailed distance guide for each point in the scene. This provides depth perception so the 2D scene can be interpreted as 3D. Then we can remove the background by simply selecting a depth threshold and treating everything beyond the threshold as background.
While light field technology is well understood, applying it to video and cinematic photography introduces new challenges. Issues like synchronising the videos together were straightforward to solve, while others required more effort and ingenuity. Light field set-ups often have close to 100 cameras, which we reduced to nine cameras to keep the set-up manageable by camera operators. We also kept a conventional cinema camera and lens combination as the centre camera, allowing the directors of photography to exercise their usual creative freedom. The cameras surrounding it need to be much smaller, so we used GoPro cameras.
We have come a long way in these four years, with a usable rig and experience with it in real video shoots with multiple camera set-ups, on set and on location.
The main remaining challenge is to calibrate the cameras with pixel-level accuracy to allow our algorithms to determine depth maps accurately.
Sound Bites
• In acting, if you are connected to Kevin Bacon via a certain number of links, you have a Bacon number. In academia, if you are connected to the mathematician Paul Erdős via co-authored papers, you have an Erdős number. Natalie Portman has both (Bacon 2, Erdős 5).
• Scholars have discovered evidence of a sixth basic taste. The tongue responds to ammonium chloride, a popular ingredient in some Scandinavian candies. The OTOP1 protein receptor, previously linked to sour taste, is activated by ammonium chloride. The ability to taste ammonium chloride may have evolved to help organisms avoid harmful substances.
For more soundbites, listen to Radio Mocha every Saturday at 7.30pm on Radju Malta and the following Monday at 9pm on Radju Malta 2 https://www.fb.com/RadioMochaMalta/.
DID YOU KNOW?
• Word of the DAY: AI-NXIETY (neologism) – feeling anxious about how AI could affect your life.
• The Korean equivalent of ‘once upon a time’ is ‘in the old days, when tigers used to smoke’.
• Languages that are less ‘efficient’ (they use more syllables to communicate the same meaning) are usually spoken faster so the rate of information exchange remains roughly similar.
• The average new car in Europe has grown half a centimetre wider every year since 2001.
• For Valentine’s Day, an American animal shelter will bury your ex’s name in a cat litter tray for a $5 donation. Then a cat will poo on it.
For more trivia, see: www.um.edu.mt/think.