What is DeepFovea?

What is DeepFovea?
Facebook announced the release of DeepFovea, a new state-of-the-art foveate rendering using AI technology.
New York, April 2, 2020
Facebook Reality Labs engineers have developed an image wizard that can be used to create a „plausible peripheral image“ instead of actual images that are blurred and unfocused. This image reproduction is called „Foveated Reconstruction“, which is achieved by compressing the pixels on the RGB (Red, Blue, Green) video 14 times without compromising quality, and is realistic and view-dependent.
DeepFovea is one of the first generative networks (GAN) capable of producing natural video sequences, say Facebook developers about the technology. „DeepFovea can reduce the amount of computing resources required for rendering by up to 10-14x, while keeping image differences imperceptible to the human eye,“ Facebook says.
Foveation means bending an eye to focus the view. When the human eye focuses on an object, it sees the object directly and tends to exclude the peripheral region where objects are only vaguely registered. DeepFovea takes advantage of recent advances in generative advertial networks (GANs), a deep machine learning system that predicts outcomes based on observations that can produce the missing peripheral details by creating content that is consistent in perception.
An immersive AR/VR experience requires precise and accurate eye tracking performance. The cameras in the VR headsets track the movement and position of your pupils, so the GPU knows where to focus for high-quality image reproduction. Foveated rendering is an essential component of virtual reality (VR) and augmented reality (AR). Foveated rendering can save 30 percent of the GPU’s power consumption, which can be used to enhance detail in areas viewed by the student.
DeepFovea delivers higher quality peripheral renderings with greater efficiency compared to what is available today for standard faceted renderings.
Facebook notes that the software uses 14x reduced renderings and also delivers real-time vdeos without delay with gaze recognition based on Deep Machine and AI processes. This will contribute to the development of better VR headsets.
Current VR and AR headsets are unwieldy and impractical for 3D mapping due to their high power consumption. The foveated reconstruction technology used in DeepFovea will significantly increase the rendering efficiency in current VR and AR headsets. Facebook first gave an indication of their work on DeepFovea at the Oculus Connect 5 last year. They talked about coming to market with high-quality VR and AR headsets within the next five years. They said that Foveation can be used in combination with deep learning and gaze tracking to deliver high resolution headsets.
Current VR and AR headsets are bulky and impractical for 3D mapping due to their high power consumption. The Foveation reconstruction technology used in DeepFovea will significantly increase rendering efficiency in current VR and AR headsets. Facebook first announced their work on DeepFovea last year at Oculus Connect 5. They talked about coming to market with high-end VR and AR headsets within the next five years. They said that Foveation can be used in combination with deep learning and gaze tracking to deliver high resolution headsets.
Facebook is releasing a sample version of DeepFovea’s network architecture for researchers, VR engineers and graphic designers. The company plans to present the research paper soon at Siggraph Asia and will make the samples available after the event.