Content material streaming and engagement are getting into a brand new dimension with QUEEN, an AI mannequin by NVIDIA Analysis and the College of Maryland that makes it potential to stream free-viewpoint video, which lets viewers expertise a 3D scene from any angle.
QUEEN may very well be used to construct immersive streaming functions that train expertise like cooking, put sports activities followers on the sphere to observe their favourite groups play from any angle, or convey an additional stage of depth to video conferencing within the office. It may be utilized in industrial environments to assist teleoperate robots in a warehouse or a producing plant.
The mannequin might be introduced at NeurIPS, the annual convention for AI analysis that begins Tuesday, Dec. 10, in Vancouver.
“To stream free-viewpoint movies in close to actual time, we should concurrently reconstruct and compress the 3D scene,” stated Shalini De Mello, director of analysis and a distinguished analysis scientist at NVIDIA. “QUEEN balances components together with compression fee, visible high quality, encoding time and rendering time to create an optimized pipeline that units a brand new normal for visible high quality and streamability.”
Scale back, Reuse and Recycle for Environment friendly Streaming
Free-viewpoint movies are sometimes created utilizing video footage captured from totally different digital camera angles, like a multicamera movie studio setup, a set of safety cameras in a warehouse or a system of videoconferencing cameras in an workplace.
Prior AI strategies for producing free-viewpoint movies both took an excessive amount of reminiscence for livestreaming or sacrificed visible high quality for smaller file sizes. QUEEN balances each to ship high-quality visuals — even in dynamic scenes that includes sparks, flames or furry animals — that may be simply transmitted from a bunch server to a consumer’s gadget. It additionally renders visuals sooner than earlier strategies, supporting streaming use instances.
In most real-world environments, many components of a scene keep static. In a video, which means a big share of pixels don’t change from one body to a different. To avoid wasting computation time, QUEEN tracks and reuses renders of those static areas — focusing as an alternative on reconstructing the content material that adjustments over time.
Utilizing an NVIDIA Tensor Core GPU, the researchers evaluated QUEEN’s efficiency on a number of benchmarks and located the mannequin outperformed state-of-the-art strategies for on-line free-viewpoint video on a spread of metrics. Given 2D movies of the identical scene captured from totally different angles, it sometimes takes beneath 5 seconds of coaching time to render free-viewpoint movies at round 350 frames per second.
This mix of pace and visible high quality can help media broadcasts of live shows and sports activities video games by providing immersive digital actuality experiences or prompt replays of key moments in a contest.
In warehouse settings, robotic operators might use QUEEN to raised gauge depth when maneuvering bodily objects. And in a videoconferencing utility — such because the 3D videoconferencing demo proven at SIGGRAPH and NVIDIA GTC — it might assist presenters reveal duties like cooking or origami whereas letting viewers choose the visible angle that greatest helps their studying.
The code for QUEEN will quickly be launched as open supply and shared on the undertaking web page.
QUEEN is certainly one of over 50 NVIDIA-authored NeurIPS posters and papers that function groundbreaking AI analysis with potential functions in fields together with simulation, robotics and healthcare.
Generative Adversarial Nets, the paper that first launched GAN fashions, gained the NeurIPS 2024 Take a look at of Time Award. Cited greater than 85,000 occasions, the paper was coauthored by Bing Xu, distinguished engineer at NVIDIA. Hear extra from its lead creator, Ian Goodfellow, analysis scientist at DeepMind, on the AI Podcast:
Be taught extra about NVIDIA Analysis at NeurIPS.
See the newest work from NVIDIA Analysis, which has a whole lot of scientists and engineers worldwide, with groups centered on subjects together with AI, pc graphics, pc imaginative and prescient, self-driving automobiles and robotics.
Educational researchers engaged on massive language fashions, simulation and modeling, edge AI and extra can apply to the NVIDIA Educational Grant Program.
See discover relating to software program product data.