The future of immersive content experiences has arrived. Using the latest RED cameras, processors, graphics cards and distribution technologies, producers can now create true 8K content, live or recorded, with a supercharged workflow that consigns low resolution to history.
“We believe that a VR production at full 8K quality now meets the bar for attracting a wide, general audience to enjoy live events in VR headsets,” says Fabrice Lorenceau, vice president of Live Production and Video Technology at immersive content producer COSM. “We can now produce trail¬blazing 8K live VR experiences.”
As far back as 2015, expectations were high that 360-degree cinematic quality video would soon be a game-changing viewing experience for everything from sports to music promos to longform narrative. A combination of disjointed camera workflows, uncomfortable headgear, glitchy distribution and poor-quality displays has dampened adoption.
Those issues are rapidly being solved.
“The gap between user expectation and the less-than-stellar reality of the immersive experience has been erased,” says Lewis Smithingham, senior vice president of innovation at global marketing and advertising services company Media.Monks. “The opportunity to create content that fulfils the promise of VR is here now.”
At the root of the dramatic technology breakthrough is the ability to not only capture pristine large format 8K images but to stream the RAW 8K files live and direct from the camera.
RED Connect is a new tool which blasts open a whole set of creative applications for content creation in film, TV, live sports and experiential next-generation media.
To understand the impact of this breakthrough, it’s important to realize the massive disadvantage that immersive content experiences have previously been produced under.
“4K resolution is not nearly enough to meet the expectations of viewers typically watching HD television,” explains Lorenceau. “VR video that is spread across 180 or 360 degrees requires a very high source resolution to meet the expectations of viewers typically watching HDTV.”
Smithingham agrees. He has been pioneering VR content for years, previously as CTO and virtual director at Doug Liman’s production house 30 Ninjas.
“The headset resolution really matters. The fact that the screen is inches from your face and the image is being distorted, the data gets squeezed and stretched, means the difference between 4K and 8K is tremendously noticeable. It is ‘in your face’ noticeable even to the untrained eye.”
The next generation of headsets coming to market from major brands including Meta, TikTok and Sony will all feature much improved display resolution.
“That means producers need to deliver experiences at the highest possible resolution and quality,” says Smithingham. “Over the years we have all struggled through a wide range of cameras, rigs and workflows to produce VR, none of which were particularly efficient or delivered the final quality required. That has changed forever.”
For acquisition, COSM and Media.Monks chose RED’s V-RAPTOR, the next generation DSMC3 platform, which offers 8K Large Format coverage and 17-stop dynamic range.
Now used in tandem with RED Connect, users can unlock the unique ability to stream RAW files direct from the V-RAPTOR over IP to a CCU (camera control unit) in real-time, opening an extraordinary range of creative applications from VFX to live broadcast and true 8K VR.
GLOBAL SPORTING EVENT 8K VR first
Earlier this year sports fans got a first glimpse of what’s on the horizon when viewers enjoyed the most advanced global sports experience ever delivered with 150+ hours of live competition broadcast from the winter sporting event. COSM captured and produced VR 180-degree live coverage and 360-degree on-demand immersive feeds covering figure skating, ice hockey, snowboarding, freestyle skiing and more.
Viewers could access the coverage in VR in a few different ways. US viewers could use the US broadcaster’s app on Meta Quest 2 headsets, and fans in China used Pico Neo 3 headgear via the Chinese broadcaster app. No matter how they got there, enthusiasts were treated to ultra-high resolution immersive 8K streaming feeds of the worldwide winter sporting event.
It was the first experience – live or on-demand – that employed 8K VR180, a resolution that significantly, and visibly, improves over VR360. In addition, this was a 60fps workflow, a framerate that a proper sports experience demands.
COSM partnered with Intel and RED to design, develop and execute this ground-breaking project that was also the first use case for RED Connect. The innovation of RED Connect is set to dramatically enhance live entertainment experiences worldwide.
“This event was a huge leap forward and a world first,” notes Lorenceau. “We delivered an 8K live VR telepresence.”
“VR180 is a major quality step over 8K VR360,” he adds. “This is because in the VR180 feeds, the full 8K resolution was used for a (180-degree) hemisphere instead of a full (360-degree) sphere, more than doubling pixel density over a traditional 8K VR360 video to consumers around the world and that was a huge milestone for immersive media. It was a huge uplift in end user experience that has raised the bar so high there is no going back.”
The quality of the VR180 feed was un¬precedented, with a source signal that had a 360 ERP (equirectangular projection) equivalent of about 15K in the centre of the sensor.
The most significant technical change from past workflows was the 8K capture. Five RED V-RAPTOR bodies fitted with Canon EF 8-15mm f/4L Fisheye USM lenses were used to record the VR180 footage at 7,680 x 4,320 resolution. The 8K RAW feed was then streamed out of the camera via Secure Reliable Transport (SRT) to CCUs located at the IBC in Beijing using RED Connect (in beta at the time).
By simply connecting an ethernet cable to the V-RAPTOR, a CCU can process and output up to 8K at the 60p frame rate essential for fast action. Once the frame data is on the GPU memory of the CCU, the data can be shaped to fit the needs of the live output.
“Our goal was to give fans the feeling of being there, and we delivered,” says Lorenceau. “We showcased our first-in-class capability for delivering real-time sports content via ultra-high resolution 8K streaming using V-RAPTOR with RED Connect. It’s realistic to say that this set the stage for the future of immersive entertainment.”
Media.Monks on-set 8K workflow
RED Connect is also an accelerant for non-live workflows by capturing 16-bit original camera files (OCF) direct to cloud for instant on-set or remote editorial collaboration. On-set monitoring is enabled in full 8K straight from camera.
This is exactly what Media.Monks has been working on in their latest immersive experience productions.
“When you are shooting VR, it makes no sense to view on a 2D monitor,” notes Smithingham. “It’s simply unproductive to watch a flat image when you’re shooting immersive. It is vital to come up with a way to feed the live camera to a headset.
“Only RED Connect does this. We are now able to have a live monitor on set that is the same 8K resolution as the post files we deliver.
“The entire immersive experience from capture to delivery has changed so rapidly in the last six months, let alone since 2015 when VR began in earnest. The complete shift in what we did then to what we can do now is staggering.”
He adds, “We will be using RED V-RAPTOR cameras and RED Connect on every immersive shoot we do going forward.”
Future of immersive content creation
With RED Connect you can process full resolution, live R3D video for SDI output into a conventional broadcast backbone and power a live 8K broadcast. A single V-RAPTOR positioned high and central in a stadium can cover a sports match with artificial intelligence algorithms tracking play and switching the 8K UHD frame for multiple 4K UHD camera outputs. Volumetric capture for post manipulation is enabled with the same solution in real-time where ultra-high resolution OCF are available direct from camera into a connected environment.
COSM has created some of the richest immersive experiences in the world. Its divisions design and build complete domes delivering immersive hardware, software and physical structures.
The company’s more recent endeavours include the announcement of its first public venue, set to open in Hollywood Park, Los Angeles, California, and the COSM Experience Center in Salt Lake City, Utah, both locations outfitted with LED Dome displays capable of broadcasting the biggest sporting events in 8K resolution.
Lorenceau says, “We believe that at 8K quality of production and distribution, live VR video has finally crossed the experience threshold that enables a fantastic and engaging experience for a mass audience. I want to applaud RED’s leadership for helping us make this huge leap forward.”
