What is virtual production and how it is redefining filmmaking
Due to a demand for incredibly high-quality film and TV production, at a fraction of the time and cost, filmmaking timelines need to be reviewed, bringing creative decisions much earlier into the production process.
Virtual production harnesses the power of virtualising technologies to create digital environments in, and through which film and television can be made. Virtual production enables filmmakers to render photorealistic computer-generated scenes on an LED background and interactively visualise them in real-time. Together with high-power graphics cards, camera tracking, as well as augmented and mixed reality, these technologies marry the virtual and physical world, where what you see in camera is what you get.
Virtual Production Technologies
The range, application and combination of digital technologies used in virtual production marks a shift in the way film and television are made. We can trace the development of this shift through some of the key technologies (and their histories) that enable virtual production approaches.
Game Engines
A game engine is a software development programme which was originally developed to create video games. However, game engines are now being used for different forms of visualisation and collaboration including virtual production. Game engines work by laying a framework which allows the user to create something more easily than if they had to make something for scratch. This framework for creation varies across different game engine creators but typically includes a 2D or 3D rendering engine.
Achieving Parallax
A virtual camera updates to match the perspective and parallax which the physical camera records on set. Parallax is the perceptual difference in an object's position when it is seen from different points of view. For virtual production, trackers and computers calculate the correct relative position of an object to the position of the camera in the real world in order to create accurate parallax. The game engine matches the physical camera coordinates to that of the internal virtual camera within the virtual environment allowing the viewpoint of the virtual camera to be rendered and displayed on the LED wall.
Motion Capture ('mo-cap')
LED volumes need to have a motion-capture (or 'mo-cap') system in order to be aware of where the physical camera is and how it is moving to ensure appropriate parallax on the volume. The movements of the paired virtual and physical cameras are rendered in real time, allowing the perspective of the scene to move with the camera angles. Seamlessly displaying high quality imagery in LED volumes and achieving a convincing parallax effect requires the combination of complementary technologies.
Blue or Green Screen
Many film and TV productions utilise green or blue screen backdrops that allow CGI (computer generated imagery) environments to be added in post-production. In virtual production, green screens allow real-time rendering of virtual worlds as the live action occurs in a studio, rather than being added in post-production.
LED Volumes
Major advancements in LED panel technology, developed in the live events field and combined with technologies from the aviation industry and military flight simulation systems, have enabled virtual production using large walls of LED screens called volumes. Virtual environments are projected onto the LED screens in camera-ready quality. The often-large size of these LED volumes allows the use of physical props and sets in conjunction with virtual environments, and are typically big enough for cast and crew to operate in.
Virtual Reality (VR)
VR is typically experienced using a headset (such as Oculus Rift or Google Cardboard) with headphones and involves experiencing some degree of virtual realm or ‘reality'. VR technologies enable filmmakers to be able to scout a digital version of a real place or construct a virtual location to visualise a scene to plan the filming process – allowing for real-time collaboration across the world. VR platforms have also been used to run virtual production systems, with trackers on cameras, props and real spaces used to provide the tracking needed to accurately combine movements and filming angles in the physical world with virtual environments.
Augmented Reality (AR)
Augmented Reality (AR) enhances the user's view of their surroundings by overlaying it with digital objects in real time. Augmented Reality is different to Virtual Reality (VR) as AR adds to real world experiences rather than creating whole new 3D worlds - virtual information is simply placed on top of the real world to enhance the user's experience. In AR, digital overlays are typically viewed through a device (such as a smartphone or glasses). Pokemon Go is perhaps the most well-known application of augmented reality for smartphones. Users view and ‘collect' digital Pokemon which are overlaid on real world locations through their phone.
Virtual Production Stages
There are several virtual production stages, each contributing to the creation of immersive and visually stunning cinematic experiences. These stages typically include:
- Pre-visualization (Previs): Previsualization is the initial stage where filmmakers conceptualize and plan the scenes using digital tools. Artists create rough 3D models, storyboards, and animatics to map out camera angles, actor movements, and overall scene composition. This stage helps filmmakers visualize their ideas and make informed decisions before entering production.
- Virtual Set Design: In this stage, virtual environments and sets are designed using advanced 3D modeling software. These virtual sets can range from fantastical landscapes to realistic urban environments, tailored to the specific requirements of the project. Attention is paid to detail, lighting, and texture to create believable and immersive virtual worlds.
- Performance Capture: Performance capture involves recording the movements and expressions of actors to animate digital characters or enhance real-time virtual environments. Actors may wear motion capture suits equipped with sensors to track their movements accurately. Facial capture technology captures subtle facial expressions, adding depth and realism to virtual characters.
- Real-Time Rendering: Real-time rendering is a crucial stage where virtual environments and characters are brought to life in real-time during filming. High-performance rendering engines render graphics instantaneously, allowing filmmakers to see virtual elements seamlessly integrated with live-action footage on set. This enables directors to make immediate creative decisions and visualize the final look of the scene during production.
- In-camera Visual Effects (ICVFX): In-camera visual effects involve the integration of virtual elements directly into the camera feed during filming. LED screens or projection systems display real-time virtual environments or backgrounds behind actors, creating immersive surroundings. This technique enables filmmakers to capture complex shots with realistic lighting and reflections, reducing the need for extensive post-production work.
- Post-Production: Post-production is the final stage where footage from virtual production is refined, edited, and enhanced. Visual effects artists add finishing touches, such as compositing virtual elements with live-action footage, color grading, and refining special effects. Sound design, music composition, and other audio enhancements are also completed during this stage to deliver a polished cinematic experience.
These stages collectively demonstrate the intricate process of virtual production, combining creativity with cutting-edge technology to bring captivating stories to life on the screen.
Conclusion
In conclusion, virtual production stands poised at the forefront of a cinematic revolution, promising to redefine the very essence of filmmaking. With its seamless integration of digital technology and traditional filmmaking techniques, virtual production offers a transformative approach to storytelling that empowers creators with unprecedented creative freedom and flexibility. By leveraging advanced 3D modelling, real-time rendering, and in-camera visual effects, filmmakers can craft immersive worlds, capture dynamic performances, and realize ambitious visions with unparalleled efficiency and precision. Moreover, virtual production democratizes access to cinematic storytelling, breaking down barriers of cost and logistics, and opening doors to new voices and narratives from around the globe. As we embrace the boundless possibilities of virtual production, we embark on an exciting journey into the future of filmmaking, where imagination knows no bounds, and the art of storytelling evolves in ways yet unimagined. Together, let us seize this moment of innovation and exploration, as we usher in a new era of cinematic excellence through the transformative power of virtual production.
Tags:
Image Credits: Freepik
