The new computer animated version of Disney’s The Lion King is so convincingly photorealistic, many writers and fans have been referring to it as a “live-action” remake. In a sense though, as The Lion King crew member and 2009 BS in Computer Science in Real-Time Interactive Simulation graduate Adrian Sotelo explains, the animated film is live-action.
“In all cases,” Sotelo says, “the camera shots were controlled by a human. If there was a dolly shot, there was a flesh-and-blood person — called a grip — running back and forth pushing a dolly on tracks that were installed on a stage, putting his back into it so that the camera wouldn’t jostle too much as he decelerated.”
Wait a minute! What?
If you’re a little confused right now, don’t feel bad. The bridge Sotelo and his team built between animated and live-action filmmaking, a cutting-edge technique called “virtual production,” is mind-blowing even after you fully understand what it entails. As The Hollywood Reporter wrote, it “could be a game-changer for filmmaking.”
Before Sotelo earned his name in the credits on The Lion King, he was working at Blizzard Entertainment in Irvine, California, as a gameplay engineer on World of Warcraft. Serendipitously, as he started looking for work a little closer to home in Los Angeles, he got a call from fellow DigiPen graduate Jody McAdams about an open position at the L.A.-based virtual reality studio Magnopus.
“They were taking the very avant-garde technology that is VR/AR and putting it to work on an innovative project, doing things I hadn’t considered using VR for, one of which was something called virtual production,” Sotelo says. “[Jody] was very hush hush about the details, but I felt that this was an excellent opportunity for me to grow as both a developer and technologist.”
Once Sotelo was hired, he discovered that the “hush hush” project happened to be Disney’s The Lion King. As the new virtual production systems lead, he suddenly found himself coordinating a team of programmers and technical personnel to create something “that had never been done before.”
“In a nutshell,” Sotelo says, “virtual production is the use of a virtual space to aid in the production of a movie.” While other films and games have utilized the technique in bits and pieces to achieve special shots, The Lion King was the first feature-length film entirely created with it. The whole movie was “filmed” using the virtual production system Sotelo and his team built, inside of what is essentially an advanced multiplayer VR game.
Moving Picture Company VFX supervisor Adam Valdez breaks down how virtual production technology worked on the film.
Here’s how it works.
First, the studio responsible for The Lion King’s animation, the Moving Picture Company, created the 58 square miles of CGI African landscapes that serve as the film’s set. Those 58 square miles of digital land were then dropped into the game engine that Magnopus’ virtual production system was built upon. Low-fidelity “previsualization” versions of characters — from the young lion Simba to the wise baboon Rafiki to the large throngs of wild creatures — could then be placed and rearranged in the system, acting out their pre-animated scenes in a loop on the virtual set. Then, using either real or stand-in cameras tracked by the simulation with painstaking accuracy — cameras which were often attached to real-world tools of the trade like cranes, dollies, and steady cams — the virtual production magic that Sotelo and his team made possible would begin.
The whole movie was ‘filmed’ inside of what is essentially an advanced multiplayer VR game.”
“The crew could put on VR headsets and step into the world they were filming. They could walk around and scout locations together. If they wanted to, they could literally get on one knee and see what a shot looked like from a lower angle,” Sotelo says. “Even though the characters and sets were all computer generated, that CG world was filmed in the real world just as if they were standing in the savannahs of Africa.”
The Magnopus tech crew was on site in the same building as The Lion King’s directors and film crew during filming. As a matter of fact, the real-time simulation the crew was virtually filming was oftentimes being developed by Sotelo’s team in real-time too, a necessity given how unprecedented the technology was. “We were building the system at the same time it was being used,” Sotelo says. “My team and I were very involved in helping to set up shots and were on-hand to address any problems as they arose. It wasn’t uncommon for me to be called over to the stage and for me to sit down and write code because the director, or the director of photography, needed something.”
The Lion King director Jon Favreau discusses virtual production with BBC, calling it “a bespoke multiplayer VR filmmaking game.”
The Lion King’s use of virtual production is being hailed as a game changer, because it saves time and money on both the traditional and computer animated end of filmmaking. For one, director Jon Favreau and his team didn’t have to spend millions of dollars shooting and reshooting scenes on location in Africa or compositing animated characters on top of the footage. All the filming happened in one warehouse in Los Angeles.
On the animation end, instead of having to wait weeks to review scenes, make changes to the scenes, and then re-render everything, virtual production reduces that weeks-long iteration process to a matter of seconds. The VR-equipped directors can change the lighting, the set, and the location of characters and assets in real-time and instantly see the results. Once everything’s been tweaked to the director’s liking and filmed by the crew, those low-fidelity versions of the scenes, camera moves and all, can be sent off for final touch-ups to the animation studio — no further edits necessary.
After years in the game industry, Sotelo admits the jump to film was a lot to handle at first. “There was definitely a feeling of being Dorothy landing in Oz when I made the switch,” Sotelo says. “The acronyms alone made my head spin.” Now that The Lion King is out in theaters, already ranking as one of the top 20 highest box-office earners of all-time, Sotelo has jumped back into the game world, working at Respawn Entertainment on the studio’s highly anticipated Star Wars Jedi: Fallen Order. Even though Sotelo and his team’s achievements on The Lion King are being hailed as a technological breakthrough, he hasn’t let it get to his head.
“From a practical standpoint I learned so many things at DigiPen that I use on a daily basis — 3D math, operating systems, the rendering pipeline,” Sotelo says. “However, in a more intangible way, I would say one of the biggest takeaways I try to use on a daily basis is the attitude I learned while at school — leave your ego at the door, and work as a team.”