Virtual production has rapidly become one of the most promising techniques in media. But while a huge and sophisticated real-time display instead of a green screen solves many problems, it also creates new ones. The creators of Netflix’s mind-bending Dark took on this challenge for the production of their new show, 1899 — and combined old-school cinema chops with cutting-edge tech and one giant, ambitious mechanism to build something literally revolutionary: a facility that they call Dark Bay.
Most people will probably be familiar with the virtual production style through the publicity surrounding The Mandalorian, which was the first content to be shot largely in this way. But even that show shot half its scenes outside the volume — something that was more or less impossible for 1899.
I talked with Philipp Klausing, producer on the show and managing director of Dark Bay (the overarching production company is called Dark Ways).
“We’re a little different because we come to it from a creative perspective,” he said. “The story was the driving input in setting up the stage: we have a show set on the ocean, on a steam ship that doesn’t exist in the world, and we knew with corona we couldn’t really do the show classically. How can we film these locations?”
In other words, they didn’t see what ILM was doing and think, that looks cool, let’s try it. Showrunners Baran bo Odar and Jantje Friese had a vision but the tools to create that vision didn’t exist…yet.
“This is what made us different, we had content we were working towards and had to come up with solutions, with ideas, to make these worlds real,” he continued — but as it turned out, “we came into it totally naive.”
The means of virtual production
A virtual production facility, also called an LED wall or simply a volume, is essentially a giant high-performance display, and in many ways a dream come true for filmmakers. To be able to switch out photorealistic backgrounds at will, to include effects in-camera rather than shoot against a green screen and add in post, to use the enormous walls of panels as variable and creative lighting — it really is a box of toys that creators have only just begun to unpack.
But Klausing explained that these capabilities come with some serious limitations that aren’t immediately obvious.
“However you [build a volume], you try to achieve a full cylinder or circle around you so you can place your camera wherever you like,” he explained. “You soon realize you have all kinds of limitations on all sides and you have issues with how to access the stage. The volume itself is a restrictive room, even if you build it big.”
For instance, large, sweeping camera movements like crane and drone shots simply aren’t possible in a volume. But it’s more than that kind of common location restriction — you can’t really use cranes or drones effectively inside either.
ILM shows off the new Stagecraft LED wall used for season 2 of ‘The Mandalorian’
“In order to crack VP, you need to see that the focus is not on the background — it’s LED, so it’s out of focus. The focus is on the foreground. You need to give it a way to be flexible as well,” he continued — but therein lies the problem: “If you just put the deck in the volume, and you want to shoot over the railing into the ocean, it makes the reverse shot of coming from the ocean over the railing impossible.”
In other words, using a volume restricts a number of standard film techniques that simply don’t work with the way you access and film the set. If you imagine the volume as a pizza with a couple slices missing on one side where the camera and crew are set up, the set as well all the shots and props and equipment need to be organized around that particular orientation. If you’re shooting a living room scene it’s one thing, but a ship?
A revolving revolution
“We thought, we’ll have to compromise, because we have a whole ship we have to put in there — we can’t just put it in there piece by piece, because it will not fit with the schedule. But one of the showrunners, it was probably Baran, said, why don’t we just do a turntable? Then we can rotate the physical set,” Klausing recalled.
“People were skeptical… but it kept coming up on calls.”
Indeed, being able to back the cameras and crew out and revolve the set not only removes the inability to shoot 180s, but adds an interesting new capability to the whole idea of virtual production.
“This is where the concept of the revolving stage and the ‘pizza slice’ technique was born,” said Netflix’s head of virtual production, Girish Balakrishnan. But it’s the exact opposite of the pizza slice problem mentioned earlier. And to be clear, it’s not about rotating the set during a shot.
“While there have been innovations on panel technology, camera tracking and real-time rendering, it was important to take a closer look at the art of building sets for LED volumes. While adjusting a virtual set is a push of a button, physical set changeovers often required significant production time between setups,” he said.
“The unique benefit of the revolving stage is that you can dress the physical set in sections and have the digital content on the LED wall match that section specifically. This means you can have multiple physical and digital sets on the stage within a single shooting day, and rotate as needed to shoot multiple angles and coverage.”
For example, you could shoot the over-the-shoulder conversation at the railing, then rotate the set 30 degrees, shoot a scene on the deck, rotate the set 100 degrees, then shoot the reverse of the conversation, then change the lighting, add and subtract props, and shoot a single of one character at the railing at night. (And because you can control the weather, time of day, and all the rest, these need not even be part of the same sequence.)
It was an elegant, if largely unprecedented, solution to a problem faced in volumes around the world. Now they just had to build it. Only one problem — no one on the team had ever done that before.
All in the dark
Building a virtual production studio is no simple matter. It’s incredibly technically demanding, not to mention expensive, and that’s before you add in the whole “the entire set rotates” thing. And to add to their difficulties, the pandemic meant no one could leave and go learn from an existing facility.
“We had no volume to go to; there was nothing in Germany, the travel ban being active. We had the Netflix team supporting us and giving us some introductions, and basically the podcast of The Mandalorian — that was it,” Klausing explained. “We couldn’t leave that theoretical realm until we had the stage operational. So it was all in the dark.”
“Fundamentally we believe that the filmmakers have the right to choose the right tool for the job, and our role is to help them succeed,” said Balakrishnan. “Dark Bay was born out of necessity from the challenges of filming during the pandemic with the aim of not sacrificing creative scope and scale, and instead bringing the diverse landscapes of Europe and the grandeur of a ship on the high seas to an immersive sound stage in Babelsberg.”
Balakrishnan credited production designer Udo Kramer and construction manager Andreas Vieweg especially with the creation of the revolving stage.
But it was very much a group effort, Klausing said.
“Our partners — Framestore, Faber AV for panels, Epic of course, Arri — our [director of photography] and our director, we tried to get all the info from everybody and then… godspeed,” he said. “We tried to get the best results for directing the actors; we wanted enough room to move them around. The DP did a process with the lenses so we could extend the bokeh — it gives more foreground to direct in. But we had no way to test that stuff. It was a frustrating phase in the beginning.”
The promise, or rather necessity, of rotating the set both simplified and complicated, but ultimately Dark Bay was assembled (you can watch the timelapse below, first shown in this interesting American Cinematographer article).
Klausing deferred to the technical authorities to describe some of the work done on their side. For instance, Framestore’s Jack Banks mentioned some of the advances they made during the setup of the LED walls, starting from the panels that made them up. These had to be both high resolution enough to provide good imagery but as a consequence required more computing power, and additionally needed to interact well with water, of which there was plenty to be found on set.
“Practically, the panels are the most expensive part of the system and the hardest to change out, so I would say it’s probably best to get the panel right for your project — computers are easy to deal with,” Banks said. “We worked very closely with Epic and Nvidia to bring dual graphics card support online in our custom Unreal Engine build, giving us roughly 25-40% increase in frame rate on our environments. With 1899’s Digital Imaging Technician Richard Muller, we built a robust color pipeline. Finally, working very closely with Arri we developed new software and firmware to pass data from the camera to our system with the smallest amount of delay possible allowing true real time lens tracking.”
Digital sets were designed on paper, built digitally, scouted and blocked virtually, constructed in the real world, shot with virtual and physical lenses and cut into an edit on-set,” said Balakrishnan. “To facilitate the set creation process, a digital proxy of the Dark Bay volume was developed in-engine so that the physical and virtual art department could work in tandem to determine what would be practically built and what would be a digital set extension.”
As you can see, productions like this one are as much about refining and creating digital cinema techniques as they are about using them to tell a story. Without, for example, the improved lens tracking and foreground flexibility, the director and DP would have fewer tools on their belt, restricting creative freedom.
Obstacles to transformation
While the team certainly seemed satisfied with the work they’ve done (1899 is in post-production now, no release date set), Klausing said that the process was a difficult one to adapt to in some ways and not one that every creator will be able to embrace.
If you think about how a film or show is normally shot, the decisions on sets, backgrounds and other aspects of a scene often don’t need to be finalized until the day of shooting, or even long after that in post if there’s a green screen involved. But the nature of a virtual production is that a lot of those effects and scenery, such as what’s happening behind an actor, or props and extras, need to be locked in long before shooting.
“It’s the future, as a producer I see that, being able to shoot these fantastic worlds without leaving the soundstage, and you can edit it the next day,” he said. “But we need to work on the asset process. Right now you need to allocate a lot of work from post-production to pre-production and change costs and commit to work on them in order to shoot them.”
Logistically speaking, it concentrates resources on the early stages of production, meaning if you want to do VP you’ll need your money, storyboards, actors and locations lined up much earlier than in a traditional shoot. It’s one thing for a major studio with the backing of Netflix to commit to scouting and shooting locations, locking in actors, finalizing 3D assets and other tasks from day one — but a smaller operation or independent filmmaker might not have that option.
Not that this is an insurmountable barrier: it’s just an optional change to how the production process works. Klausing, and others I’ve talked to on this topic, suggested that better sharing of virtual assets is a big part of making this easier on everyone.
“All these game engine platforms, they’re not ‘camera ready’ the way we would define it as filmmakers,” he said. “I’d like to see the studios share whatever we collect and create, generic assets like a facade or mountaintop or the moon, things that come up in other shows and can be manipulated by those shows, but are already constructed. Then it’s there, it’s available, it’s a worldwide library we can just scan through.”
It also changes the balance of power on set away from directors and actors and toward cinematographers, production designers and VFX artists. Again, not a bad thing, but something that needs to be prepared for, by making the creative process more inclusive of every branch of the production team.
Lastly, Klausing suggested that while productions like 1899 and The Mandalorian are certainly pushing things forward, we should look to historically under-resourced areas to see how VP really affects filmmaking.
“Whoever is restricted by their language, their financing — for instance, some people may have to spend their budget in their own country — a volume opens up a whole new world,” he said. “Everybody talks about Netflix’s success in Europe, but also Korea, Asia in general, you can tell your local language story but give it that international touch of your characters being able to travel, etc. We will see a lot of innovation in those markets, I think, because they have the need and the appetite to tell their stories differently.”
1899 has wrapped and will debut on Netflix whenever post production has finished. Dark Bay is open for others and has already hosted at least one other production. And given the benefits realized by this innovation, it seems likely that soon it won’t even be the only rotating volume out there.
Comment