LED Stage Virtual Production Notes v.01

Scott Squires

These notes only cover LED Stage Virtual Production utilizing a 3D environment.

Choices

The first question is whether LED Virtual Production makes sense for a particular project. What are the budget and time comparisons to go to real locations? What about building a full interior set if that’s the intention?

Do you have the pre-production time and team to build the virtual world for the LED screen?

Do the scenes work within the limits of an LED stage? (Avoiding hard sunlight, height, size of the stage, etc.)

Would green or bluescreen make more sense? You can still do a form of virtual production with greenscreen and can previs* the shots on the stage. If the project requires heavy post-production VFX or animation, how does an LED stage balance into that equation? This should be discussed with the Visual Effects Supervisor.

* Previs (Previsualization) is using computer graphics tools to create a simplified representation of a film sequence.

Benefits

An LED stage has many benefits for filmmakers. The ability to shoot under any lighting all day long. No travel times to locations, no rain days, and full control over the scene, including the background.

The benefit of greenscreen is the ability for everyone on the set to see the scene. The actors see what they’re supposed to be looking at and the camera operator can frame for the shot. Any reflections will be correct. The LED wall aids in providing basic lighting and the dailies are able to be cut directly into the film without waiting for composites.

Pre-Production

If you’re going to do a complex LED stage shoot then the director, producers, cinematographer, and production designer must buy into and embrace the process. This is more than traditional pre-production. It has a closer resemblance to heavy visual effects or stunt production that requires decision-making from the director and planning in the early stages.

You can’t simply have someone throw together a virtual world and toss it onto the LED wall when you show up for the first day of shooting. The Virtual Production Supervisor should do an intro covering what is involved in an LED wall production, including the environment build and lighting, for the director and other keys. There are a lot of LED Virtual Production sales materials, but this should focus on the practical implications of what needs to be done and the time required.

The director and other keys should consider learning how to navigate in Unreal Engine so that they can have the freedom to explore and review without having to go through an operator.

One of the first tasks is to get an estimate of the amount of time and money the virtual environments will require to be built. This time will likely be measured in months if there are multiple complex environments to be built. The project will typically need to be delivered to the stage at least 2 weeks before the shoot to allow the stage team to vet the project and check playback speeds. Schedules should include testing and lighting days on the stage sporadically during the environment building process and before shooting. The time to build an environment is critical and needs to be protected if you wish to shoot on the scheduled stage day.

Also, make sure you have enough budget to physically dress the stage (including the ground if you plan to see the actor's feet). The budget should include the cinematographer and editor to be involved in pre-production.

Keep in mind that you will be working in this virtual world as your stage during pre-production. First to layout and refine the environment and then to do camera previs and lighting. Any attempts to shortcut this will likely cost much more time on stage. With a large LED stage going from $100,000-200,000 a day just for the stage it is indeed cost-saving to pre-plan properly.

Goal

The intent of the LED wall is to mimic what could be a real background yet done within a stage. The LED wall is more than a scenic backing. It provides a moving perspective background that exactly matches the camera and provides overall ambient lighting, especially in a circular LED wall stage. No matter what environment you’re portraying, the idea is to have it look like a real environment with a seamless transition from the physical stage to the virtual environment

One way to think of the environment is by having a stage on the salt flats and knocking out a wall. Physical set pieces are on the stage and everything out that wall is built in the virtual world (i.e. on the computer). If this is a narrative film it’s not just out that wall, you will need side views and reverse views most likely. In that case, whether you’re shooting a simple LED wall or a full enclose 360 LED stage, you will have to construct the entire area around the stage in computer graphics.

Don’t underestimate the scope of the work involved. Think how much time and money would be required to build a complete physical set on this scale from scratch, in all directions, to the horizon. In large VFX productions, some CG model environments are started in pre-production and work continues while shooting and well into post-production. This same work that might have been done in post, over quite a few months, now must be done in pre-production before the shoot if you want it on the LED wall.

VAD (Virtual Art Department) The virtual art department typically falls under the Production Designer and the art department. The VAD is responsible for building the virtual environment. This work can be done ‘in-house’ with an experienced crew or can be subcontracted out to a company specializing in this type of work.

Before construction can start, the actually LED stage should be locked in for specific dates, including testing. Testing days should be budgeted and scheduled to happen at key points during pre-production where it’s necessary to check the environment in progress on the wall.

The VAD team will want all the dimensions and other specs for the stage. How large is it, how high is it, what type of shape is it (flat wall, half circle, full circle, etc.), is there a ceiling or floor, dual camera support, the type of LED wall material and its pitch (LED size/spacing). Are there specific camera systems they are using?

There may be a gap between the bottom of the LED and the floor, or they may have a deck with a gap around the wall that will have to be accounted for when dressing the set. It will also be important to know capabilities such as high-speed filming and real-time compositing.

The stage should provide a 3D model of the entire stage. This will be placed in the virtual environment so it’s clear what is physical and what is virtual.

The stage should also provide specs for how the Unreal projects should be organized and delivered.

Storyboards

Even though previs will be generated further into the process, the director should sit down with a professional storyboard artist to document what they’re envisioning for at least major sequences and shots. This allows the director to turn their visions into a more concrete form and provides a visual starting point for the rest of the team.

Some directors prefer making shot decisions on set but doing preliminary work can save so much on-stage time to allow focus on performances and refinement rather than struggling to make the day. And for Virtual Production, it’s necessary to build the world and avoid limits on the stage.

The storyboards should give a sense of the environment the characters are in and make any camera moves clear. The cinematographer and editor should provide input on the storyboards since this is the cheapest and fastest place to make changes beyond the script. Storyboards and previs are both useful to find any shots that need to be added or changed to tell the story.

References

Reference images and videos should be gathered based on what the finished scene should look like. Documenting specific locations much as you would with a location scout is a good start. This will help ground the environment and help the artists.

Concept art

Concept artists should create color concept art for the different environments. What do we see, what’s the light and mood of these scenes? These, like the boards and previs, should all be in the same aspect ratio as the final film. There’s no point in a director falling in love with a vertical poster when the film result will be in a widescreen format.

Layout and Production Design

With the boards and concept art, the Production Designer can start laying out the world for each scene on paper. It’s best to consider it as one continuous world of the real set and the virtual environment, whether it’s an exterior or large interior. Lose the concept of working on a stage long enough to think of it as acres of free space that need to be designed and built. In the end, you want it to seem like a large environment that the actors can only walk so far in.

Where the actors are moving and where there is a convenient split point between the stage and virtual determines where the orientation and placement of the wall be relative to the set.

Be flexible. Your paper layout is the first step and will be revised and refined once it’s in the virtual space. The view from the virtual camera or a VR headset will provide more insights on the layout and objects.

This is still movie-making construction, not reality construction. You only need to build what the camera will see. If you have scenes on main street and then have a later scene in a meadow you don’t need to draw a world map nor do you have to build that into one Unreal project. Each of these scenes will be its own environment and stage setup. It may be easiest to think of these as movie locations that rely on editing and general stage direction to give us a sense of connection. (An exception to this is if you plan to render full computer-generated shots of these environments for coverage or inserts)

If you are dealing with a flat LED wall set up, then with every major change of view you can rotate the Unreal world easily but it will require a complete redress of the stage set if there is a stage set. A redress can take a long time if there are a lot of pieces on the stage. In a circular LED stage, you can cover more angles without redressing the set.

Even though changing Unreal environments is quick you have to account for loading in and dressing the stage sets (a day or more) and then removing all stage sets (another day) and loading in the next. Moving multiple platforms can make this process quicker.

Are there any shots where it is expected to see the stage set in the virtual world? Example: You have a fountain on stage and the rest of the park in virtual and then later have a shot from the hot dog vendor in the park looking back at the fountain. If your stage isn’t large enough to incorporate both, then the stage sets will have to be recreated in computer graphics so they can be shown on the LED wall.

Don’t place actors or key focus props too close to the wall since you need the wall to be slightly out of focus at all times (to avoid seeing the LED pattern and artifacts). The camera also can’t be at too shallow of an angle to the screen or there will be a moiré effect of colored lines visible to the camera.

To sell the stage and virtual world as a continuous space, try to incorporate items that live in both worlds. If you have barrels on set and matching barrels in the virtual world that will help make the environment appear seamless. That may require custom modeling or photogrammetry to provide matching props.

Design and shoot shots to take full advantage of the LED wall to provide more expansive vistas than you could get just on a stage. Avoid having large objects on the stage or in the virtual environment close to the wall that simply blocks the wall. Have some large openings where you can see further into the distance on outdoor shots to try to make the most of it.

Besides vistas try to use layers of 3D objects (real and virtual) to provide depth. By having practical objects in the foreground, actors in the midground, and objects behind the actor you create depth. Have this same notion continue in the virtual environment. All of this will help break the sense of looking like a scenic backing.

If there is any camera movement, especially lateral, the parallax re-enforces the 3D depth of the stage and background to look like one space. The objects in-depth also provide the eye with a natural fall off of focus. A sharp in focus actor with a very blurred background and no sense of foreground or partial focus background objects can look flat (screen-like).

Floor

The floor of the stage can be problematic where it meets the LED wall. As noted previously there can be a gap or height difference that needs to be addressed. In addition, the texture, details, and color of the physical flooring (carpet, grass, etc.) need to match as close as possible. The higher the camera angle the more the difference will be apparent.

Testing time on stage is required so the physical materials and the virtual materials can be selected and adjusted to match. Color regions and other controls in Unreal may have to be used to get a closer color match.

The line between the floor and the wall requires strategically placing physical props or set dressing to try to block or obscure the line where they meet. Avoid simply making a straight line of small props along the wall that would emphasize the line.

Trying to create the outdoors on a stage is very difficult to get the ground and lighting to look natural even for a non-LED shoot.

Any type of ground cover can be expensive to cover a large stage so make sure to have an adequate budget to make it look real. Some covering (like inexpensive stage ‘grass’) may appear even more fake when compared to virtual grass seen on the LED wall. The real outdoor ground is seldom perfectly flat so varying this can help sell the idea it is outside and not just a perfectly flat stage floor.

Because of the cost to cover the floor and the issues trying to blend the floor and the wall, sometimes the virtual production emphasis is more on shooting ‘cowboy’ framed shots (from above the knees) or waist up. This eliminates or reduces the floor issues. Another benefit is a possible reduction in redressing for different angles.

Previs and tech vis can be used to explore camera angles to minimize the floor or the seam showing. It can also be used to show where and how much floor covering is required.

VAD build

There are a few methods to build a virtual environment. The fastest and easiest is of course to find preexisting environments for Unreal that satisfies the needs of the film. As more virtual productions are done it’s likely there will be more virtual backlots that could be used or modified.

The next best and more likely is to find existing assets that can be combined and modified into the required scene. Unreal assets from the Unreal Marketplace have the advantage of loading right in with the correct scale, materials, and any special blueprints. But if any of these assets require a lot of modification then it may be best to get assets elsewhere. Other potential sources include Turbosquid, SketchFab, CGTrader, and others. It can be difficult to gauge quality and flexibility until the asset is purchased and downloaded. Not everything available to buy is high enough quality to use on an LED wall. Check the scale of the models and parts to make sure they are correct.

The most time-consuming and costly is to custom-build models and assets. Models are typically built and textured in other applications such as Maya, Blender, Substance Painter, etc. If the film has a very specific environment or needs to have an absolute match to a live-action set, this is likely required.

Asset wrangler Building a large environment requires a lot of assets (models, vegetation, and other items that make up the environment). There should be a point person (Asset Wrangler and backup person) to track and organize the assets for the VAD team. One of the first tasks of the VAD team is to do a search for available 3D assets that could be used. No sense in building from scratch if there is a suitable version available or a model that might be sufficient given slight adjustments. Once assets are selected, they will need to be purchased and downloaded so that the team can access them. Likely there will be added to some type of version control software. These assets should also be cataloged into a searchable database that has a thumbnail and link to the actual asset.

Photogrammetry

Photogrammetry is the process of taking numerous photos of a real object or location from all angles and having the computer align those images to make a 3D replica. That means a prop or a set could be captured as 3D and used in the virtual environment. But photogrammetry takes a lot of skilled and time-consuming handwork to make useable as a high-quality asset. If Photogrammetry is expected to be used, then it should be started as soon as pre-production allows.

Physical set in the virtual world

The VAD team will need to build virtual set pieces and props that match what is expected to be on the real stage. These can be less detailed but the correct size and shape just for blocking. Or they can be fully detailed if you want a more accurate representation of the final stage shoot. These will be used when doing previs or other previewing. If these are to be used for an actual shot, they will of course need to be higher quality.

Version Control

Before the VAD can start in earnest a Version Control system has to be set up and configured. Version control is used in software development or any process that requires several people to be working on shared assets (files). This prevents two people from trying to change the same file at the same time and it provides a place to put all new files that need to be shared. Every time a file is changed it’s assigned a new version number (think of it as a new take). That allows going back to a previous version of a file if desired.

Perforce is the software typically used for Unreal version control.

Someone experienced on the team must be the point person for Perforce who sets it up and becomes the perforce problem solver. There should be a second person for this role as a backup. They will need to assign people and determine what different streams may be required for multiple environments.

Any VAD or Previs team member will need full boot camp training on Perforce since it’s still a work in progress and has a steep learning curve for those unfamiliar with the concept. This training would include when and how to check out, add, check-in, and reconcile as well as basic Perforce etiquette. Everyone using version control should also know how to set up their machines not to upload/download unnecessary folders and how to zip an unreal project in Unreal.

Unreal Engine

Once the version control is set up then the starting environment should be created in Unreal. This should be done by an experienced user and should include expected sublevels (Landscape, Geo, lighting, stage, set dec, etc.). A config file should be adjusted to include the actual camera, aspect ratio, and lens set that is expected to be used on the project.

This should be submitted to version control as the base project to work from.

Start Building

The 3D model of the stage should be placed in the project and the LED walls should be semi-transparent to allow checking position and layout.

In Unreal start by working on the basic landscape that which the scene will be built. Using Landscape layers is useful to refine the landscape.

Keep the stage area completely flat. The area right around the stage should also remain flat for 10 feet or more to avoid any issues with the landscape being at a different level or slope relative to the real stage.

Gray boxing

Grayboxing is a term used for building a lower-quality proxy environment. This allows getting the layout and basic sizing of large objects figured out before detailing them. If you’re placing houses you might just use a cube scaled to the size of the house.

This is useful if you plan to do a lot of custom modeling so you can work out size and placement before creating detailed models. However, if your plan is to use purchased models it may be more productive to start placing them in the scene directly. Even if you’re doing a gray box approach, you should shift to the real environment as soon as possible to avoid overdesigning the gray box. At this stage don’t worry about the details or small props. You want to start with the broad strokes and refine them to more details as decisions are made.

To review this world at a minimum you want to have someone set up a CineCamera in Unreal to exactly match the camera and lenses you plan to use. Place it at virtual human height and rotate around the to check the environment. Another option is to put on a VR headset to see what it will look like on stage. It should be easy to re-arrange and refine the basic layout of the environment. Once key areas are worked out and confirmed to work in previs, then detail work can begin.

Plan to have consistent reviews of the work in progress with key creatives so feedback can be provided and acted upon.

Don’t over art direct

Because everything in the environment can be changed and adjusted it can suffer from over art directing, the same as VFX can run into. With the flexibility and control it can be tempting to go in and start tweaking and noodling endlessly- place a tree in a specific place on the mountain, rotate a trash barrel a bit more, make a sign a little more red, etc. Will this make any real difference in telling the story? Does this make it more realistic to the audience? It’s worth stepping back and judging it the same as you would a real location. Would this be acceptable as is in the film if this were all real? You want to avoid spending all your time on one section only to have to rush to complete another section.

Previs (Previsulaization)

Previs is using computer graphics to get a more accurate sense of what the shots will look like. Previs can be used for different purposes so it should clarify what is being reviewed.

Previs should start even while the environment is still being refined since it helps inform the construction.

As a starting point, virtual cameras can be set up inside Unreal that correspond to the storyboards. Having clear storyboards makes this step straightforward. Simple static human figures can be placed to represent the characters. Make sure the virtual camera and lens set to match what is planned to be used on the stage. Camera moves can be done including mimicking camera dollys and technocranes.

Lateral camera moves will emphasize the reality beyond the wall by providing a parallax, especially if there are objects in physical and virtual depth.

Note that longer lenses will tend to compress the background which can start to make the LED wall seem flattered of course.

The 3D LED stage model can be toggled on and off to understand where the stage ends and the virtual begins. This also shows you if you’re shooting beyond the edge of the LED wall.

This is a process that benefits from having the cinematographer and editor available. Rendered previs shots can be cut into sequences to determine whether there need to be any major changes or additions to the scene. Previs also provides the director and cinematographer the ability to experiment with different angles and types of camera moves. Any changes or explorations at this point are much less expensive than on the stage.

The director and cinematographer really need to use this process to get the most of their stage days and the LED wall.

Previs can become more elaborate with motion capture characters but don’t confuse this with the actor's performance. It’s still just a simplified tool for basic decisions.

Techvis is a form of previs to answer technical questions. Are there problems with the shots such as seeing off the wall or seeing the LED pattern given the distance and framing? Because the camera and lenses being used match the real camera, the exact placement and angle of the camera for each shot are already known. This can make camera setups much faster on the stage.

If the intent is to minimize the floor issues or the budget only covers covering so many square feet this can be done with colored floor sections as a reference. Then the camera position and angle can be adjusted to work for the shot but avoid unwanted areas.

The other big use for the previs is to determine which areas of the background will be seen in the film.

The areas that will be seen and are close to the wall will require the highest quality models. Areas not seen or further back can be of lower quality. This is another previs savings since the highest quality models will take the most time and cost the most to create.

Previs Lighting This virtual environment allows the cinematographer the ability to control the light of this whole world. They can choose the look of the sky by selecting 360-degree sky backgrounds (look of the clouds, color of sky, time of day), set the location of the sun, and color of light. This is where the reference photos from real locations come in handy. If this is an outdoor scene, the cinematographer should let go of the idea it will be shot on a stage. The ‘outdoor location’ is built by the art dept in its entirety and the lit virtual world will be the location with that lighting.

Additional computer graphics lights can be placed in the scene to highlight certain areas or the equivalent of flags or Cucoloris can be used to shadow areas. The cinematographer has full control over all the light in the scene by working with a lighting artist. Nighttime scenes can have a moon, stars, lit windows, neon, and wet streets if desired. The amount of haze and fog is completely controllable also. As they light the environment, they should keep in mind what they expect to do on stage and what motivates the light.

The cinematographer can work with the VAD team to set lights not only in the virtual environment but also mock up the lighting they will use on the stage. Lights in Unreal can do a good job of mimicking real movie lights – focused lights, large diffused light sources, etc. The process allows the cinematographer to paint the world with light to set their background and to pre-light and plan their stage lighting. The previs gives the cinematographer and director the ability to see what the film results will look like. The target is a complete view with matching lighting. This is another time-saving step since there’s a real lighting plan, not just a diagram, that can guide the placement of lights as well as the camera on the stage. The pre-light in Unreal won’t be an exact replica of the stage light but should serve as a good starting point.

Pre-Production Stage tests Time should be allotted on the actual stage while the environment is being built. These can be just a few hours here and there or can be whole days depending on budget and schedules. Tests are for the cinematographer and the Production Designer.

The cinematographer should test their camera and lens package on the stage to get a feel for the LED wall. How soft does the focus need to be to lose the LED pattern with different lenses and distances? How far off-axis can the camera be to the wall before moiré becomes an issue? Are there strobing issues from panning speeds? Hands-on knowledge can inform the previs and make it more accurate. The cinematographer will also want to test color response since LED walls do not have the same color fidelity as LED lights. There are some specialty lights that can tie into Unreal and provide accurate live color adjustments.

The Production Designer will want to see the environment on the wall and check the ground cover color and look, how are any major props that appear on stage and in the virtual world matching?

Additional stage tests will be required right before the shoot.

Wrapping VAD Plan to hand over the project 2 weeks before shooting. While the environment can be changed a bit on stage it’s best to review and buy off on it before getting to the stage. To get the best lighting and look the lighting in the environment will sometimes be ‘baked’. This is a process that can take several hours and if objects are moved or changed on stage, it could bring production to a standstill for part of a day.

Pre-shoot days In the days right before shooting starts the art department will need to load in and set up the stage. In addition, there will be final tests and adjustments to make sure colors are matching and any other technical issues are sorted out. After that, there should be at least one pre-light day. The cinematographer should make full use of this day and not have it become a rehearsal day. It’s important to take the knowledge learned in previs and apply it on stage. The cinematographer should work with the virtual production supervisor and stage supervisor to adjust the color and brightness of the ceiling and sidewalls to achieve the look they want.

Shooting If the cinematographer has been involved in pre-production then by the time they shoot they should have set the lighting look of the background and know where the cameras will be set up. If they pre-lit the virtual stage they can start with that on the stage.

Lighting for an LED stage: At the very least an LED wall provides a background that is already lit. Most LED stages have additional panels or even an entire 360 of walls and a ceiling. The light from a full LED stage provides the correct ambient light for the stage based on the background. This is the same process used in computer graphics. Take a 360 image of a location and wrap it around an object and it will be lit correctly and look like it’s in the environment. The LED wall itself can’t provide the harsh direct sun look so that will require additional practical lighting. That’s one of the reasons to minimize hard sun scenes on an LED stage.

The worst thing a cinematographer can do is to ignore the wall and simply light it as you would an empty stage. This is why some green or bluescreen shots stand out. Poor lighting that doesn’t match the background reference or having to shoot on stage before the background is even filmed.

In this case, the background is on stage and easily visible in real life and on the monitor to match. Take a step back from the monitor. Does that look like a real scene? A gray sphere could be placed in the virtual background close to the wall to get a better sense of the direction and lighting ratio of the ambient light if needed. A gray sphere could be used on stage for comparison if desired.

How would you light an overcast outdoor scene on location? Since you’re not fighting the hard sun you don’t have to tent the area or set overhead diffusion. The ceiling is a large controllable diffused light source so use it as needed. Bring in motivated side or rim light as desired.

Don’t block that ambient light, take advantage of it.

A cinematographer can augment the LED light with stage lighting to enhance it. Use the light in the background, which was already lit by the cinematographer and lighting artist, as the motivation. All effort should be made to make the entire scene work together, which means balancing the foreground light with the background. Same angle, color, and contrast. The cinematographer painted the background with light and now they have a chance to paint the foreground and complete the image.

The stage can change the color of the ceiling and its brightness to create a large soft light. The stage can also adjust the side walls to provide whatever color and brightness might be required.

It should be obvious that practical lighting should not hit the LED walls since it will wash out the image and destroy the illusion. So be sure to flag them off appropriately. For stages with a ceiling, the ceiling may have the same problem of light spilling onto the walls. In these cases, the ceiling may need a gradation to darken near the wall (or doughnut shape for a 360-degree stage)

Keep an eye on the black levels at the transition from stage to the wall. If there is haze or fog in the background and none or minimal on the stage, then there will be an obvious miss-match of the wall. Have a black square placed close to the wall in the virtual environment and set a black reference object or card close to the wall on the stage. Then adjust the distance or density of the Unreal fog to match the blacks.

The more haze in the background the more it will tend to flatten the image so keep that in mind when adjusting. Anytime the background is flattened a lot with haze or a long lens, care has to take to avoid the flat-screen look. Keeping haze levels consistent and even on a stage can be difficult. Priorities If pre-production and lighting previs is used as a guide, shooting should be fairly efficient. If it looks as if there will be difficult making the days then things to consider (in addition to key story beats of course) is to prioritize wide LED wall shots where you need the full wall and the stage set, followed by tighter LED wall shots and lowest priority would shot with no LED wall (against a set piece or shots looking down). These last ones could be picked up on a less expensive insert stage if you really run out of time.

Sound

One overlooked issue with LED stages is the problem of audio recording. The very flat and smooth surface reflects all sound. A 360 stage with a ceiling is worse than audio in a gym or an empty pool since sound is bounced from all over. This audio will be difficult to make it sound like it’s shot in the great outdoors. To minimize this the sound team may put up sound blankets or other barriers to absorb the sound. If these block the LED light they can be problematic for the photography.

Many thanks to the team on FATHEAD and The Third Floor and the stage which I can not name.