Virtual sets

In the most competitive markets, the television industry becomes an angry consumer of technology. Despite the crises, the pressure of the advertising market and the need to originate successful programming turn television producers into compulsive buyers of novel equipment. While the big filmmakers continue their furious technological career and continue to rotate teams in cycles of two or three years, those who move in smaller markets are forced to plan their investments differently. When the business moves less money the cost-benefit ratio becomes more important and sometimes the decisive factor is the cost, even if the technology has a few years in tow. In small markets, equipment procurement is based on durability and efficiency. This pragmatic trend leads some producers to almost automatically disqualify the most recent releases, the novel applications of cutting-edge technology. Before adopting the novelties, others are expected to try them. And that prices go down, eventually.

In some cases, however, this conservative trend turns against the prudent investor. For example, when competitors timely acquire some visionary product that makes their operations more productive, and the precarious balance of the market begins to be altered. Perhaps this is the case of the latest developments in the field of digital video, which promise spectacular results with investments that a few years ago would seem ridiculous.

One of the recent advances in this line is the technology of virtual studies. This is precisely the type of product that arouses distrust, especially because of its relative complexity and high cost. Even for large producers, setting up a virtual set installation is a monumental investment and its operation seems to be so distant from traditional production systems that it tends to intimidate experienced producers.

But the topic of virtual studies deserves a detailed review. The costs of study operations are very high. The maintenance of structures, decorations and lighting is so onerous as to justify the development of alternative work systems. In many cases it is preferred to undergo the inconveniences of outdoor production to avoid the complicated processes of assembly and disassembly of sets. Sometimes it is necessary to allocate a studio to a single application because it is impossible to change the sets in a short time, or store them while using the studio for a different job.

An installation of virtual sets makes the study area more productive, minimizing downtime due to project change. The production of a news program for prime time can be solved in a physical area of 80 or 90 square meters, preserving the look of the set that would require 300 or 400 square meters ... if it actually existed. A few minutes later the same study of the newscast may be originating a magazine of great visual richness. And a couple of hours later a children's program with interactive scenery could be broadcast.

Each project can have its own environment, limited only by the imagination of the creatives. The visual possibilities are truly revolutionary. And it is very possible that a more or less long term will be obtained in a favorable economic balance.

But what is a virtual studio? In principle it is a variation of the familiar blue screen technology. Image composition using chroma key procedures is a daily resource in many production operations.

However, composing realistic images is such a complicated process that in most cases it is limited to high-budget projects. Sometimes it is impossible to correctly coordinate all the elements to achieve a perfect composition. And when this is achieved the images are limited to static shots, because any movement can make it necessary to delay adjustments of the frame of the camera to maintain the illusion of reality.

Virtual studios use image composition technology to place talent within an artificial environment, which can be created from real images or computer-generated graphics. A computerized control unit is responsible for keeping the perspective of the images adjusted, making it possible for the actors to move within the set and constantly maintaining a realistic composition. In the most sophisticated systems it is possible to make talent interact with virtual elements. It is also possible to make multi-camera installations, in which a correctly adjusted composition will be achieved for all possible camera angles.

And how is this apparent wonder achieved? Below we will review the solutions offered by the main manufacturers of virtual studio systems. They all agree on the basic points, and have similar requirements. A suitable studio is needed for chroma key work, with the appropriate lighting equipment and all visible surfaces painted in a uniform color. In addition, this studio must have enough space to accommodate the displacements that the actors must really do.

Sometimes systems can be integrated with existing analog equipment. While some use ordinary chroma key procedures others use some variant of the ULTIMATTE process. Most require significant investment in graphics stations and other computer equipment, and almost all use robotic camera mounts. But despite their similarities there are basic differences that can determine which system is most suitable for a specific application.

The first solution is of an amazing naivety, and precisely for that reason the best way to present it is through an example. Suppose you want to present a large sporting event live using a set with windows to the stage. The effect you want to achieve is that the commentator presents the action that is seen through one of the windows of the set. The problem is that the set needs an area of 60 square meters, and the stadium's transmission booth is a 4-square-meter cubicle. However, the transmission cabin has a really privileged point of view. The obvious solution would be to install a temporary studio, with its windows, around the transmission cabin. This would present problems due to the differences in light between the interior and exterior of the studio, which would produce an unpleasant image.

In addition, this installation has a prohibitive cost. How can the problem be fixed?

Suppose a camera is installed in the transmission booth and its signal is transmitted by a microwave link to the station studio. In the studio you can compose an acceptable image by simulating the window using a combination of chroma key and digital effects. Each camera is correctly balanced and a homogeneous image is achieved. The problem is that you can not take shots with movements, as it would be impossible to coordinate the movement of the studio camera and the remote camera to maintain the correct perspective.

If robotic camera heads are used, it is possible to solve this problem with relative ease. The studio camera is installed in a hot head and operated by a remote control unit. This camera is connected by means of a digital link (which may well be a modem connection) to the stadium camera, in such a way that the latter reproduces all the movements that are made in the studio.

To achieve a well-adjusted image it will be necessary to perfectly synchronize the movements on all axes and also the lens settings. Obviously the initial setup will require a lot of testing to achieve a composite image with the right perspective and depth of field, and the installation will be easier to adjust if the cameras are totally identical. If you wanted to do a multi-camera installation it would be necessary to mount several camera control lines in parallel, and possibly use several microwave links simultaneously.

The end result is a composite image that could only have been achieved through a very complicated installation of limited utility. This solution can be applied to all cases where it is necessary to combine "living" images originating from different sites. It can also have application in special effects work. If the movements and adjustments that a camera used to achieve a given shot are recorded, it is possible to combine it a posteriori with a living camera while maintaining the correct perspective.

For the illusion of the virtual set to work, all camera movements must affect both the actors and the background image. In practice many of these movements are pannings, tilts and zoom settings that do not affect the perception of the perspective of the background image. These are the cases where talent appears in front of the set, and the movements are basically displacements and adjustments parallel to the plane of the action.

On such occasions, a digital effects unit can be used to manipulate the background and keep the composite image tight. The ideal is that it is a high quality DVE unit, which does not deteriorate the image much when enlarged in correspondence with the zoom settings. For this system to work, it is necessary to install the camera in a robotic head that encodes the information about camera and lens settings and transmits it to the DVE.

In this way the effects unit "follows" the movements of the camera in real time, accommodating the background to achieve a correct composition. It is even possible for the effects unit to manipulate the "focus" of the background image to achieve the effect corresponding to the variations in depth of field. The background image can originate in many ways. It can be a 3D graphic, "living" images or photographs.

As the possibilities of manipulation of the background image are limited, the realization of the shots requires a very careful planning. When it comes to using computer-generated 3D images as a background, it is necessary to make a very rigorous setup to put the camera in the correct position. However, once the initial position that delivers the correct perspective is established, it is relatively easy to maintain it through programmed movements.

In some cases, the object interaction effect can be achieved by composing the image with multiple layers. For example, if you want the character to pass behind a piece of furniture, the image of the piece of furniture can be superimposed on the composite image while maintaining the correct perspective. This involves using an additional DVE channel that is also synchronized with the camera. In the same way, additional channels of effects can be used to insert "live" video achieving video wall effects. In some cases masks can be used to achieve this, but usually the results detract from the realism of the final product.

To implement multi-camera applications with this type of solution, several systems are needed in parallel. However, it is likely that in the near future it will be possible to use a single unit of effects to manipulate the background of several cameras in productions that use only cuts.

In the systems that we reviewed above, the illusion of the real image is achieved through a careful work of alignment and synchronization between the elements that make up the shot. However, these systems offer few possibilities because in each setup only a limited repertoire of camera positions can be used. A viable alternative is the use of virtual sets created on graphics computers as three-dimensional models. Based on a coordinate system it is possible to visualize the set as it would look from any position and with any camera setting. Thanks to the software tools for graphics it is possible to apply textures and lights to achieve photographic quality graphics, with real appearance.

This technology has been developed based on the enormous computing capacity of SILICON GRAPHICS workstations, especially the Onyx RealityEngine supercomputers. These devices allow rendering of images of appreciable complexity in real time, which makes it possible to continuously generate the background images corresponding to any camera position, without limits. As a virtual environment is being generated, the set can contain any element that the designer can imagine and model. It is possible to incorporate real images, sudden changes of environment, flight effects or physically impossible positions.

As in the other systems, to coordinate the movements of the camera with the images of the set it is necessary to transmit to the graphics computer all the information about the adjustments of the lens, the focus and the zoom and the movements of the camera head. It is also necessary to accurately locate the position of the camera on the studio floor, as well as its orientation. To obtain this information, a tracking system based on printed pattern recognition is used. The camera is equipped with optical sensors that establish its position by "tracking" a series of indicators that are installed on the walls of the studio.

A control unit takes the positional information from the camera and lens and feeds the graphics station to generate the background images that will allow to achieve a perfect composition. With these systems it is possible to extend the virtual set outside the real limits of the studio, and even combine the composite images with views of the set made with "virtual cameras".

As the elements that make up the set are perfectly proportioned it is relatively easy to generate shots in which the actors interact with these elements, and the necessary masks can be generated to obtain depth effects. To ensure that the talent correctly manages the virtual space, it may be necessary to incorporate into the studio a series of guides, blocks or dummies that indicate to the actors the distribution of the spaces and the size of the objects. In many cases it is more practical to use real objects when the actor must interact with them, since the inaccuracies that can occur when talent must be understood with imaginary objects are avoided.

There are several options for deploying multi-camera applications with this type of virtual studios. Some of them may share control units or graphics stations, but if you want to achieve a fully functional system you need to mount a complete system for each camera line.

Another way to take advantage of this type of graphics is to program camera movements and generate the corresponding graphics to achieve correctly coupled sequences. Usually this formula is used for programs with repetitive movements, in which the sequence of previously recorded graphics is combined with the camera movements made "live" according to the previously established parameters. This solution is insufficient for most applications, since the filmmaker only has a limited repertoire of movements. However, it can be attractive for low-budget projects, as it requires a relatively low investment in equipment and software.

An intermediate option is to pre-record the image of the front and record the tracking information of the cameras to render the graphics later, in low-cost equipment that does not necessarily have to generate the images in real time. Later the shots are composed in post-production obtaining a result similar to that of the studios that operate "live".

A variant with respect to the previous systems uses a software-based tracking system that allows to perfectly locate the cameras, their position, orientation and adjustments without the need to use robotic heads or optical recognition systems.

To obtain the information about the cameras, a pattern of straight lines that intersect at irregular intervals is added to the walls of the study. A computer analyzes the distortions of these lines in the image delivered by the camera and "deduces" all the relevant data and with this data the graphics station is fed to obtain the background images. It is also possible to mark the dummies that are installed in the real set to identify the level of depth of the objects that correspond to each of them and to be able to apply masks automatically, without the intervention of a human operator. As the cameras do not have to use robotic heads they can be operated by cameramen, partially retaining the advantages of conventional work systems while taking advantage of virtual sets. On the other hand, by not having the restrictions imposed by the optical sensors of other systems it is possible to make "shoulder" shots that preserve a perfect perspective. The intelligent tracking system allows you to install multi-camera systems with a single graphics station, even with the possibility of making dissolves.

Despite the obvious advantages of virtual studies over traditional systems, several inherent drawbacks of each of the solutions we have examined must be taken into account.

On the one hand, most virtual sets are based on very complex installations that require a permanent process of maintenance and fine tuning to preserve their original performance. It should be considered that increasing the complexity of the technical apparatus also increases the possibility of failures. And the fear of not finding trained personnel for the operation of the equipment is fully justified.

In some cases the nature of the tracking systems or the need to preserve the orientation of the cameras makes it impossible to use them manually. Some filmmakers feel that the exclusive use of robotic heads gives their products a cardboard look. On the other hand, it can be difficult to give a conventional treatment to a set that only exists on monitors.

Consider the problem of actors. A typical example would be the case of the character who must place an object on a flat surface. The character must move fluidly in a virtual space that he does not really know. It will not always be possible to refer to a monitor to find some key. Instead of seeing the table on which he must deposit something, the actor sees a dummy, a block of plywood painted the same color of the walls and floor. Surely the study of this type of situation will develop a new school of acting technique, but while this happens it will be necessary to work with a disoriented cast.

A major technical issue is audio synchrony. In all digital video processing systems there are delays that can reach up to 5 or 6 frames, almost a quarter of a second. This can be critical for "live" applications or when some kind of talkback is required.

In many cases the design of synthetic sets falls into the commonplace, in the characteristic modes of computer graphics. This makes the programs have a science-fiction look that can often be unsuitable for a particular product. It should be remembered that the design of the sets is the work of architects, and it is not necessarily simpler because they are computer models. Sometimes you give up the elements that can give a real look to the sets in order to improve the speed of rendering or for other reasons. It is necessary to establish economies on all fronts, but we must give the aspect of the sets the importance it deserves.

By way of conclusion we can affirm that the technology of virtual studios is beginning to offer mature products, really functional and that can face the rigors of everyday work. It is foreseeable that in the medium term intermediate cost systems will appear that allow to propose investments easier to manage. In any case, they must be understood as real products with an important potential to improve the productivity of current systems for the production of television programs.