Digital Composite in Special Effects

INTRODUCTION

A massive spacecraft hovers over New York, throwing the entire city into shadow. A pair of lizards, sitting in the middle of a swamp, discusses their favourite beer. Dinosaurs, long extinct, live and breathe again, and the Titanic, submerged for decades, sails once more.

Usually the credit of all these fantastic visuals given to “CGI” (computer generated imagery) or “computer graphics”. Computer graphics techniques, in conjunction with a myriad of other disciplines, are commonly used for the creation of visual effects in feature films. Digital compositing is an essential part of visual effects that are everywhere in the entertainment industry today: In feature films, television commercials, and many TV shows, and it’s growing. Even a non effects film will have visual effects. Whatever will be the genre of the movie there will always be something that needs to be added or removed from the picture to tell the story. It is the short description of what visual effect are all about – adding elements to a picture that is not there, or removing something that you don’t want to be there. Digital composite plays a key role in all visual effects.

It is the digital compositor who takes these disparate elements, no matter how they were created, and blends them together artistically into a seamless, photorealistic whole. The digital compositor’s mission is to make them appear as if they were all shot together at the same time, under the same lights with the same camera, then give the shots a final artistic polish with superb color correction.

I mentioned earlier that digital compositing is growing. There are two primary reasons for this. First is the steady increase in the use of CGI for visual effects, and every CGI element needs to be composited. The second reason for the increase in digital compositing is that the compositing software and hardware technologies are also advancing on their own track, separate from CGI. This means that visual effects shots can be done faster, more cost effectively, and with higher quality. There has also been a general rise in the awareness of the film-makers in what can be done with digital compositing, which makes them more sophisticated users.

STRUCTRURE

Introduction Phase I will deal with the history and introduction of compositing. Olden compositing techniques such as optical compositing, in camera effect, background projection, hanging miniatures etc. Apart from all that I will focus on how they were creating ground breaking effects during optical era. What are the advantage and disadvantage of optical compositing?

Information hub Phase I will deal with the core concept of live action and Multipass composting with a brief introduction of stereoscopic composting. Under live action compositing I will discuss the basics and core concept of live action compositing such as rotoscopy, retouching, motion tracking with more emphasis on keying. Inside multipass compositing section simply I will focus on core concept of passes, different types of passes, use of passes. Finally a brief introduction of Stereoscopic compositing an emerging technology in the world of computer graphics.

Incredible masters Phase I will discuss upon the contribution of pioneers of this sector to develop it up to this extent and also give a brief introduction of the new technologies being used and developed.

Case study Phase which is also the last segment of my dissertation proposal I will discuss on the ground breaking effect techniques used in the Hollywood blockbusters such as Terminator, Golden compass and Finding Nemo etc.

History of compositing

In the summer of 1857, the Swedish-born photographer Oscar G. Rejlander set out to create what would prove to be the most technically complicated photograph that had ever been produced. Working at his studio in England, Rejlander selectively combined the imagery from 32 different glass negatives to produce a single, massive print. It is one of the earliest examples of what came to be known as a “combination print.”

Motion picture photography came about in the late 1800s, and the desire to be able to continue this sort of image combination drove the development of specialized hardware to expedite the process. Optical printers were built that could selectively combine multiple pieces of film, and optical compositing was born.

Introduction of Optical compositing

Not to be confused with laboratory effects done on an optical printer these use optical attachments which go in front of the lens. The intention of such apparatus is to modify the light path between subject and lens. There are many such accessories available for hire or purchase but frequently they will be constructed for a particular shot.

Techniques of Optical compositing

Glass Shot

Otherwise known as the glass painting, Hall Process or (erroneously) glass matte or matte painting, the glass shot takes the mask painted on a sheet of glass to its logical conclusion. The next stage of complexity is to make these additions to the frame representational instead of purely graphic. For example, let’s say that we have a wide shot of a farm with fields stretching off into the distance and require a silhouetted fence in the foreground. If the camera is focused on the distant hills then, with a sheet of glass positioned at the hyper focal distance (near point still in focus when focused on infinity), we can actually paint the piece of fence on to the glass. This is made possible by the two-dimensional quality of motion pictures. So long as nothing passes between the glass and the lens, and the glass is in focus, then an object painted to be the correct size for the scene when viewed through the lens will appear to be actually in that scene. Thus the silhouette of a fence painted on the glass will appear totally believable, even if a cowboy and his horse pass by in the scene beyond.

This minor change actually represents a fundamental leap in our effects capability, for now our mask has become a modification to the picture content itself rather than just an external decoration. However, once we have made this philosophical leap it is a small step to move on to creating photorealistic additions to the scene.

The next stage is to light the camera side of our glass and paint details into the image thereon. In the example of the fence we now paint in the texture of the wood and expose it as required to blend in with the scene.

Glass painting is a fundamental technique of VFX and can be applied to the latest digital equipment just as easily as it was to film prior to the First World War. Basically, if opaque paints are used (or are painted over an opaque base paint) what one is effectively doing is covering over detail in the real image with imaginary additions. This is a replacement technique and is the first of many in the VFX arsenal which permits falsification of real images.

Rotoscopy

Frequently, it comes to pass that a character or object that was not shot on bluescreen needs to be isolated for some reason, perhaps to composite something behind it or maybe give it a special color correction or other treatment. This situation requires the creation of a matte without the benefit of a bluescreen, so the matte must be rotoscoped, which means it is drawn by hand, frame by frame. This is a slow and labor-intensive solution, but is often the only solution. Even a bluescreen shot will sometimes require rotoscoping if it was not photographed well and a good matte cannot be extracted.

Virtually all compositing programs have some kind of rotoscoping capability, but some are more capable than others. There are also programs available that specialize in just rotoscoping. Each frame of the picture is put up on the monitor and the roto artist traces an outline around the character’s outer edge. These outlines are then fi lled in with white to create the familiar white matte on a black background, like the example in Figure 1-12. Large visual effects studios will have a dedicated roto department, and being a roto artist is often an entry-level position for budding new digital compositors.

There has even been a recent trend to use rotoscoping rather than bluescreen shots for isolating characters for compositing in big-effects fi lms. I say big-effects films because it is much more labor-intensive, and therefore, expensive to rotoscope a shot than to pull a bluescreen matte. The big creative advantage is that the director and cinematographer can shoot their scenes on the set and on location “naturally,” rather than having to shoot a separate bluescreen shot with the talent isolated on a bluescreen insert stage. This allows the movie’s creators to focus more on the story and cinematography rather than the special effects. But again, this is a very expensive approach.

Rotoscoping is the process of drawing a matte frame-by-frame over live action footage. Starting around the year 50 B.C. (Before Computers), the technique back then was to rear project a frame of fi lm onto a sheet of frosted glass, then trace around the target object. The process got its name from the machine that was used to do the work, called a rotoscope. Things have improved somewhat since then, and today we use computers to draw shapes using the splines we saw in Chapter 5. The difference between drawing a single shape and rotoscoping is the addition of animation. Rotoscoping entails drawing a series of shapes that follow the target object through a sequence of frames.

Rotoscoping is extremely pervasive in the world of digital compositing and is used in many visual effects shots. It is also labor intensive because it can take a great deal of time to carefully draw moving shapes around a moving target frame by frame. It is often an entry-level position in the trade and many a digital compositor has started out as a roto artist. There are some artists who fi nd rotoscoping rewarding and elect to become roto kings (or queens) in their own right. A talented roto artist is always a valued member of the visual effects team. In this chapter, we will see how rotoscoping works and develop an under – standing of the entire process. We will see how the spline-based shapes are controlled frame-by-frame to create outlines that exactly match the edges of the target object, as well as how shapes can be grouped into hierarchies to improve productivity and the quality of the animation. The sections on interpolation and keyframing describe how to get the computer to do more of the work for you, and then fi nally the solutions to the classic problems of motion blur and semi-transparency are revealed.

ABOUT ROTOSCOPING

Today, rotoscoping means drawing an animated spline-based shape over a series of digitized fi lm (or video) frames. The computer then renders the shape frame-byframe as a black and white matte, which is used for compositing or to isolate the target object for some special treatment such as color correction.

The virtue of roto is that it can be used to create a matte for any arbitrary object on any arbitrary background. It does not need to be shot on a bluescreen. In fact, roto is the last line of defense for poorly shot bluescreens in which a good matte cannot be created with a keyer. Compositing a character that was shot on an “uncontrolled” background is illustrated beginning with Figure 6-4. The bonny lass was shot on location with the original background. A roto was drawn (Figure 6-5) and used to composite the woman over a completely new background (Figure 6-7). No bluescreen was required.

There are three main downsides to roto. First, it is labor intensive. It can take hours to roto a simple shot such as the one illustrated in Figure 6-4, even assuming it is a short shot. More complex rotos and longer shots can take days, even weeks. This is hard on both schedules and budgets. The second downside to roto is that it can be diffi cult to get a high quality, convincing matte with a stable outline. If the roto artist is not careful, the edges of the roto can wobble in and out in a most unnatural, eye-catching way. The third issue is that rotos do not capture the subtle edge and transparency nuances that a well-done bluescreen shot does using a fi ne digital keyer. If the target object has a lot of very fi ne edge detail like a frizzy head of hair, the task can be downright hopeless.

SPLINES

In Chapter 5, we fi rst met the spline during the discussion of shapes. We saw how a spline was a series of curved lines connected by control points that could be used to adjust the curvature of those lines. We also used the metaphor of a piano wire to describe the stiffness and smooth curvature of the spline. Here we will take a closer look at those splines and how they are used to create outlines that can fi t any curved surface. We will also push the piano wire metaphor to the breaking point. A spline is a mathematically generated line in which the shape is controlled by adjustable control points. While there are a variety of mathematical equations that have been devised that will draw slightly different kinds of splines, they all work in the same general way. Figure 6-8 reviews the key components of a spline that we saw in Chapter 5, which consisted of the control point, the resulting spline line, and the handles that are used to adjust its shape. In Figure 6-8, the slope of the spline at the control point is being adjusted by changing the slope of the handles from position 1 to position 2 to position 3. For clarity, each of the three spline slopes is shown in a different color.

The handles can also adjust a second attribute of the spline called tension, which is shown in Figure 6-9. As the handles are shortened from position 1 to 2 to 3, the “piano wire” loses stiffness and bends more sharply around the control point. A third attribute of a spline is the angle where the two line segments meet at the control point. The angle can be an exact 180 degrees, or fl at, as shown in Figure 6-8 and Figure 6-9, which makes it a continuous line. However, a “break” in the line can be introduced like that in Figure 6-10, putting a kink in our piano wire. Figure 6-10 Adjusting angle. Figure 6-11 Translation. Figure 6-12 Mr. Tibbs. Figure 6-13 Roto spline. Figure 6-14 Finished roto. In addition to adjusting the slope, tension, and angle at each control point, the entire shape can be picked up and moved as a unit. It can be translated (moved, scaled, and rotated), taking all the control points with it. This is very useful if the target has moved in the frame, such as with a camera pan, but has not actually changed shape. Of course, in the real world it will have both moved and changed shape, so after the spline is translated to the new position, it will also have to be adjusted to the new shape.

Now let’s pull together all that we have learned about splines and how to adjust them to see how the process works over an actual picture. Our target will be the insufferable Mr. Tibbs, as shown in Figure 6-12, which provides a moving target that also changes shape frame-by-frame. Figure 6-13 shows the completed shape composed of splines with the many control points adjusted for slope, tension, and angle. The fi nished roto is shown in Figure 6-14.

One very important guideline when drawing a shape around a target object is to use as few control points as possible that will maintain the curvatures you need. This is illustrated by the shape used to roto the dapper hat in Figure 6-15, which uses an excessive number of control points. The additional points increase the amount of time it takes to create each keyframe because there are more points to adjust each frame. They also increase the chances of introducing chatter or wobble to the edges.

ARTICULATED ROTOS

Things can get messy when rotoscoping a complex moving object such as a person walking. Trying to encompass an entire character with crossing legs and swinging arms into a single shape like the one used for the cat in Figure 6-13 quickly becomes unmanageable. A better strategy is to break the roto into several separate shapes, which can then be moved and reshaped independently. Many compositing programs also allow these separate shapes to be linked into hierarchical groups where one shape is the “child” of another. When the parent shape is moved, the child shape moves with it. This creates a “skeleton” with moveable joints and segments rather like the target object. This is more effi cient than dragging every single control point individually to redefi ne the outline of the target. When the roto is a collection of jointed shapes like this, it is referred to as an articulated roto.

Figure 6-17 through Figure 6-19 illustrates a classic hierarchical setup. The shirt and lantern are separate shapes. The left and right leg shapes are “children” of the shirt, so they move when the shirt is moved. The left and right feet are children of their respective legs. The light blue lines inside the shapes show the “skeleton” of the hierarchy.

To create frame 2 (Figure 6-18), the shirt was shifted a bit, which took both of the legs and feet with it. The leg shapes were then rotated at the knee to reposition them back over the legs, and then the individual control points were touched up to complete the fi t. Similarly, each foot was rotated to its new position and the control points touched up. As a result, frame 2 was made in a fraction of the time it took to create frame 1. Frame 3 was similarly created from frame 2 by shifting and rotating the parent shape, followed by repositioning the child shapes, then touching up control points only where needed. This workfl ow essentially allows much of the work invested in the previous frame to be recycled into the next with just minor modifi cations.

There is a second, less obvious advantage to the hierarchical animation of shapes, and that is it results in a smoother and more realistic motion in the fi nished roto. If each and every control point is manually adjusted, small variations become unavoidable from frame to frame. After all, we are only human. When the animation is played at speed, the spline edges will invariably “wobulate” (wobble and fl uctuate). By translating (moving) the entire shape as a unit, the spline edges have a much smoother and more uniform motion from frame to frame.

INTERPOLATION

Time to talk temporal. Temporal, of course, refers to time. Since rotos are a frameby- frame animation, time and timing are very important. One of the breakthroughs that computers brought to rotoscoping, as we have seen, is the use of splines to defi ne a shape. How infi nitely fi ner to adjust a few control points to create a smooth line that contours perfectly around a curved edge, rather than to draw it by hand with a pencil or ink pen. The second, even bigger breakthrough is the ability of the computer to interpolate the shapes, where the shape is only defi ned on selected keyframes, and then the computer calculates the in-between (interpolated) shapes for you.

A neat example of keyframe interpolation is illustrated in Figure 6-20. For these fi ve frames, only the fi rst and last are keyframes, while the three in-between frames are interpolated by the computer. The computer compares the location of each control point in the two keyframes, then calculates a new position for them at each in-between frame so they will move smoothly from keyframe 1 to keyframe

There are two very big advantages to this interpolation process. First, the number of keyframes that the artist must create is often less than half the total number of frames in the shot. This dramatically cuts down on the labor that is required for what is a very labor-intensive job. Second, and perhaps even more important, is that when the computer interpolates between two shapes, it does so smoothly. It has none of the jitters and wobbles that a clumsy humanoid would have introduced when repositioning control points on every frame. Bottom line, computer interpolation saves time and looks better. In fact, when rotoscoping a typical character it is normal to keyframe every other frame. The interpolated frames are then checked, and only an occasional control point touch-up is applied to the in-between frames as needed.

KEYFRAMES

In the previous discussion about shape interpolation, the concept of the keyframe was introduced. There are many keyframing strategies one may use, and choosing the right one can save time and improve the quality of the fi nished roto. What follows is a description of various keyframe strategies with tips on how you might choose the right one for a given shot.

On 2’s

A classic and oft used keyframe strategy is to keyframe on 2’s, which means to make a keyframe at every other frame—that is, frame 1, 3, 5, 7, and so forth. The labor is cut in half and the computer smoothes the roto animation by interpolating nicely in between each keyframe. Of course, each interpolated frame has to be inspected and any off-target control points must be nudged into position. The type of target where keyframing on 2’s works best would be something like a walking character shown in the example in Figure 6-21. The action is fairly regular, and there are constant shape changes, so frequent keyframes are required. Figure 6-21 Keyframe on 2’s.

On shots where the action is regular but slower, it is often fruitful to try keyframing on 4’s (1, 5, 9, 13, etc.), or even 8’s (1, 9, 17, 25, etc.). The idea is to keep the keyframes on a binary number (on 2’s, on 4’s, on 8’s, etc.) for the simple reason that it ensures you will always have room for a new keyframe exactly halfway between any two existing keyframes. If you keyframe on 3’s (1, 4, 7, etc.) for example, and need to put a new keyframe between 1 and 4, the only choice is frame 2 or 3, neither of which is exactly halfway between them. If animating on 4’s (1, 5, 9, etc.) and you need to put a new keyframe between 5 and 9, frame 7 is exactly halfway between them.

Figure 6-22 shows the sequence of operations for keyframing a shot on 2’s in two passes by fi rst setting keyframes on 4’s, then in-betweening those on 2’s. Pass 1 sets a keyframe at frames 1, 5, and 9, then on a second pass the keyframes are set for frames 3 and 7. The work invested in creating keyframes 1 and 5 is partially recovered when creating the keyframe at frame 3, plus frame 3 will be smoother and more natural because the control points will be very close to where they should be and only need to be moved a small amount.

Bifurcation

Another keyframing strategy is bifurcation, which simply means to fork or divide into two. The idea is to create a keyframe at the fi rst and last frames of a shot, then go to the middle of the shot and create a keyframe halfway between them. You thengo mid-way between the fi rst keyframe and the middle keyframe and create a new keyframe there, then repeat that for the last frame and middle frame, and keep subdividing the shot by placing keyframes midway between the others until there are enough keyframes to keep the roto on target.

The situation where bifurcation makes sense is when the motion is regular and the object is not changing its shape very radically such as the sequence in Figure 6- 23. If a keyframe were fi rst placed at frame 1 and frame 10, then the roto checked mid-way at frame 5 (or frame 6, since neither one is exactly mid-way), the roto would not be very far off. Touch up a few control points there, and then jump midway between frames 1 and 5 and check frame 3. Touch up the control points and jump to frame 8, which is (approximately) mid-way between the keyframes at frame 5 and frame 10. Figure 6-24 illustrates the pattern for bifurcation keyframing.

While you may end up with keyframes every couple of frames or so, bifurcation is more effi cient than simply starting a frame 1 and keyframing on 2’s, that is, assuming the target object is suitable for this approach. This is because the computer is interpolating the frames for you, which not only puts your shape’s control points close to the target to begin with, but it also moves and pre-positions the control points for you in a way that the resulting animation will be smoother than if you tried to keyframe it yourself on 2’s. This strategy effi ciently “recycles” the work invested in each keyframe into the new in-between keyframe.

Extremes

Very often the motion is smooth but not regular, such as the gyrating airplane in Figure 6-28, which is bobbing up and down as well as banking. In this situation, a good strategy is to keyframe on the extremes of the motion. To see why, consider the airplane path plotted in Figure 6-25. The large dots on the path represent the airplane’s location at each frame of the shot. The change in spacing between the dots refl ects the change in the speed of the airplane as it maneuvers. In Figure 6-26, keyframes were thoughtlessly placed at the fi rst, middle, and last frames, represented by the large red dots. The small dots on the thin red line represent where the computer would have interpolated the rotos using those keyframes. As you can see, the interpolated frames are way off the true path of the airplane.

However, in Figure 6-27, keyframes were placed on the frames where the motion extremes occurred. Now the interpolated frames (small red dots) are much closer to the true path of the airplane. The closer the interpolation is to the target, the less work you have to do and the better the results. To fi nd the extremes of a shot, play it in a viewer so you can scrub back and forth to make a list of the frames that contain the extremes. Those frames are then used as the keyframes on the fi rst roto pass. The remainder of the shot is keyframed by using bifurcation.

Referring to a real motion sequence in Figure 6-28, the fi rst and last frames are obviously going to be extremes so they go on our list of keyframes. While looking at the airplane’s vertical motion, it appears to reach its vertical extreme on frame 3. By placing keyframes on frame 1, 3, and 10, we stand a good chance of getting a pretty close fi t when we check the interpolation at frame 7 (see Figure 6-29). If the keyframe were placed at the midpoint on frame 5 or 6, instead of the motion extreme at frame 3, the roto would be way off when the computer interpolates it at frame 3.

Final Inspection

Regardless of the keyframe strategy chosen, when the roto is completed it is time for inspection and touch-up. The basic approach is to use the matte created by the roto to set up an “inspection” version of the shot that highlights any discrepancies in the roto, then go back in and touch up those frames. After the touch-up pass, one fi nal inspection pass is made to confi rm all is well.

Figure 6-30 through Figure 6-32 illustrates a typical inspection method. The roto in Figure 6-31 was used as a mask to composite a semi-transparent red layer over the fi lm frame in Figure 6-32 to highlight any discrepancies in the roto. It shows that the roto falls short on the white bonnet at the top of the head and overshoots on the side of the face. The roto for this frame is then touched up and the inspection version is made again for one last inspection to confi rm all the fi xes and that there are no new problems. Using this red composite for inspection will probably not work well when rotoscoping a red fi re engine in front of a brick building. Feel free to modify the process and invent other inspection setups based on the color content of your personal shots.

MOTION BLUR

One of the historical shortcomings of the roto process has been the lack of motion blur. A roto naturally produces clean sharp edges as in all the examples we have seen so far, but in the real world, moving objects have some degree of motion blur where their movement has smeared their image on the fi lm or in the video. Figure 6-33 shows a rolling ball of yarn with heavy motion blur. The solution is an inner and outer spline that defi nes an inside edge that is 100% solid, and an outside edge that is 100% transparent as shown in the example in Figure 6-34. The roto program then renders the matte as 100% white from the inner spline graduating off to black at the outer spline. This produces a motion-blurred roto such as the one shown in Figure 6-35. Even if there is no apparent motion blur in the image, it is often benefi cial to gently blur the rotos before using them in a composite to soften their edges a bit, especially in fi lm work.

One problem that these inner and outer splines introduce, of course, is that they add a whole second set of spline control points to animate which increases the labor of an already labor intensive process. However, when the target object is motion blurred, there is no choice but to introduce motion blur in the roto as well. A related issue is depth of fi eld, where all or part of the target may be out of focus. The bonny lass in Figure 6-4, for example, actually has a shallow depth of fi eld so her head and her near shoulder are in focus, but her far shoulder is noticeably out of focus. One virtue of the inner and outer spline technique is that edge softness can be introduced only and exactly where it is needed so the entire roto does not need to be blurred. This was done for her roto in Figure 6-5.

SEMI-TRANSPARENCY

Another diffi cult area for rotoscoping is a semi-transparent object. The main difficulty with semi-transparent objects is that their transparency is not uniform as some areas are denser than others. The different levels of transparency in the target mean that a separate roto is required for each level. This creates two problems. The fi rst is that some method must be devised for reliably identifying each level of transparency in the target so it may be rotoscoped individually, without omission or overlap with the other regions. Second, the roto for each level of transparency must be made unique from the others in order to be useful to the compositor. A good example of these issues is the lantern being carried by our greenscreen boy. A close-up is shown in Figure 6-36. When a matte is created using a high quality digital keyer (Figure 6-37), the variable transparency of the frosted glass becomes apparent. If this object needed to be rotoscoped to preserve its transparency, we would need to create many separate roto layers, each representing a different degree of transparency. This is usually done y making each roto a different brightness; a dark gray roto for the very transparent regions, medium brightness for the medium transparency, and a bright roto for the nearly solid transparency. While it is a hideous task, I have seen it done successfully.

Motion tracking and Stabilizing

MOTION TRACKING

One of the truly wondrous things that a computer can do with moving pictures is motion tracking. The computer is pointed to a spot in the picture and then is released to track that spot frame after frame for the length of the shot. This produces tracking data that can then be used to lock another image onto that same spot and move with it. The ability to do motion tracking is endlessly useful in digital compositing and you can be assured of getting to use it often. Motion tracking can be used to track a move, a rotate, a scale, or any combination of the three. It can even track four points to be used with a corner pin.

One frequent application of motion tracking is to track a mask over a moving target. Say you have created a mask for a target object that does not move, but there is a camera move. You can draw the mask around the target on frame 1, then motion track the shot to keep the mask following the target throughout the camera move.

This is much faster and is of higher quality than rotoscoping the thing. Wire and rig removal is another very big use for motion tracking. A clean piece of the background can be motion tracked to cover up wires or a rig. Another important application is monitor screen replacement, where the four