Preprocessing: How Real Movie Magic is Made
This chapter is from the book
This chapter is from the book
This chapter is from the book
You have your video shot, captured, and edited, and it's ready to compress, encode, and deliver. But one more step is left before encoding can begin. This step is called preprocessing, and it consists of a variety of optimizations you need to perform on the video and audio before you can hand them off to the encoder. These optimizations include deinterlacing, inverse telecining, cropping, scaling, aspect ratio adjustments, noise reduction, brightness and color corrections, and corrections to audio.
Preprocessing is almost always necessary to getting your video to look its best. The goal of preprocessing is to both clean up any noise in the video and optimize its basic elements for playback on the devices you are targeting (TVs, iPods, computers, and so on). Preprocessing is often referred to as the "magic" part of compression because it takes lots of practice to achieve the desired results. This is the most artistic, and frequently misused, part of the compression process, and it's easy to go overboard with the changes until you get a feel for how much you really need to do to your audio and video to optimize it for encoding.
Fortunately, preprocessing is a craft that can be learned, and practice will only make you better at it. Understanding why you preprocess video and audio and how the various types of optimization that occur at this stage affect your final product will help you make your preprocessing choices.
Deinterlacing Video
As discussed in Chapter 1, most digital video is interlaced when it's created because it is assumed that at some point it is meant for playback on a standard TV. This will not always be the case, and the trend toward progressive (noninterlaced) video will continue; however, for now, most of the clips you compress will come to you as interlaced video. Because interlaced video is unacceptable for Web delivery, deinterlacing video for playback on the Web or other progressive displays is a fundamental and necessary step.
If the source video is left with interlacing lines intact, the output will appear as jagged lines. The lines do not make for a good viewing experience and are also difficult to encode for a couple of reasons. Because moving objects will keep merging and reassembling (the two fields moving out of step), motion estimation will become difficult, making the encode very inefficient. The interlaced image will also have more detail than is necessary to display the image, so additional bits are wasted when relaying this redundant detail.
You can perform a deinterlace in several ways. Each is designed to optimize different types of images and different types of motion within the video. All in all, there are approximately eight ways to perform a deinterlace, though those eight have several different names by which they are recognized, depending on who you are working with.
Blend
The first common deinterlace method is referred to as blending, also known as averaging and combining fields. This method involves both fields being overlaid. This method gives you good results when there's no movement, but it results in unnatural, low-quality movements. The advantage of this approach is that it's a fast way to deinterlace and is good for low-motion scenes, such as interviews, but you will get ghosting every time an image moves.
Weave
The next commonly used method is weaving, which shows both fields in each frame. This method basically doesn't do anything to the frame, leaving you with jagged edges but with the full resolution, which can be good.
Area-based
Next is area-based deinterlacing, which blends nothing but the jagged edges. You do this by comparing frames over time or by space/position. It gives you good results in quiet scenes with little movement, because in those circumstances there is nothing to blur.
Motion Blur
The motion blur method blurs the jagged edges where needed, instead of mixing (that is, blending) them with the other field. This way, you get a more film-like look. You may have to apply Motion Blur with a program such as Apple Final Cut Pro or Adobe After Effects before using a compression application.
Discard
With the discarding method, you discard every second line (leaving the movie at half the original height) and then resize the picture during playback. Because this is the same as skipping Field 2, Field 4, Field 6, and so on, you could also call this Even Fields Only or Odd Fields Only. Although you won't get artifacts from trying to blend or merge the images, you'll lose half the resolution, and motion will become jerky.
Bob
The bob approach displays every field (so you don't lose any information) one after the other (that is, without interlacing) but with double the frames per second. Thus, each interlaced frame is split into two frames (that is, the two former fields) at half the height. Sometimes bobbing is also called progressive scanning. However, since the bob approach doesn't analyze areas or the differences between fields, the two approaches are not really the same (see the next section).
Progressive Scan
Progressive scanning analyzes the two fields and deinterlaces only the parts that need to be deinterlaced. The main difference between progressive scanning and area-based is that progressive scanning gives you a movie with twice the frames per second instead of the standard 25 fps or 30 fps movie, thus leaving you with perfect fluidity of motion. To say it more academically, it has high temporal and vertical resolution.
This method is also variously called motion-adaptive, bob and weave, and intelligent motion-adaptive.
Motion Compensation
The motion compensation method analyzes the movement of objects in a scene, while the scene consists of a lot of frames. In other words, it involves tracking each object that moves around in the scene, thus effectively analyzing a group of consecutive frames instead of just single frames.
Although effective for horizontal motion, some software for this technique does not handle vertical motion at all and may fall back on selective blending or other techniques when it is unable to resolve the motion vectors.