Joe Maller: FXScript Reference: RGB and YUV Color

Up until I wrote Joe's Color Glow all my filters seemed to work fine in whatever color space they were given. But while trying to figure out the source of shimmering highlights, not-matching results and general wackiness, I noticed that all of Final Cut Pro's built-in keying filters were YUV aware. Since Color Glow is built around a keying operation, I started dissecting the built-in filters, looking for clues about when and how they used and switched between color spaces. Unfortunately there is virtually nothing documented about scripting YUV colors, so most everything on this page is based on my experiments and observations.

Diff'rent Spaces, Diff'rent Numbers

Most computer graphics people are probably familiar with RGB, and most people with a video background are familiar with YUV. Hopefully this page can help to bridge any gaps.

Final Cut Pro has three predefined color spaces constants available to scripts:

• kRGB255
• kRGB219
• kYUV219

RGB can be thought of as three grayscale images (usually referred to as channels) representing the light values of Red, Green and Blue. Combining these three channels of light produces a wide range of visible colors. Final Cut Pro contains two different RGB color spaces, kRGB219 and kRGB255. kRGB255 is the standard computer color space, the same as used by Photoshop and most other computer graphics applications, this was the original colorspace Final Cut Pro used but it has since been depreciated and replaced by kRGB219. kRGB219 is a restricted RGB color space, designed to match the YUV broadcast standard's range of acceptable values. When I first started with FXScript, the differences between RGB formats was not clear, but I've since learned that kRGB219 is the preferred working space for any function which will operate in RGB.

History of YUV (Y′CbCr)

YUV is the product of some amazing hacks. YUV uses RGB information, but it creates a black and white image (luma) from the full color image and then subtracts the three primary colors resulting in two additional signals to describe color. Combining the three signals back together results in a full color image.

The engineers who invented this needed to find a way to make color television broadcasts backwards-compatible with black and white TVs. The color signal they came up with also needed to conserve bandwidth because three channels of RGB data would not fit into the limited broadcast signal space. By combining color information, YUV uses far less bandwidth than RGB and maintained compatibility with black and white TVs.

YUV uses a matrixed combination of Red, Green and Blue to reduce the amount of information in the signal. The Y channel describes Luma (slightly different than Luminance), the range of value between light and dark. Luma is the signal seen by black and white televisions. The U (Cb) and V (Cr)channels subtract the Luminance values from Red (U) and Blue (V) to reduce the color information. These values can then be reassembled to determine the mix of Red, Green and Blue.

Some deeper research into YUV reveals two reasons why Blue always looks so crummy when extracted from video images. The U channel ranges from Red to Yellow, the V channel ranges from Blue to Yellow. Because Yellow is Red and Green, Red is essentially sent three times, Green twice and Blue only once. Reconstructing the Luminance component reveals another reason Blue suffers, the Blue channel is only 11% of Luminance. The following formula shows the weighting of each channel in the Luminance mix:

Luma = 30% Red + 59% Green + 11% Blue

The large percentage of Green and the small percentage of Blue (along with Green being sent twice) help to explain why chroma-keying for video is done against greenscreens and not bluescreens like film.

Here are a few links to more information:

• Avoiding Color Distortion on Television (MSN/WebTV Developer info)
• Two very good descriptions of how YUV works (Everything2)
• A detailed technical explanation RGB and YUV (Denny Degan)
• A brief explanation of YUV color (Stewart Fist)

It's not often fussing around with things on a computer lets me directly play with anything created before I was born. (except for transistors, binary data, QWERTY, Basic... well, I still think YUV is cool.)

Open Spaces

Any FXScript can work in any of the three color spaces by adding the following line to the definition section of the script, before the code break:

InformationFlag("YUVaware")

This tells FXScript to look for and convert between RGB and YUV pixel formats where specified.

Personal Space

FXScript uses the GetPixelFormat(image) function to return the current image buffer's pixel format. (Image buffers are discussed with FXScript Variables, Joe's Color Glow contains an example of GetPixelFormat() in use.) The three possible colorspaces are equivalent to the numeric values below:

kRGB255 = 1  kRGB219 = 2  kYUV219 = 3

Final Cut Pro's built-in filters usually define a variable to contain the format for comparison and formatting throughout the script. An example which stores the Pixel Format of the image buffer Dest in a variable called ColorSpace would look like this:

float ColorSpace;
Colorspace = GetPixelFormat(Dest)

Because the value returned is really a number, the script uses a float to define a floating point variable. While this can save a few keystrokes and clean up the code a bit, I see no reason not to use GetPixelFormat(Dest) directly. It's a bit messier to look at, but the results are immediate and accurate. The only time I store the colorspace in a variable is to allow the filter to exit gracefully. Near the beginning of the script, copy the pixelformat of dest into a variable, then at the end of the filter, check to see if the outgoing colorspace for dest is the same as the stored starting pixelformat. If it's not, convert it to match.

Trading Spaces

FXScript offers two ways of changing color spaces. Image buffers can be declared to be a certain color space or they can be converted to a specific color space.

• SetPixelFormat(image, format)

  This redefines an image buffer's contents without converting them. Each of the three color channels values will be reassigned to their matching value position in the new format. This is the equivalent of telling your dog it is now a cat. Of course it's still a dog, but now it's confused. Using this on anything but empty image buffers will result in wacky colors and unpredictable behavior.

  Image selects an image buffer. Format chooses one of the three PixelFormat constants.

• ConvertImage(srcImage, dstImage, dstFormat)

  This applies a conversion operation onto the pixels in the source image buffer and places the converted image into the destination image buffer. The destination image buffer can not be the same as the source image buffer. SrcImage is is the image buffer to be converted. DstImage is the location where the converted image buffer will be placed. DstFormat chooses one of the three PixelFormat constants.

Leaving Well Enough Alone

Adding the info flag introduces another interesting problem. Final Cut Pro doesn't check the PixelFormat before applying a conversion operation, so it's possible for an image buffer to be double-converted. Double or mis-converting an image buffer causes really strange problems and artifacting. Because of this, it's important to trap format conversions inside of conditional statements, to make sure the script won't be applying a double conversion. The following conditional mode change is being used in Joe's Levels:

if ColorSpace != kFormatYUV219;
    setpixelformat(xbuffer, kFormatYUV219)
    ConvertImage(src1, xbuffer, kFormatYUV219)
else
    xbuffer = src1
end if

The goal of that snippet is to convert RGB images back to YUV. If the images are not YUV already, the script first sets the format of xbuffer to kYUV219, then converts SRC1 to kYUV219, placing the result into xbuffer. Assigning xbuffer to kYUV219 seems to be a safeguard.

One of the built-in YUV filters seemed to constantly check to make sure it's image buffers didn't mysteriously revert to another pixel format. It's a good idea to be extra cautious when working with YUV in FXScript, finding mistakes can take a long time.

Assume Nothing

The area I've had the most trouble with is forgetting the format of an existing image buffer. Consider the following code which will work sometimes but not others:

ConvertImage(dest, xbuffer, kFormatYUV219);
dest = xbuffer;

The reason that failed, resulting in a flashing green/pink mess, was that dest was never reassigned to match the colorspace of xbuffer. Since dest is being converted to YUV219, it must be start as something else. This is the source of the problem. Copying xbuffer into dest moves image data from one colorspace into another colorspace without first converting or reassigning the receiving image buffer. In this case, dest started as RGB219, was converted to YUV and placed into xbuffer. Then xbuffer was copied back into dest, which was still an RGB219 image buffer.

The fix for this problem is fairly simple, just add a setpixelformat command to reset the colorspace of dest before copying xbuffer back in:

ConvertImage(dest, xbuffer, kFormatYUV219);
setpixelformat(dest, kFormatYUV219)
dest = xbuffer;

Manual Color Conversion

Final Cut Pro handles most of the conversions between YUV and RGB colors spaces, but these conversions are based on image buffers and not single color values. I couldn't figure out a more efficient way to change a color from RGB to YUV then converting every pixel in an image buffer. But after some searching around online, I found this page with the following formula for changing RGB color values into YUV color values:

Y = R * 0.299 + G * 0.587 + B * 0.114
U = R * -0.169 + G * -0.332 + B * 0.500 + 128;
V = R* 0.500 + G * -0.419 + B * -0.0813 + 128;

Out of several formulas I found, this one seemed to work the best. Converting three numbers has to be faster than converting every pixel in an image buffer, so there is a definite advantage to those three seemingly arbitrary lines of numbers.

Space Tools

Joe's PixelFormat Tester is an exploration filter that shows and reports on each of the pixel formats. I wrote it to help better understand the changes between YUV and RGB color spaces. The filter is a part of my free set of Exploration and Debugging filters, available with Joe's Filters.

Links to more YUV information

The following sites helped me understand what is happening with YUV color:

• The Color FAQ by Charles Poynton
• Avoiding Color Distortion on Television (MSN/WebTV Developer info)
  originally written by the late Jos Claerbout, read his original site
• Two very good descriptions of how YUV works (Everything2)
• A detailed technical explanation RGB and YUV (Denny Degan)
• A brief explanation of YUV color (Stewart Fist)

Terminology

Since writing this document, I've learned the terms "YUV" and "luminance" are largely inaccurate. Video luminary Charles Poynton makes an elegant case for proper terminology in his article YUV and luminance considered harmful.

My uses are based on what I found online to correspond with the information in Final Cut Pro's documentation and example scripts. Based on Mr Poynton's article, these terms seems inaccurate.

I've thought about the propagation of misinformation before this, and it's a serious problem. For example, there is no "sweet-zone" on your tongue, every part can taste every flavor, but that bogus theory is represented in classroom textbooks today even though it's been proven wrong for 50 years.

While this document concerns terminology associated with Final Cut Pro, these concepts extend far beyond this one program and using the correct terminology will help myself and everyone else outside of the Final Cut Pro universe.

I will continue using YUV to refer to Final Cut Pro's image buffers since the FXScript format name uses this term. Luminance refers to true CIE Luminance, while Luma describes Y′. In most cases these should be interchangeable, but they are slightly different. Before reading Mr. Poynton's article, I assumed Luma was just shorthand for Luminance. While my grasp on these terms is not 100%, I'm making an effort to use the correct terminology.

Other notes

I've noticed that most all of the color modes seem to work regardless of the color space they are given, but of course, there are exceptions. Add, Subtract and Difference freak out when applied to YUV color. The reason for this is that YUV's color channels work outward from the middle, instead of straight 0 -255 like RGB. When values are subtracted outside of the allowable YUV range, dark colors turn dark green. When pushed above the maximum allowable limit, bright colors turn pink. I've also seen RGB images exceed their color allowance and their bright pixels seem to vibrate on the video output. This RGB vibration is often caused by conversion through the kRGB255 color space, and can usually be avoided by switching to kRGB219 instead.

Conclusion

Switching between RGB and YUV isn't the nightmare I was afraid it would be. Now that I have something of an understanding of what is happening, and can recognize the wacky color symptoms of incorrect pixel formats, it's gotten easier add YUV support to my filters. Adding the Y only checkbox to Joe's Levels took less than an hour to implement, figuring out what was happening on Joe's Color Glow the first time took several days.