Proprietary Codecs, 2006

A research report published by StreamingMedia.com

Question:  Congratulations on finishing the report, Proprietary Codecs, 2006.  Who did you write if for? 

Ozer:  Thanks. When developing my test plan, I had three readers in mind; the technology decision maker who’s choosing a codec; the implementer who's shooting and encoding the footage and the technologist who’s charged with knowing the “state of the state” of streaming codecs.

Who do you mean by technology decision makers?

Anyone choosing a codec, or continuing to use a codec they’ve previously selected, which is a broad range of folks. At one end are chip and cell phone manufacturers trying to figure out which codecs to support in their equipment. The 3GPP market for video is heating up, and we saw a huge difference in quality that any vendor considering Windows Media needs to take a look at.

Content distributors, whether network or Hollywood, obviously need to know which codecs presents their shows and movies at the highest quality. Our 500 kbps profile, at 640x480 resolution, was designed for the Hollywood group, while the 300 kbps profile at 320x240, targets the CNNs and ESPNs of the world. And the test file contains many scenes relevant to this group, from sports, to ballet to talking head news shots with a streaming ticker.  

The broadest group, of course, are corporate, governmental and educational streaming producers who have to choose the codec that presents their content most effectively.  Again, with several training-oriented talking head clips, as well as the others mentioned above, the test clip is very representative.

Question:  Sounds great.  Give me the high level value proposition for this group.

Ozer:  The technologist and decision maker's needs are closely aligned. For both categories, the report objectively and specifically identifies the best codec for their content at their intended distribution rates.

I say objectively because Apple, Microsoft and Real encoded the clips used in the comparison, and I encoded the Flash clips with lots of support from On2. Having the vendors produce most of the clips obviously ensures that the quality is representative. I'm now any other codec study that uses clips encoded by the vendors.

I say specifically because the test clip included 4-8 scenes in each of five categories, business, sports, entertainment, animation and pan and zoom, encoding at 34 kbps, 80 kbps (for 3GPP), 100 and 300 kbps for LAN distribution and 500 kbps for broadband distribution. If a web publisher wants to know which codec looks best for entertainment oriented clips at 300 kbps, the report tells them. Business oriented clips at 3GPP rates? The report tells them that, too.

The test clip also provides real value. There are many studies on the web that test scenes from Matrix Reloaded, or short sequences of a pretty girl tossing her hair - not real relevant to a decision maker trying to choose the best codec for corporate training. My test clip has 38 scenes including actual business, action and entertainment clips, which are just like the scenes these decision makers and their customers are trying to publish. So they get a realistic preview of how their video will look.

Question: How did you compare codec quality?

Ozer:  I looked at four criteria. First was temporal quality, where I compared quality during real time playback, looking for motion artifacts like blockiness and banding. I also looked at playback smoothness, checking for dropped frames and jitter, and the color accuracy of the encoded clip. Finally, I assessed still frame quality, comparing frames captured from each scene in the test clip.

In my rating scheme, still frame and temporal quality were each given full weight, while smoothness and color quality received half weight. That's because most viewers don't mind a few dropped frames here and there, and small color differences are hard to pick up without side by side comparisons.

Question:  Sounds like a lot of information to wade through, especially at 117 pages. How do you make the information accessible to the readers? 

Ozer:  I knew that some readers would just want quick answers, while others would want to dig deep into our conclusions. So the first nine pages of the report summarize our findings, and that’s all some readers will ever want to know. 

If you want to dig deeper, however, you can.  For example, the section on 300 kbps comparisons itself is nine pages long. And, of course, all buyers can go see for themselves, as well.

The report has summaries like this for each data rate.

Table 31: 500 kbps Summary by Content Class. 

Plus tables within each data rate for each content class showing the results in the four rated categories.

Table 21: Entertainment clips – 100Kbps rates.

Question:  What do you mean, “see for themselves?”

Ozer:  Report buyers can download all clips used in the trials, plus all comparison screenshots I created to assess still image quality, plus all the figures embedded in the report. Or they can request a CD with the files. In both the download and CD version, a simple PowerPoint presentation guides the reader through the content, providing links to all clips and slides shows of the still image comparisons.

For example, if a publisher is interested at our findings at 300 kbps, they can click over to that page, which contains links to all four video files and a slideshow of the still image comparisons. They can use this to understand our findings and to demonstrate them within their organizations.

Boucher clip at 3GPP rates (from the business category). This is a sample of how the still frame comparisons were performed. 

Question:  Sounds useful. What does the report offer to the implementer?

Ozer:  If they’re encoding into Windows Media or Real Video formats, the report contains a section comparing the quality of constant bit rate (CBR) encoding to variable bit rate encoding (VBR) for both codecs. The results were surprising - VBR is not universally superior.

We also compared the output of Real and Windows Media files produced by three batch encoders, Autodesk Cleaner XL, Canopus ProCoder and Sorenson Squeeze, and saw a huge difference in quality. Before you plunk down a few hundred dollars to buy any of these tools, you should read our findings.

Finally, after encoding 38 test files literally hundreds of times, the benefit of using optimal lighting and shooting techniques became obvious. I detail these in a section called Producing Top Quality Streaming Video.

Question:  Your report focuses on video quality, which is only one part of the codec buying decision. Given that focus, how useful can it truly be?

Ozer:  Good question. I’ll answer by saying that while there are other buying criteria, quality is the only one your viewer actually sees and cares about. If your viewers perceive that your video quality is poor, you’ve obviously made a bad technology decision.

Question: You’ve been looking at codecs for a long time now, since 1993. What surprises did you encounter when performing your research?

Ozer:  Frankly, I was surprised at how poorly Microsoft’s Windows Media Video performed. Given that I wrote a column entitled MPEG-4 is Dead a scant 24 months ago, I was surprised at how far MPEG-4 has come. Ditto for Flash video, which used to be a joke.

I was also surprised at the difference in encoding quality produced by the batch encoders. There’s a perception that all the encoding tool does is hand off the video file to DLLs provided by the codec company, so quality is the same between the tools. That sure wasn’t the case in our tests.

Questions: You obviously encoded a range of different clips. Did you notice whether characteristics like background and lighting affect quality? 

Actually, yes, I did. In fact, the final five pages of the report is a primer on how to setup, light and shoot a scene for maximum quality. Anyone creating a set for streaming production would find this really helpful (see Taimo clip below).

Another fun section is an illustrated glossary of artifacts. Since I was tossing around words like banding, color bleeding and mosquitoes, I felt a glossary was essential to having folks understand what I meant, and I'm sure most readers will find it useful.

Question: The report costs $295, which may be steep for some buyers. What are your thoughts there?

Ozer:  A lot depends on who you are.  If you’re a cell phone or chip vendor about to make a multimillion technology decision, it’s cheap. If you’re distributing streaming content for news or entertainment, it’s also cheap, since quality is absolutely paramount.

Corporate, government and educational shops need an objective look at codec quality. Let’s face it, many technology decisions are made for the wrong reasons – did your webmaster choose Apple’s MPEG-4 because they’re diehard Apple fanatics or because the quality is truly the best? Is the “free” Windows Media really cheaper than Real if you have to boost the data rate significantly to match Real’s quality?

If you stream video inside or outside your organization, whether for sales, training, marketing or communications, your organization is judged by the quality of the video. Paying $295 to identify the best codec, encoding tools and procedures seems like a very wise investment.

Question:  I see you also have a report out on Flash codecs? What does that cover?

Ozer: With Flash, the On2 codec is obviously the best, so the report focuses on which tool produces the best quality video using that codec. I looked at 3 encoders; On Flix, Sorenson Squeeze and the Macromedia Flash 8 Video Encoder. Surprisingly, Macromedia’s encoder, which only supports CBR encoding, outperformed both Flix and Squeeze in some trials.

During my research, I spent a lot of time learning to optimize quality with all three encoders, which settings to use, which pitfalls to avoid. I detail these in the report as well, so producers using any of these tools will find lots of value.

Finally, I compared the best Flash output to Real Video, just to test some of the hype surrounding Flash, as well as Wildform, another Flash codec offered in Autodesk Cleaner. 

Proprietary Report

Flash Report