|
|
This legacy article remains on the web site for historical and research purposes. For a list of current article by Jan Ozer, please click here: |
Introduction to Streaming Multimedia
All rights reserved © Doceo Publishing, Inc.
Without question, the most significant new medium for transmitting information is the Internet. Very few digital products of any kind -- in-house or commercial, reference or entertainment - are being developed without some vision for using the 'Net for marketing or for disseminating the same data over the Internet.
Unfortunately, the multimedia elements that enrich our titles are extremely bulky, resulting in an almost audible sigh of relief when double spin (2X) CD-ROMs became standard. While Internet hype is available in unlimited supply, however, Internet bandwidth isn't, making the net quite inhospitable to multimedia data formats.
How inhospitable? Well consider that the 150 Kilobytes per second capacity of the now obsolete single spin CD-ROM drive is roughly 41 times faster than a 28.8 kilobits per second modem. That even ISDN, the digital network that represents the next bandwidth increment for most businesses and consumers, is still nine times slower than a 1X drive.
| Connection | Throughput | |
|---|---|---|
| 1X CD-ROM |
 | 150 Kilobytes per second |
| ISDN Dual Connection |
 | 16 Kilobytes per second |
| 28.8 modem |
 | 3.6 Kilobytes per second |
| 14.4 modem |
 | 1.8 Kilobytes per second |
But don't be daunted by mere statistics. Necessity is the mother of invention, and the Internet, as the next big computer marketplace, has created an urgent need for low bandwidth multimedia. This has spawned some very innovative approaches towards delivering multimedia over these narrow pipes in more or less real time.
This article overviews technologies providing video, animation, and electronic documents over the Internet. We start by examining the new concepts and buzzwords common to all formats.
Technology Primer
Internet Architecture
Let's start with a scenario, then fill in the pieces. Assume we're surfing by the White House, running Netscape Navigator. We click over to the President's area to hear a state of the union message. If everything's configured correctly, a program that looks like Media Player automatically launches and you begin to hear President Clinton's address..
As you may know, when you arrived at the White House, the Internet server downloaded information to your browser in HTML format, for Hypertext Markup Language. Even though it looked like you were pressing buttons in the White House basement, all information displayed on your screen was present in your computer.
Servers, Players and Tags
When you pressed the button "Presidents greeting," your browser called the RealAudio helper application, which sent a message to the White House server to begin sending you the audio file. After a few moments of buffering time, audio playback starts sounding a touch warbled, but quite understandable.
All Internet multimedia delivery involves these common programs and processes. The multimedia data file is stored on the Internet server, or in some cases another server connected to the Internet server. The web pages contain an HTML "tag" that references the data file and tells the server to start transmitting the data when activated. This tag structure is identical to how other data types are integrated into the web page -- if you know how to embed an icon, you can embed streaming video.
On the client side, the browser requires a helper application, generically called a player, compatible with the multimedia data format. The player must be installed and registered with the browser, or the browser will send an error message when receiving the data instructing the client to configure a player for that format or store the file to disk.
All first generation players launched and ran separately from the browser, in effect presenting the user with two programs and two interfaces. In our example, the audio player that let you stop, start and rewind President Clinton's voice ran separately from the browser, almost as if you had loaded Media Player and played the file from disk.
Netscape's Navigator 2.0 premiered an advanced level of integration called a "plug-in," which allows web developers to integrate helper applications directly into the web pages. Continuing with our example, this would let the White House webmeister integrate the audio controls directly into the page so that the user wouldn't know that a separate application was running.
To function as a Netscape 2.0 and above plug-in, the player must conform to a set of specifications available on Netscape's web site. Eager to benefit from this advanced level of integration, most leading developers of Internet delivered multimedia have elected to support the new specification.
Not to be left out, Microsoft has also proposed their own plug-in specification for their Internet Explorer browser, based upon the Object Linking and Embedding specifications, now called ActiveX. Since Internet Explorer is rapidly gaining in popularity, many developers have chosen to support ActiveX as well.
All multimedia data formats referenced in this article have at least two components, a program that creates the multimedia file, commonly called an encoder, and the helper application or player. Several formats also have "server" applications that reside on and work with the web server software. These servers provide advanced functionality during data delivery, and also to moderate data flow for greater overall server performance. We'll address the server side of the equation in the individual sections.
Data Delivery Paradigms
| Paradigm | User Experience | Technologies |
|---|---|---|
Download and Play |
Delay until file completely downloaded |
Shockwave (Director 5.0), VRML |
Streaming  |
Continuous play after short buffer period |
Streaming audio and video codecs |
Progressive Decompression |
Immediate coarse appearance, steady improvement |
Still image technologies, Adobe Amber |
Page at a Time |
Data delivered incrementally upon user demand |
Electronic documents |
Download and Play
The mode of data delivery is one of the critical implementation issues affecting both initial and continued responsiveness. The status quo, tried and true method of data delivery is called "download and play," where the entire file is downloaded before playback begins. All multimedia files can be downloaded over the Internet, stored on the client computer and then played without special programming or software. The problem with this approach is responsiveness.
For example, a one minute 11 khz/mono/8-bit audio file would take three minutes to download at 28.8 kbps, stretching the patience of most casual 'Net surfers. A one minute AVI file compressed to 150 KB/S would take 42 minutes to download, way past the attention span of your average Generation X-er.
"Streaming" Data Formats
The most common alternative for real time technologies like audio and video is called "streaming," where playback starts after a short period for data buffering. During playback, data streams from Internet server to the user, providing essentially real time play back.
Like the video compression technologies used for CD-ROM publishing, Internet streaming technologies are "codecs" that shrink bulky audio/video bit rates down to Internet bandwidths. Given the Internet's comparatively narrow bandwidth, however, compression rates are much greater than those used for CD-ROM publishing, or the data won't stream smoothly, causing noticeable breaks that interrupt audio or video playback.
Also like CD-ROM video codecs, streaming Internet codecs are all "lossy," meaning that the data thrown away during compression degrades quality. We'll discuss the impact of compression on video and audio quality in their respective sections.
Progressive Decompression
Progressive decompression is the technical term describing the now familiar effect of an image or icon that initially appears blocky on your screen but gains detail as additional information is received. Initially used only for still images, progressive decompression is now used by other formats, like Adobe's Acrobat, to provide a rough but readable image very quickly, and then improve fidelity over time.
Formats that progressively decompress are similar to streaming technologies in two aspects. First, they divide the information into sequential blocks of data for measured delivery to the user. For real time technologies like audio and video, the logical division is time, and streaming encoders package and deliver the data from start to finish. Still image technologies divide images into layers, each containing more detail, and send the layers sequentially.
The second common aspect is the player's ability to work with file segments rather than a complete file. Where most computer programs work only with complete files stored on disk, Internet players must store and display information as it's being downloaded. This is why programs like Media Player can't play streaming audio or video files as they're being downloaded, even if they could play the files from disk.
Page-At-a-Time
Page-at-a-time technologies are a hybrid approach, used primarily for electronic documents like Adobe's Acrobat and Common Ground's Digital Paper. When both Adobe and Common Ground built their first Internet players, they couldn't handle incomplete data and essentially functioned like normal computer programs, displaying the electronic document after the file was downloaded and stored to disk. This forced the user to download an entire document before viewing the first page, a decidedly unresponsive approach.
The next logical step was page-at-a-time, where the encoder divides the document into pages, transmitting only the first page to the user when activated. When the user turns the page, or hyperlinks to another location, the browser sends a message to the server to transmit the appropriate target page. Overall, this provides users with much faster access to the first document page at the cost of repeated slight delays when turning pages inside the document.
Comparing the Contenders
With this background as prolog, let's take a look at the individual technologies. Our analysis will focus on high level issues, like those to consider before incorporating one of these formats onto your web page. Once you make the decision to go forward, and start comparing alternatives, make sure you evaluate the following criteria for each reviewed technology:
Encoder - How does the encoder compare from a feature/function standpoint. For example, for audio delivery, does it allow batch compression of multiple files, or real time compression from a live source. Also, what environments does it run on, what types of inputs does it support and how simple is the program to operate?
Player - Feature/function comparison, with emphasis on player environments, like Macintosh and UNIX support, since this defines the limits of your audience. Also integration issues like whether the player supports the Netscape 2.0 Plug-in format or Microsoft's Active X.
Format Quality - Comparing output side by side and analyzing which technology produces higher quality. For example, at similar audio bandwidths, which audio format sounds better?
Server - If there is a server, what environments does it support -- a Windows NT server can't run on a Sun web server. Also, a feature/function analysis, comparing performance and feature set.
Price - Finally, how does the technology compare price wise with others in its class, considering prices for all of the above components.
Internet Multimedia Technologies
Streaming Audio
At the risk of gushing (bad form for technology writers), streaming audio is here, it's now, it's today. Despite the 150:1 compression ratios required to squeeze audio down to below 14.4 kbps, quality is acceptable for many applications.
Note that speech is much less complex than music and compresses much more effectively, meaning less distortion. Still, if judged only by the quantity of music available on the Internet, audio quality is sufficient for the casual surfer, if not the dedicated audiophile.
At the time of this writing, five companies produced streaming audio technologies; DSP Group with TrueSpeech, RealAudio from Progressive Networks, Internet Wave from VocalTec, Toolvox for the Web by Voxware and Streamworks from Xing Technologies.
Three of the technologies, TrueSpeech, Internet Wave and Toolvox were free, but business models change quickly on the Internet so be sure to check early in your decision making process. Xing and Progressive Networks both charge for their server components, and Xing charges for their real time encoder.
The first point on your streaming audio decision tree is the number of simultaneous streams you want your server to support. Xing and Progressive Networks have server programs that manage the outflow of data more efficiently than the web server software itself, allowing more simultaneous streams. These servers also provide advanced features like random access to the audio stream, pause and fast forward, all useful when playing long audio files.
What's the magic number beyond which you need server software? The DSP Group reports satisfying over 50 simultaneous streams without problem, but the server was a Silicon Graphics workstation. On a Pentium box, with sufficient outgoing bandwidth, it's probably safe to assume that most technologies can support up to five or six simultaneous streams. Of course, you can always try the free technologies and upgrade to RealAudio/Streamworks if performance is unacceptable.
The next consideration is playback feature set, where Progressive Networks dominates, not only by providing free real time encoding, but also with an advanced Application Programming Interface (API) that functions almost like an authoring system. For example, the you can trigger page changes at specific points in the audio stream, letting you build a cost effective kiosk on the 'net.
Xing also scores high on advanced features, with proven, industrial strength real time encoding and a standalone player that works almost like a car radio, letting you configure buttons to switch URL's at one touch.
Another key consideration is helper application. Support for Netscape's Plug-in specification and Microsoft's ActiveX are obviously important, as are functionality and appearance.
Most players include status bar and volume control. As mentioned above, Server based technologies also offer random access to various points on the audio stream and pause and resume.
Helper appearance is key when you chose not to embed the player. Where DSP Group, Progressive Networks and Voxware created simple, Media Player-like interfaces, VocalTec built an engaging, less buttoned down player with a boom-box toting character reflecting player status. The character plugs the box into the wall when you select a file, dances happily when audio is playing and jumps up and down on the box when playback is interrupted. This interface would be very desirable if distributing music to Generation X-ers.
Moving down the decision tree to audio quality, let's start with simple rules. First, dismiss Voxware if your page contains content other than simple speech, as Voxware handles speech incredibly well, but mangles music beyond recognition. Contrary to its name, TrueSpeech works well with music at low bandwidths, but compresses to only one bandwidth -- 14.4 bps -- so it can't scale quality for intranets with higher bandwidths.
Streaming video
Streaming video is probably the most technically and artistically challenging multimedia format. Publishers currently working with video are familiar with the production, lighting, filming and other issues related to analog video, and the capture and compression issues relating to digitization. On the Internet, add the fact that your bandwidth suddenly shrinks from 150 KB/s to as low as 14.4, including audio.
As a result, even at 28.8 kbps, video quality is much, much lower than even the 160x120 video that graced many of our early CD-ROMs. On the other hand, at 128 kbps, double line ISDN or simply a compact data stream for intranets, quality looks surprisingly good.
At the time of this writing, there are five streaming video technologies, VDOLive from VDONet Corporation, VivoActive from Vivo Software, ClearFusion from Iterated Systems, Streamworks from Xing and Web Theatre from VXtreme, Inc. Unlike streaming audio technologies which are fairly homogenous, the two video technologies are striking in their key contrasts.
Streaming video technologies have all the same concerns as streaming audio. One unique consideration is the technology's "scaleability," or ability to throttle video quality to maintain audio unbroken audio playback. Xing and VDOLive are both scaleable, and maintain audio playback even when bandwidth becomes limited.
In most instances, Xing, which uses MPEG-1 based technology, preserves audio by slowing or ceasing to deliver video frames, and at 28.8 kbps frames update as slow as one every five or six seconds. In contrast, using its scaleable wavelet technology, VDO first drops the quality of the individual frames, then slows the frame rate, stopping video delivery, like Xing, if required to preserve the audio transmission. The obvious victim here is video quality, either as a result of dropped frame rate, a la Xing, or degraded frame quality, courtesy of VDO.
Vivo, which uses H.261 technology, and ClearFusion, based on Iterated's patented fractal algorithms, have no scaleability mechanism, so when bandwidth drops, audio and video playback simply stop, restarting when bandwidth allows. This works well over high bandwidth Intranets, less well over the Internet if you prioritize audio over video quality. Those who favor video quality, however, like the fact that video quality never drops, per se, it just starts and stops a bit.
The one caution about streaming video is that of all the multimedia technologies, it is most adversely effected by the 'Net's challenging data delivery environment. This means that for the most part, the frame per second rating quoted by each manufacturer - e.g. 15 fps @ 160x120@ 28.8 kbps - is optimistic at best, and probably unattainable in anything other than a laboratory environment.
Electronic Documents
While HTML has come a long way since its inception, viewed and printed output quality doesn't approach what you can achieve with a good word processor like Lotus Word Pro. This is obviously frustrating to those who spend hours (or thousands of dollars) beautifying their brochures and product specifications.
One alternative is electronic documents, like those produced by Adobe Acrobat or Common Ground Software's Digital Paper. These documents can contain both text and images, and once created, can be viewed and printed on a multitude of platforms, while looking virtually identical to their original form.
Electronic documents can also contain internal and even external hyperlinks, and are search-able, both key advantage over HTML or native word processing documents. Long a favorite for CD-ROM publishers, neither format significantly penetrated the Internet because of two significant disadvantages.
First was the fact that both companies used the download and play metaphors, so the entire document had to be downloaded before viewing even the first page. A ten page document could easily be 100-250 KB/s and take 30 to 90 seconds to download. This would be acceptable if you knew you wanted to view and print, but not if you're just window shopping.
The second objection to portable documents was integration, since before the Netscape 2.0 plug-in there was no way to integrate portable documents into a web page. The forced the user to spawn a second a full screen player which was slow and unwieldy. In addition, while hyper-linking within an electronic document was possible, linking to outside URLs was not. Unlike the eminently linkable and viewable HTML, electronic documents were like inert lumps of information, useful perhaps, but clearly a foreign object.
Both Adobe and Common Ground have dramatically improved web functionality over the last twelve months. Both support the Netscape 2.0 plug-in specification, so the formats can be seamlessly integrated into a web page. Both are also moving towards page at a time paradigms to improve interactivity.
Common Ground has an early lead over Adobe in this regard, having shipped their "Page-on-Demand" technology in October 1995. Essentially, this breaks the document into multiple pages, and transmits pages to the client as requested. For example, page 1 of the document downloads and displayed immediately. If the user selects the next page, page 2 is downloaded. However, if the user selected a hyperlink to page 15, this page would be loaded, saving the interim load time of pages 2-14.
In December , 1995, Adobe announced and shipped the initial components of their web-enhanced Acrobat software, code-named Amber, which ultimately shipped in the summer of 1996 as Acrobat 3.0. In addition to page on demand technology, Adobe also implemented a progressive rendering, which sends text first, then graphics, then final font and sizing instructions.
By way of background, Adobe uses PDF files, which stands for Page Description Format. Not surprisingly, the format is based on Adobe's crown jewel, the Postscript specification. Acrobat 3.0, which retails for $295, includes the three utilities necessary to create PDF files from word processors and similar applications (Exchange), convert them from standalone EPS files (Distiller) or to scan them from hard copy (Capture). To implement Adobe's page on demand technology, your Internet server must support "byte range serving," either directly or through a CGI (Common Gateway Interface) script. You also need the Adobe 3.0 plug-in, which you can download from the Adobe web site.
Common Ground uses the DP format, for Digital Paper, based upon their own proprietary format. Their Common Ground application ($185) creates .dp files, and a server program called the Common Ground Web Server Tools ($995) is required to implement their page-on-demand system.
For both technologies, creating electronic documents is as simple as printing. When you install Exchange or Common Ground, the programs add a printer driver that converts print output to their respective formats. This driver contains controls for image compression, embedding fonts and other characteristics.
After finishing the document in your word processor or publishing program, you select the correct driver and print. The software prompts you for a file name and a few seconds later you have your file.
The next step is to load the file into the respective programs and create the inter and intra-document links. At this stage, you can also add security features protecting your data from copying, editing or even printing, if so desired.
Figure 4.8 - creating a link to an external URL from within
Common Ground
Both companies take an extra step to prepare their files for the web. Adobe Exchange "linearizes" the document, placing all assets like fonts and graphics in the most efficient order and compressing the result using a mix and match of lossless technologies like LZW, Group III and IV facsimile and RLE, and lossy JPEG for graphics. This process also catalogs all internal hyperlinks by byte range so that the player can request a specific page directly.
Common Ground's Web Server System is a bit more complicated, cataloguing all documents and creates all HTML pages necessary to view the document. In both cases, programming is not required.
Once you're set up, 'Net surfers requesting a document prepared in either format from a web page will receive one page, which Adobe will display progressively and Common Ground will display when fully received. Thereafter, touching any hyperlink in either format will launch the retrieval of the corresponding page. While waiting for each page to download may feel a touch unresponsive, it's tons better than waiting for a 500 MB document to download before you can even tell if you really want it.
Both Adobe and Common Ground freely distribute their players over the Internet, so your documents can be downloaded, viewed and printed without charge to the user. Adobe has taken the lead on the plug-in side, as the Acrobat 3.0 player is fully embeddable under both Netscape 2.0 and Internet Explorer 3.0, in fact defaulting to embedded mode even when displaying pre-Version 3.0 documents. Common Ground's player isn't embeddable, and in fact doesn't even install itself as a helper application, a significant issue for those selling to non-technical users.
A note to Acrobat users. If you've got PDF files up on your web site, you'll need to download the new version of Exchange to linearize your files and register their byte order for page at a time download. Otherwise, your viewers won't be able to move from page to page without downloading the entire file.
With the HTML 2.0 specifications recently finalized, the industry is moving towards 3.0 definitions, and two camps are emerging. On one side is Adobe, Netscape and Apple, pushing the Postscript-based standard. On the other is Microsoft, once again hoping to break through Adobe's format monopoly with a TrueType-based standard.
Whatever the result, the advancement of HTML allows web developers to improve the quality of their video both onscreen and during printing. However, neither spec will be in place in the short term, and it's unclear whether either will deliver as much quality as Common Ground or Acrobat.
While the situation is in flux, Web architects should consider portable documents for two different purposes. First is for integrating more elaborate text and formatting into their pages using the Netscape 2.0 plug-in specification.
Those who chose not to integrate should still consider distributing specifications, press releases and other product literature from their web site in Acrobat or Common Ground format. Remember that you're counting on this literature to sell your product or services, and while it's an unfair fact of life, sometimes the best looking document wins.
Shockwave and VRML
One of the most visible multimedia on the web developments has been Macromedia's Shockwave for Director. Director is an extremely flexible and capable hybrid used by many CD-ROM developers as both a standalone authoring program and to create animations embedded into other authoring programs. Shockwave is a player or helper application that lets Internet clients play Director files from within their browsers.
As it exists today, Shockwave is a download and play technology rather than a streaming technology. In most instances, this limits its use to small files in the "spinning logo" category, interesting appetizers but not a substantial main course.
This summer, Macromedia shipped Shockwave for Director 5.0, which supports streaming technologies like VDOLive and RealAudio, but will not stream itself. Developers can now create a Director 5.0 front end for RealAudio playback, download the file to the client and then initiate audio playback of the Real Audio stream.
By the end of 1996, Macromedia promises a post processing program called Afterburner that will arrange the bitmapped graphics, video and audio assets in a Director file to enable streaming. It will be interesting to see exactly how the company manages the trade-offs necessary to accomplish full streaming.
At its essence, streaming technology implies the ability to maintain some measure of synchronization, usually with audio. Without sound, for example, you couldn't tell whether slow motion television was an instant replay or some snafu at the station. Similarly, you couldn't tell if one frame per second animation was relaxation therapy or a bandwidth problem unless the audio was breaking up or clearly out of synch.
A streaming Director file means that the animated sequences will maintain synchronization with the audio. Note that the two key streaming technologies, audio and video, have established strategies for maintaining synchronization, all of which trade bandwidth for quality and degrade the overall experience to maintain synchronization. For example, audio and video codecs use "lossy" compression which shrinks the stream but degrades quality. Video codecs also shrink the viewing resolution and drop frames to maintain synch.
In the "there ain't no such thing as a free lunch" Internet environment, Macromedia will have to employ some or all of these technologies. Let's assume a simple case, where a full screen 24-bit bitmap (768 KB) changes every four seconds with accompanying audio commentary. This would be a piece of cake for a dual speed CD-ROM, but impossible over a 28.8 kbps connection unless you degrade image quality beyond recognition.
This raises two key issues. First, will there be some degradation trade-off to maintain synchronization? Second, with or without this trade-off, will there be development tools like bandwidth monitors to help developers create Director files that stream at bandwidths relevant to the developers?
One thing not in question is Macromedia's commitment to Shockwave and streaming. At press time, the company announced a Shockwave component for their popular Authorware product which will operate similarly to Shockwave for Director.
While not quite as loud, the Internet has also been buzzing about VRML (Virtual Reality Modeling Language), a language that enables 3D graphics over the web. Like Shockwave, VRML files are download and play technologies. Open a .VR file, and you'll see structured text instructions that list the audio and video files manipulated in the 3D sequence.
For example, one instruction might tell the computer to draw a triangle in the upper left hand corner and texture it with a certain jpg bitmap. Another might start a certain audio file playing three seconds after animation starts.
This mode of operation means that all the required assets must be present on the client computer before a VRML file can start playing. Absent an initiative like Shockwave and Afterburner, VRML will remain a download and play technology.
Summary
1. Multimedia on the Internet is just another phenomena that looks easier than it is. At bandwidths up to 40 times smaller than even an obsolete CD-ROM, there will be trade-offs for every multimedia format.
2. Netscape's Plug-in specification and Microsoft's ActiveX have enable the seamless integration of HTML and multimedia formats like audio and video.
3. There are four data delivery paradigms: Download and play, where you wait for the entire file to download before anything happens; streaming, where playback starts soon after calling for the file, and continues (in theory) unbroken until the transfer completes; progressive decompression, where a very rough image appears very quickly with additional detail filling in over time, and: Page at a time, where the first page of a document is transferred immediately, and subsequent pages transferred upon demand.
4. All Internet multimedia formats have at least two if not all three of the following components: Encoder, that compresses the material into final format for placement on the server; player, that resides at the client to decode the compressed stream; Server, usually to dole out the compressed bandwidth or enable special features like fast forward or page at a time.
5. Audio is the most mature of the streaming technologies, with good alternatives at all price ranges, including free.
6. Streaming video is still in its infancy. Over the bandwidth limited Internet, it's suitable primarily for Generation-X'ers and those seeking technology demonstrations. On intranets, however, the quality of a 128 kbps talking head video stream is remarkably high.
7. If your home page has brochures, spec sheets, pamphlets and other marketing or sales materials, HTML probably can't duplicate the quality of your normal hard copy output. Electronic documents, however, like Adobe Acrobat, can, and also perform remarkably well in the web environment.
8. Shockwave is currently a download and play technology best suited for very short movies of the spinning logo genre. If Afterburner succeeds in converting Shockwave to a streaming technology, however, hundreds of thousands of rabid Director authors will be set loose on our Internet, and things may never be the same.
