MediaSource Extensions v0.2

Status of this Document

This document is a draft specification proposal with no official status. Send comments to Aaron Colwell. It is inappropriate to cite this document except as a work in progress.

Abstract

This proposal extends HTMLMediaElement to allow JavaScript to generate media streams for playback. Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams.

Table of Contents

1. 1. Introduction
2. 2. Media Element Extensions
3. 3. Event Summary
4. 4. Algorithms
1. 4.1 Attaching to a media element
2. 4.2 Detaching from a media element
3. 4.3 Seeking
5. 5. Examples
6. 6. Revision History

1. Introduction

Historically <audio> and <video> have only allowed media playback from container file formats. Support for use cases like live have required hacks to the container format, like 0 size segments in WebM, or required the browser to support manifest formats like HLS. Supporting new adaptive streaming proposals like 3GP-DASH also require significant browser changes and the adaptation algorithms cannot be changed once the browser is released. The file container model also puts constraints on on-demand playback. Efficient seek support requires an index to be transfered to the browser which results in either delayed startup, or delays during the first seek. All of these issues stem from the fact that the browser is expected to manage all media fetching and do so based on the information contained in a single URL.

This proposal introduces a new object called MediaSource that allows JavaScript to control how media data is fetched. This will enable significant flexibility for media fetching without requiring browser changes. Manifest formats like HLS & 3GP-DASH could be handled in JavaScript. Indexes for on-demand playback would not be needed because JavaScript could make an AJAX call to a web service to find an appropriate seek point. Adaptive streaming algorithms could be implemented and evolved over time without needing to upgrade the browser. Fetching media from multiple CDNs or splicing in media from the local file store could also be implemented without browser changes. Allowing JavaScript to control media stream construction opens may avenues for experimentation and innovation.

2. Media Element Extensions

We extend HTML media elements to allow media data to be streamed into them from JavaScript.

partial interface HTMLMediaElement {
  // URL passed to src attribute to enable the media source logic.
  readonly attribute [URL] DOMString webkitMediaSourceURL;

  bool webkitSourceAppend(in Uint8Array data);

  // end of stream status codes.
  const unsigned short EOS_NO_ERROR = 0;
  const unsigned short EOS_NETWORK_ERR = 1;
  const unsigned short EOS_DECODE_ERR = 2;

  void webkitSourceEndOfStream(in unsigned short status);

  // states
  const unsigned short SOURCE_CLOSED = 0;
  const unsigned short SOURCE_OPEN = 1;
  const unsigned short SOURCE_ENDED = 2;

  readonly attribute unsigned short webkitSourceState;
};

    

The webkitMediaSourceURL attribute returns the URL used to enable the MediaSource extension methods. To enable the MediaSource extensions on a media element, assign this URL to the src attribute.

The webkitSourceAppend(data) method must run the following steps:

1. If the first argument is null or data.byteLength is 0 then throw an INVALID_ACCESS_ERR exception and abort these steps.
2. If the webkitSourceState attribute is not in the SOURCE_OPEN state then throw an INVALID_STATE_ERR exception and abort these steps.
3. Copy contents of data into the media element's decode buffer.
4. Return true.

End of stream status values:

EOS_NO_ERROR (numeric value 0)

The stream ended normally without any errors.

EOS_NETWORK_ERR (numeric value 1)

The stream ended prematurely because of a network error. If the JavaScript code fetching media data encounters a network error it should use this status code to terminate playback. This will cause the media element's error handling code to run and the error attribute to be set to MediaError.MEDIA_ERR_NETWORK

EOS_DECODE_ERR (numeric value 2)

The stream ended prematurely because there was an error while decoding the media data. If the JavaScript code fetching media data has problems parsing the data it should use this status code to terminate playback. This will cause the media element's error handling code to run and the error attribute to be set to MediaError.MEDIA_ERR_DECODE

The webkitSourceEndOfStream(status) method must run the following steps:

1. If the webkitSourceState attribute is not in the SOURCE_OPEN state then throw an INVALID_STATE_ERR exception and abort these steps.
2. Change the webkitSourceState attribute's value to SOURCE_ENDED.

If status is set to EOS_NO_ERROR

Notify the media engine that it now has all of the media data. Playback should continue until all the media passed in via webkitSourceAppend() has been played.

If status is set to EOS_NETWORK_ERR

Run the "If the connection is interrupted, causing the user agent to give up trying to fetch the resource" section of the resource fetch algorithm

If status is set to EOS_DECODE_ERR

Run the "If the media data is corrupted" section of the resource fetch algorithm

Otherwise

Throw an INVALID_ACCESS_ERR exception.

The webkitSourceState attribute represents the state of the source. It can have the following values:

SOURCE_CLOSED (numeric value 0)

Indicates the source is not currently attached to a media element.

SOURCE_OPEN (numeric value 1)

The source has been opened by a media element.

SOURCE_ENDED (numeric value 2)

webkitSourceEndOfStream() has been called on this source.

When the media element is created webkitSourceState must be set to SOURCE_CLOSED (0).

3. Event Summary

4. Algorithms

4.1 Attaching to a media element

1. Set the src attribute on a media element or the src attribute on a <source> element associated with a media element to webkitMediaSourceURL

2. When the media element attempts the resource fetch algorithm with the URL from webkitMediaSourceURL it will take one of the following actions:

   If webkitSourceState is NOT set to SOURCE_CLOSED

   Abort media element's resource fetch algorithm and run the steps to report a MEDIA_ERR_SRC_NOT_SUPPORTED error.

   Otherwise

   1. Set webkitSourceState attribute to SOURCE_OPEN
   2. Fire a simple event named webkitsourceopen.
   3. Allow resource fetch algorithm to progress based on data passed in via webkitSourceAppend()

4.2 Detaching from a media element

The following steps are run in any case where the media element is going to transition to NETWORK_EMPTY and fire an emptied event. These steps should be run right before the transition.

1. Set webkitSourceState attribute to SOURCE_CLOSED
2. Fire a simple event named webkitsourceclose.

4.3 Seeking

1. The media element seeking algorithm starts and has reached the stage where it is about to fire the seeking event.
2. If the webkitSourceState attribute is set to SOURCE_ENDED, then set it to SOURCE_OPEN
3. The media element flushes all data from previous webkitSourceAppend() calls.
4. The media element seeking algorithm fires the seeking event and will proceed once data is passed to the media element via webkitSourceAppend().

This seeking algorithm assumes that the data passed to webkitSourceAppend() is packetized media with timestamps (ie Ogg pages or WebM clusters). This allows the "new playback position" to be determined by the timestamps in the data. If we want to support formats that are not packetized, like WAV files, then we will need a different mechanism to communicate the "new playback position" when a seek occurs.

5. Examples

Example use of MediaSource

<script>
  function onOpen(e) {
    var video = e.target;

    var headers = GetStreamHeaders();

    if (headers == null) {
      // Error fetching headers. Signal end of stream with an error.
      video.webkitSourceEndOfStream(HTMLMediaElement.EOS_NETWORK_ERR);
    }

    // Append the stream headers (ie WebM Header, Info, and Tracks elements)
    video.webkitSourceAppend(headers);

    // Append some initial media data.
    video.webkitSourceAppend(GetNextCluster());
  }

  function onSeeking(e) {
    var video = e.target;

    // Notify the cluster loading code to start fetching data at the
    // new playback position.
    SeekToClusterAt(video.currentTime);

    // Append clusters from the new playback position.
    video.webkitSourceAppend(GetNextCluster());
    video.webkitSourceAppend(GetNextCluster());
  }

  function onProgress(e) {
    var video = e.target;

    if (video.webkitSourceState == mediaSource.SOURCE_ENDED)
      return;

    // If we have run out of stream data, then signal end of stream.
    if (!HaveMoreClusters()) {
      video.webkitSourceEndOfStream(HTMLMediaElement.EOS_NO_ERROR);
      return;
    }

    video.webkitSourceAppend(GetNextCluster());
  }

  var video = document.getElementById('v');
  video.addEventListener('webkitsourceopen', onOpen);
  video.addEventListener('seeking', onSeeking);
  video.addEventListener('progress', onProgress);
</script>

<video id="v" autoplay> </video>

<script>
  var video = document.getElementById('v');
  video.src = video.webkitMediaSourceURL;
</script>
        

6. Revision History