MediaSource API

Status of this Document

This document is a draft specification proposal with no official status. Send comments to Aaron Colwell. It is inappropriate to cite this document except as a work in progress.

Abstract

This proposal introduces a MediaSource object which allows JavaScript to generate media streams for playback in HTMLMediaElements. Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams.

Table of Contents

1. 1. Introduction
2. 2. MediaSource Interface
3. 3. Event Summary
4. 4. Algorithms
1. 4.1 Attaching to a media element
2. 4.2 Detaching from a media element
3. 4.3 Waiting Timeout
4. 4.4 Seeking
5. 5. Examples

1. Introduction

Historically <audio> and <video> have only allowed media playback from container file formats. Support for use cases like live have required hacks to the container format, like 0 size segments in WebM, or required the browser to support manifest formats like HLS. Supporting new adaptive streaming proposals like 3GP-DASH also require significant browser changes and the adaptation algorithms cannot be changed once the browser is released. The file container model also puts constraints on on-demand playback. Efficient seek support requires an index to be transfered to the browser which results in either delayed startup, or delays during the first seek. All of these issues stem from the fact that the browser is expected to manage all media fetching and do so based on the information contained in a single URL.

This proposal introduces a new object called MediaSource that allows JavaScript to control how media data is fetched. This will enable significant flexibility for media fetching without requiring browser changes. Manifest formats like HLS & 3GP-DASH could be handled in JavaScript. Indexes for on-demand playback would not be needed because JavaScript could make an AJAX call to a web service to find an appropriate seek point. Adaptive streaming algorithms could be implemented and evolved over time without needing to upgrade the browser. Fetching media from multiple CDNs or splicing in media from the local file store could also be implemented without browser changes. Allowing JavaScript to control media stream construction opens may avenues for experimentation and innovation.

2. MediaSource Interface

This interface provides a way to stream data to a media element.

[Constructor(in DOMString type)]
interface MediaSource {
  readonly attribute DOMString type;

  // states
  const unsigned short CLOSED = 0;
  const unsigned short OPEN = 1;
  const unsigned short ENDED = 2;

  readonly attribute unsigned short readyState;
  
  readonly attribute unsigned long bytesAvailable;

  bool append(in ArrayBuffer data);
  void endOfStream();

  // event handler attributes
  attribute Function onopen;
  attribute Function onwaiting;
  attribute Function onflush;
  attribute Function onprogress;
  attribute Function onended;
  attribute Function onclose;
};
MediaSource implements EventTarget;

    

The MediaSource(type) constructor takes a single argument. This argument, type, specifies the mimetype for data that will be streamed to the media element. This is similar to the type attribute on a <source> element. If type is a null or empty string a SYNTAX_ERR exception will be thrown.

The type attribute returns the type value passed into the constructor.

The readyState attribute represents the state of the source. It can have the following values:

CLOSED (numeric value 0)

Indicates the source is not currently attached to a media element

OPEN (numeric value 1)

The source has been opened by a media element.

ENDED (numeric value 2)

endOfStream() has been called on this source. No further append() calls are allowed in this state.

When the object is created readyState must be set to CLOSED (0).

The bytesAvailable attribute indicates the current number of bytes available in the source buffer. append() will fail if called with an ArrayBuffer larger than this value.

The append(data) method must run the following steps:

1. If the first argument is null or data.byteLength is 0 then throw an INVALID_ACCESS_ERR exception and abort these steps.
2. If the readyState attribute is not in the OPEN state then throw an INVALID_STATE_ERR exception and abort these steps.

3. Run the first matching steps from the following list:

   If data.byteLength is greater than the bytesAvailable attribute

   Return false.

   Otherwise

   Copy contents of data into the media element's decode buffer.

   Decrement bytesAvailable attribute by data.byteLength.

   Return true.

The endOfStream() method must run the following steps:

1. If the readyState attribute is not in the OPEN state then throw an INVALID_STATE_ERR exception and abort these steps.
2. Change the readyState attribute's value to ENDED.
3. Fire a simple event named ended at the source.
4. Notify the media element that the end of stream has been reached.

3. Event Summary

4. Algorithms

4.1 Attaching to a media element

1. Create a new MediaSource object with new
2. Create a URL for the source using URI.createObjectURL()
3. Set the src attribute on a media element or the src attribute on a <source> element associated with a media element.

4. When the media element attempts the resource fetch algorithm with the URL for the source object it will take one of the following actions:

   If the source's readyState is NOT set to CLOSED

   Abort media element's resource fetch algorithm and run the steps to report a MEDIA_ERR_SRC_NOT_SUPPORTED error.

   If the source's type attribute contains a mimetype that is not supported by the media element.

   Abort media element's resource fetch algorithm with an error.

   Otherwise

   1. Set the source's readyState attribute to OPEN
   2. Fire a simple event named open at the source.
   3. Start a timer to call the waiting timeout algorithm after 350ms.
   4. Allow resource fetch algorithm to progress based on data passed into the source via append()

4.2 Detaching from a media element

The follwing steps are run in any case where the media element is going to transition to NETWORK_EMPTY and fire an emptied event. These steps should be run right before the transition.

1. Set the source's readyState attribute to CLOSED
2. Fire a simple event named close at the source.

4.3 Waiting Timeout

1. If the media element has received data from append() and still has data to decode, abort these steps.
2. Fire a simple event named waiting at the source.

4.4 Seeking

1. The media element seeking algorithm starts and has reached the stage where it has fired the seeking event on the media element
2. If the source's readyState attribute is set to ENDED, then set it to OPEN
3. Start a timer to call the waiting timeout algorithm after 350ms.
4. Fire a simple event named flush at the source.
5. The media element seeking algorithm will proceed once data is passed to the media element via append().

This seeking algorithm assumes that the data passed to the MediaSource is packetized media with timestamps (ie Ogg pages or WebM clusters). This allows the "new playback position" to be determined by the timestamps in the data. If we want to support formats that are not packetized, like WAV files, then we will need a different mechanism to communicate the "new playback position" when a seek occurs.

5. Examples

Example use of MediaSource

<script>
  function onOpen(e) {
    var mediaSource = e.target;

    // Append the stream headers (ie WebM Header, Info, and Tracks elements)
    mediaSource.append(GetStreamHeaders());

    // Append some initial media data.
    mediaSource.append(GetNextCluster());
  }

  function onFlush(e) {
    var mediaSource = e.target;
    var video = document.getElementById('v');

    // Notify the cluster loading code to start fetching data at the 
    // new playback position.
    SeekToClusterAt(video.currentTime);

    // Append clusters from the new playback position.
    mediaSource.append(GetNextCluster());
    mediaSource.append(GetNextCluster());
  }

  function onProgress(e) {
    var mediaSource = e.target;

    if (mediaSource.readyState == mediaSource.ENDED)
      return;

    // If we have run out of stream data, then signal end of stream.
    if (!HaveMoreClusters()) {
      mediaSource.endOfStream();
      return;
    }

    // Is the next cluster larger than available buffer space?
    if (PeekNextCluster().length > mediaSource.bytesAvailable)
      return;

    mediaSource.append(GetNextCluster());
  }

  mediaSource = new MediaSource('video/webm; codecs="vp8, vorbis"');
  mediaSource.addEventListener('open', onOpen);
  mediaSource.addEventListener('flush', onFlush);
  mediaSource.addEventListener('progress', onProgress);
</script>

<video id="v" autoplay> </video>

<script>
  document.getElementById('v').src = URI.createObject(mediaSource);
</script>