Table of Contents

• 1. Introduction
• 2. Use cases and requirements
• 3. Conformance
• 4. Dependencies
• 5. Terminology
  • 5.1 Depth map
• 6. Extensions
  • 6.1 MediaStreamConstraints dictionary
  • 6.2 MediaTrackConstraints dictionary
  • 6.3 MediaStream interface
    • 6.3.1 Implementation considerations
  • 6.4 MediaStreamTrack interface
  • 6.5 Media provider object
  • 6.6 The video element
    • 6.6.1 VideoTrack interface
  • 6.7 MediaTrackSettings dictionary
  • 6.8 WebGLRenderingContext interface
    • 6.8.1 Implementation considerations
• 7. Examples
• A. Acknowledgements
• B. References
  • B.1 Normative references
  • B.2 Informative references

1. Introduction

Depth cameras are increasingly being integrated into devices such as phones, tablets, and laptops. Depth cameras provide a depth map, which conveys the distance information between points on an object's surface and the camera. With depth information, web content and applications can be enhanced by, for example, the use of hand gestures as an input mechanism, or by creating 3D models of real-world objects that can interact and integrate with the web platform. Concrete applications of this technology include more immersive gaming experiences, more accessible 3D video conferences, and augmented reality, to name a few.

To bring depth capability to the web platform, this specification extends the MediaStream interface [GETUSERMEDIA] to enable it to also contain depth-based MediaStreamTracks. A depth-based MediaStreamTrack, referred to as a depth stream track, represents an abstraction of a stream of frames that can each be converted to objects which contain an array of pixel data, where each pixel represents the distance between the camera and the objects in the scene for that point in the array. A MediaStream object that contains one or more depth stream tracks is referred to as a depth-only stream or depth+video stream.

Depth cameras usually produce 16-bit depth values per pixel. However, neither the canvas drawing surface used to draw and manipulate 2D graphics on the web platform nor the ImageData interface used to represent image data support 16 bits per pixel. To address the issue, this specification defines a conversion into a 8-bit grayscale representation of a depth map for consumption by APIs that are limited to 8 bits per pixel.

The Media Capture Stream with Worker specification [MEDIACAPTURE-WORKER] that complements this specification enables processing of 16-bit depth values per pixel directly in a worker environment and makes the <video> and <canvas> indirection and depth-to-grayscale conversion redundant. This alternative pipeline that supports greater bit depth and does not incur the performance penalty of the indirection and conversion enables more advanced use cases.

2. Use cases and requirements

This specification attempts to address the Use Cases and Requirements for accessing depth stream from a depth camera. See also the Examples section for concrete usage examples.

3. Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MUST and MUST NOT are to be interpreted as described in [RFC2119].

This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.

Implementations that use ECMAScript to implement the APIs defined in this specification must implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [WEBIDL], as this specification uses that specification and terminology.

4. Dependencies

The MediaStreamTrack and MediaStream interfaces this specification extends are defined in [GETUSERMEDIA].

The Constraints, MediaStreamConstraints, MediaTrackSettings, and MediaTrackConstraints dictionaries this specification extends are based upon the Constrainable pattern defined in [GETUSERMEDIA].

The getUserMedia() method and the NavigatorUserMediaSuccessCallback callback are defined in [GETUSERMEDIA].

The CanvasRenderingContext2D and ImageData interfaces, CanvasImageSource typedef, and VideoTrack interface are defined in [HTML].

The ArrayBuffer and Uint16Array types are defined in [ECMASCRIPT].

5. Terminology

The term depth+video stream means a MediaStream object that contains one or more MediaStreamTrack objects of kind "depth" (depth stream track) and one or more MediaStreamTrack objects of kind "video" (video stream track).

The term depth-only stream means a MediaStream object that contains one or more MediaStreamTrack objects of kind "depth" (depth stream track) only.

The term video-only stream means a MediaStream object that contains one or more MediaStreamTrack objects of kind "video" (video stream track) only, and optionally of kind "audio".

The term depth stream track means a MediaStreamTrack object whose kind is "depth". It represents a media stream track whose source is a depth camera.

The term video stream track means a MediaStreamTrack object whose kind is "video". It represents a media stream track whose source is a video camera.

6. Extensions

partial dictionary MediaStreamConstraints {
    (boolean or MediaTrackConstraints) depth = false;
};

enum DepthMapUnit {
    "mm",
    "m"
};

partial dictionary MediaTrackConstraints {
    DepthMapUnit unit = "mm";
};

partial interface MediaStream {
    sequence<MediaStreamTrack> getDepthTracks();
};

6.6 The video element

For a video element whose assigned media provider object is a depth-only stream, the user agent MUST, for each pixel of the media data that is represented by a depth map, convert the depth map value to grayscale prior to when the video element is potentially playing.

For a video element whose assigned media provider object is a depth+video stream, the user agent MUST act as if all the MediaStreamTracks of kind "depth" were removed prior to when the video element is potentially playing.

The algorithm to convert the depth map value to grayscale, given a depth map value d, is as follows:

1. Let bit depth be the bit depth of the depth map.
2. Let near be the the near value.
3. Let far be the the far value.
4. If bit depth is greater than 8, then apply the rules to convert using range inverse to d to obtain quantized value d_8bit.
5. Otherwise, apply the rules to convert using range linear to d to obtain quantized value d_8bit.
6. Return d_8bit.

The rules to convert using range inverse are as given in the following formula:

The rules to convert using range linear are as given in the following formula:

6.6.1 VideoTrack interface

For each depth stream track in the depth-only stream, the user agent MUST create a corresponding VideoTrack as defined in [HTML].

enum RangeFormat {
    "inverse",
    "linear"
};

partial dictionary MediaTrackSettings {
    double        focalLength;
    RangeFormat   format;
    double        horizontalFieldOfView;
    double        verticalFieldOfView;
    DepthMapUnit? unit;
    double        near;
    double        far;
};