This specification extends the Media Capture and Streams specification [GETUSERMEDIA] to allow a depth-only stream or combined depth+video
stream to be requested from the web platform using APIs familiar to web authors.
Status of This Document
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This extensions specification defines a new media type and constrainable property per Extensibility guidelines of the Media Capture and Streams specification [
GETUSERMEDIA]. Horizontal reviews and feedback from early implementations of this specification are encouraged.
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
Depth cameras are increasingly being integrated into devices such as phones, tablets, and laptops. Depth cameras provide a depth map, which conveys the distance information between points on an object's surface and the camera. With depth information, web content and applications can be enhanced by, for example, the use of hand gestures as an input mechanism, or by creating 3D models of real-world objects that can interact and integrate with the web platform. Concrete applications of this technology include more immersive gaming experiences, more accessible 3D video conferences, and augmented reality, to name a few.
To bring depth capability to the web platform, this specification
extends the MediaStream interface [GETUSERMEDIA] to enable it to also contain depth-based
MediaStreamTracks. A depth-based
MediaStreamTrack, referred to as a depth stream
track, represents an abstraction of a stream of frames that can each be converted to objects which contain an array of pixel data, where each pixel represents the distance between the camera and the objects in the scene for that point in the array. A
MediaStream object that contains one or more
depth stream tracks is referred to as a depth-only stream or depth+video stream.
Depth cameras usually produce 16-bit depth values per pixel, so this specification defines a 16-bit grayscale representation of a depth
map.
2.
Use cases and requirements
This specification attempts to address the Use
Cases and Requirements for accessing depth stream from a depth camera. See also the
Examples section for concrete usage examples.
3. Conformance
As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.
The key words MUST and MUST NOT are to be interpreted as described in [RFC2119].
This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.
Implementations that use ECMAScript to implement the APIs defined in this specification must implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [WEBIDL], as this specification uses that specification and terminology.
The term video-only stream means a MediaStream object that contains one or more MediaStreamTrack objects of kind "
video" (video stream track) only, and optionally of kind "audio".
The term depth stream track means a MediaStreamTrack object whose kind is "depth". It represents a media stream track whose source is a depth camera.
The term video stream track means a MediaStreamTrack object whose kind is "video". It represents a media stream track whose source is a video camera.
5.1
Depth map
A depth map is an abstract representation of a frame of a
depth stream track. A depth map is an image that contains information relating to the distance of the surfaces of scene objects from a viewpoint.
A depth map has an associated near value which is a double. It represents the minimum range in meters.
A depth map has an associated far value which is a double. It represents the maximum range in meters.
A depth map has an associated focal length which is a double. It represents the focal length of the camera in millimeters.
A depth map has an associated horizontal field of
view which is a double. It represents the horizontal angle of view in degrees.
A depth map has an associated vertical field of
view which is a double. It represents the vertical angle of view in degrees.
The data type of a depth map is 16-bit unsigned integer. The algorithm to convert the depth map value to grayscale, given a depth map value d, is as follows:
If the user agent requests a combined depth+video stream, the devices in the constraint should be satisfied as belonging to the same group or physical device. The decision to select and satisfy which device pair is left up to the implementation.
The getDepthTracks() method, when invoked,
MUST return a sequence of depth
stream tracks in this stream.
The getDepthTracks() method MUST return a sequence that represents a snapshot of all the
MediaStreamTrack objects in this stream's track set whose kind is equal to "depth". The conversion from the track set to the sequence is user
agent defined and the order does not have to be stable between calls.
New consumers may be added in a future version of this specification.
6.2.1
Implementation considerations
This section is non-normative.
A video stream track and a depth stream track can be combined into one depth+video stream. The rendering of the two tracks are intended to be synchronized. The resolution of the two tracks are intended to be same. And the coordination of the two tracks are intended to be calibrated. These are not hard requirements, since it might not be possible to synchronize tracks from sources.
6.3 MediaStreamTrack interface
The kind attribute MUST, on getting, return the string "depth" if the object represents a depth
stream track.
It is an implementation detail how the frames are rendered to the screen. For example, the implementation may use a grayscale or red representation, with either 8-bit or 16-bit precision.
var depthVideo = document.querySelector('video');
var canvas = document.querySelector('canvas');
var context = canvas.getContext('2d');
var w = depthVideo.videoWidth, h = depthVideo.videoHeight;
context.drawImage(depthVideo, 0, 0, w, h);
var imageData = context.getImageData(0, 0, w, h);
// Create a new DataView dv on to an ArrayBuffer buffer// that exposes it as an array of unsigned 16-bit integers.var dv = newDataView(imageData.data.buffer);
// Read every fourth unsigned 16-bit value in little endian representation.for (var i = 0; i < imageData.data.length; i += 4) {
console.log(dv.getUint16(i, true));
}
// Alternatively, reconstruct unsigned 16-bit value. The result// is the same as if DataView and getUint16() was used.for (var i = 0; i < imageData.data.length; i += 4) {
console.log(imageData.data[i] + (imageData.data[i+1] << 8));
}
The near, far, focalLength,
horizontalFieldOfView, and
verticalFieldOfView constrainable properties are defined to apply only to depth stream tracks.
Note
The focalLength, horizontalFieldOfView, and
verticalFieldOfView properties could be upstreamed to a future version of the the Media Capture and Streams specification [GETUSERMEDIA] to allow them to be applied to video
MediaStreamTrack objects as well.
The near and far constrainable properties, when set, allow the implementation to pick the best depth camera mode optimized for the range [near, far] and help minimize the error introduced by the lossy conversion from the depth value
d to a quantized d8bit and back to an approximation of the depth value d.
If the near value, far value, focal length,
horizontal field of view, or vertical field of view is fixed due to a hardware or software limitation, the corresponding constrainable property's value MUST be set to the value reported by the underlying implementation. (For example, the focal length of the lens may be fixed, or the underlying platform may not expose the focal length information.)
partial dictionary MediaTrackConstraintSet {
ConstrainDoublenear;ConstrainDoublefar;ConstrainDoublefocalLength;ConstrainDoublehorizontalFieldOfView;ConstrainDoubleverticalFieldOfView;
};partial dictionary MediaTrackSupportedConstraints {
booleannear = true;booleanfar = true;booleanfocalLength = true;booleanhorizontalFieldOfView = true;booleanverticalFieldOfView = true;
};partial dictionary MediaTrackCapabilities {
(double or DoubleRange)near;(double or DoubleRange)far;(double or DoubleRange)focalLength;(double or DoubleRange)horizontalFieldOfView;(double or DoubleRange)verticalFieldOfView;
};
6.10 WebGLRenderingContext interface
6.10.1
Implementation considerations
This section is non-normative.
Note
This section is currently work in progress, and subject to change.
A video element whose source is a
MediaStream object containing a depth stream
track may be uploaded to a WebGL texture of format
RGB and type UNSIGNED_BYTE. [WEBGL]
For each pixel of this WebGL texture, the R component represents the lower 8 bit value of 16 bit depth value, the G component represents the upper 8 bit value of 16 bit depth value and the value in B component is not defined.
7.
Examples
This section is non-normative.
Playback of depth+video stream
Example 2
navigator.mediaDevices.getUserMedia({
depth: true,
video: true
}).then(function (stream) {
// Wire the media stream into a <video> element for playback.// The RGB video is rendered.var video = document.querySelector('#video');
video.srcObject = stream;
video.play();
// Construct a depth-only stream out of the existing depth stream track.var depthOnlyStream = new MediaStream(s.getDepthTracks()[0]);
// Wire the depth-only stream into another <video> element for playback.// The depth information is rendered in its grayscale representation.var depthVideo = document.querySelector('#depthVideo');
depthVideo.srcObject = depthOnlyStream;
depthVideo.play();
}
);
WebGL Fragment Shader based post-processing
Example 3
// This code sets up a video element from a depth stream, uploads it to a WebGL// texture, and samples that texture in the fragment shader, reconstructing the// 16-bit depth values from the red and green channels.
navigator.mediaDevices.getUserMedia({
depth: true,
}).then(function (stream) {
// wire the stream into a <video> element for playbackvar depthVideo = document.querySelector('#depthVideo');
depthVideo.srcObject = stream;
depthVideo.play();
}).catch(function (reason) {
// handle gUM error here
});
// ... later, in the rendering loop ...
gl.texImage2D(
gl.TEXTURE_2D,
0,
gl.RGB,
gl.RGB,
gl.UNSIGNED_BYTE,
depthVideo
);
<scriptid="fragment-shader"type="x-shader/x-fragment">
varying vec2 v_texCoord;
// u_tex points to the texture unit containing the depth texture.
uniform sampler2D u_tex;
uniform float far;
uniform float near;
void main() {
vec4 floatColor = texture2D(u_tex, v_texCoord);
float dn = floatColor.r;
float depth = 0.;
depth = far * near / ( far - dn * ( far - near));
// ...
}
</script>
Thanks to everyone who contributed to the Use
Cases and Requirements, sent feedback and comments. Special thanks to Ningxin Hu for experimental implementations, as well as to the Project Tango for their experiments.
The range linear format [XDM] is licensed under CC BY 4.0.