WebRTC Next Version Use Cases

[RFC7478] Section 2.3.12 describes a use case involving a multiparty online game with voice communications. In these scenarios, reducing time to join the game and receive media is important. To minimize this, ICE enhancements are desirable, such as the ability to control candidate gathering and pruning. Also, allowing a participant to broadcast a configuration to a “room” abstraction (maintained on a server), with other room participants responding back directly, avoiding a separate discovery step, minimizes conference establishment time. Also, managing audio quality and latency in a fair manner between multiple connections prevents queue buildup. Supporting this enhancement adds the following requirements:

Experience: This use case has been implemented by a gaming service utilizing [ORTC].

1. ORTC Issue 54
2. ORTC Issue 603

[RFC7478] Section 2.3.6 describes a simple communications service where the user changes access network during the session. This use case is enhanced by being able to ring multiple endpoints simultaneously, as well as to re-route media over an alternate path (potentially taking network cost into account) without need for signaling.

An additional enhancement is to provide management of the user experience at both ends of a call during interuptions to the media flow caused by other activities taking higher priority on the smartphone.

References:

1. Mailing list proposal
2. Mailing list proposal
3. ORTC Issue 583

Experience: This use case has been implemented by multiple native smartphone apps with call-kit integration.

[RFC7478] Section 2.4.3.1 describes a use case involving Multiparty Video Communications with a central conferencing server. In such a use case, clients with disparate capabilities such as differing bandwidth availability, screen size and maximum displayable frame rate may participate in the same conference. In such a situation it is advantageous to support Scalable Video Coding (SVC). Encoding with temporal scalability is supported by several browsers today and is utilized by most centralized conferencing services.

It is expected that spatial scalability (supported by VP9 and AV1) will become more popular with time. In this use case, if the desired video codec is known beforehand and participants are muted by default (as in a very large meeting), it is desirable to allow new participants to start receiving immediately, without negotiation. Supporting this enhancement adds the following requirements:

This use case has been implemented by conferencing services utilizing [ORTC], as well as proprietary additions to [WEBRTC].

Participants in a mesh exchange large files without disruption to audio/video sessions. It is also possible for a participant to send a large file to a user who is not currently online. Supporting this use case adds the following requirements:

References:

1. Mailing list discussion
2. Mailing list discussion

There are 'broadcast' applications that require low latency realtime media. Distributed auctions or betting as well as churches and company 'Town Hall' meetings for example. The same live video and audio and data is sent (unidirectionally) to thousands of recipients.

An IoT sensor maintains a long-term connection and seeks to minimize power consumption. Some of the sensor’s data may need to be sent reliable and ordered while other sensors may provide data that can be sent unreliable and unordered or in a partially reliable manner. Such IoT sensors may also produce realtime video or audio data for remote users which are privacy sensitive and may only be accessed by selected devices. This use case adds the following requirements:

Reference

Experience: Building a respiration monitor that works locally during an internet outage but is also available remotely when the internet connection returns.

New decentralized applications provide P2P services and client-server services for consumption in a browser.

The differences in transport layer semantics make it difficult to share code between the two modes.

Experience: Both pipe and [Matrix] have implemented systems of this sort.

A virtual reality gaming service utilizing a centralized conferencing server wants to synchronize data with media, using an existing Selective Forwarding Unit (SFU) to distribute the data. This use case adds the following requirements:

References:

A communications service that manipulates captured media prior to encoding and after decoding to provide effects including:

1. Captioning
2. Transcription
3. Language translation
4. Funny hats
5. Background removal or blurring
6. In-browser compositing
7. Voice effects
8. Stress detection

This use case requires manipulation of raw media from both local and remote sources. Since media processing can be CPU intensive, enabling it to occur off the main thread is important, as is enabling the processing to take advantage of the GPU. This use case adds the following requirements:

This usecase also requires that user agent provide non-discriminatory implementations of facetracking and body tracking algorithms that can be efficiently used by the application. This however is outside the scope of this document.

References:

1. Mailing list discussion
2. Mailing list discussion
3. Sharper Image Research

In a web game called “NameTheBird.com” participants use their devices to provide audio and video observations of birds to the service along with identifications for training purposes, allowing the service to identify birds from the provided audio and video and returning this information to the users in real-time.

The web application has a site specific federated learning-based classifier for contextual object detection, user intent prediction and media manipulation, allowing it to augment the streams it receives and inject identifying or other supplemental information into the streams sent or received.

The shared classification models are trained on the birds found by the participants and are based on the feedback of the participants. Each device client updates of the model are up-streamed to a shared model server that pushes updates of the global model to the clients.

Implementation outline:

1. Originating media (raw) streams are cloned for inference and training purposes, denoted “inference stream” and “training stream”, with the inference stream also being the media stream shared with peer(s). The cloning can occur any time during a session.
2. Inference stream: A web site specific classifier acts on the raw inference stream, with the result used to guide a custom encoder in the sender device and send metadata to the server and peer devices outside the media stream. The encoder adds proper augmentation, e.g. sign with “name this bird” hovering over the enlarged bird in case of video enrichment, or enhanced bird song if audio.
3. Training stream: Model in training classifies the raw data and evaluate the classification using user feedback, said feedback loop being web site specific. The evaluation may be “online” or “offline”, offline meaning the training is done at a later stage on the recorded encoded media set.
4. Both inference stream and training streams may use payload protection depending on trust model on compute resources for optional intermedia server side of app.
5. Both inference stream and training streams use transport object for communicating with peers or servers, the communication in some cases can be a site specific QUIC based transport solution, in others RTP based.

This use case adds the following requirements:

Cloud video conferencing systems have no need to be able to access the cleartext media and text flowing through their servers. Some of these conferencing services desire to be able to promote trust by explicitly showing they do not have access to contents of their users' calls. They are trusted to connect the right people to the conference and to route the packets but they are not trusted to access the audio and video media or text in the call.

Solutions to this problem fall into two major categories: one where the JavaScript comes from a source trusted to see the media contents, and one where it does not.

Some simpler media/data sources/sinks require only a subset of WebRTC functionality.

In such use cases the full SDP O/A could be replaced with a server generated token at runtime.

A goal might be to set the source attribute of a video tag to point at a WebRTC video source.

Care needs to be taken to avoid introducing security vulnerabilities.

Experience: Both pipe and pion have built systems of this sort.