WebRTC December 5 2023 meeting

Peter: this looks good - 2 minor remarks:
… the array argument makes sense if you 90 candidate pairs to remove, but it makes it more complicated to figure the return value
… what's the reason for preventing adding back removed candidate pairs?

Sameer: you could recreate a pair via remoteAddCandidate(), we don't have a mechanism for the app to add one

Peter: oh, so the idea is the *browser* doesn't add it back, makes sense

Bernard: canceling nomination: do you mean remove pair before nomination?

sameer: relates to the event we added which the app can preventDefault()

henrik: if performance becomes a concern, it's possible to batch internally the removals
… the PR LGTM
… no strong feeling re array vs array, but either way can be optimized

jan-ivar: I don't like the array since I don't think it's necessary, Promise.all can do the batching
… restartIce() resets things, right?

Sameer: right

jan-ivar: overall this feels a bit like a hack around nomination because nomination may be too strict
… we could clarify that this is part of the UA-behavior
… I want to make sure we don't overstep anything set in the IETF spec

Peter: ICE implementations are always free to remove candidate pairs

Sameer: I'll update the PR in that direction before it hopefully gets merged then, thanks

Issue #2170 Data channel default binaryType value is 'blob' 🎞︎

[Slide 13]

florent: using Blob to send big files, even if implemented, wouldn't work as is

Randell: having originally designed this, at this point, I agree switching the default makes sense
… it's unfortunate the lack of implementation has gained over the spec
… it is possible to send large files with blob

florent: maxMessageSize doesn't make that practical

Randell: this can be polyfilled on top of datachannel, but this is water under the bridge
… we should take this as a warning to avoid these incompatible implementations that last for so long

florent: blob implementation would be possible, but switching the default would break web-compat

Bernard: the slide should say "Firefox" rather than "Safari" shipping blob as default

RESOLUTION: change the default for binaryType to arraybuffer

Mediacapture Extensions: latency in Audio Stats 🎞︎

[Slide 14]

[Slide 15]

Harald: is there a way to figure out when resetLatency was called?

Henrik: the app knows

Harald: is reset sync?

Henrik: yes

Paul: no objection on my end

RESOLUTION: adopt PR

WebRTC Extended Use Cases: Low Latency Streaming 🎞︎

Low Latency Broadcast w/fanout PR #123 🎞︎

[Slide 19]

[Slide 20]

Bernard: the use cases in 3.2.2 have very different requirements
… e.g. very low latency is likely only relevant to sporting events
… PR #123 attemps to refocus it on very low latency use cases (auctions)

[Slide 21]

[Slide 22]

Bernard: ^ proposed new text with PR 123 applied

PR 123 preview of auction use case

jan-ivar: could we remove "millions" from the use case?

bernard: sure, more likely relevant in the context of a webinar

jan-ivar: will add to the PR review

[thumbs up from folks on the call with going forward with the PR]

Game Streaming 🎞︎

[Slide 23]

Bernard: performance is a growing issue, esp in the context of 4K

[Slide 24]

[Slide 25]

bernard: #103 is feedback from youenn - formulating N37 in terms of API requirements

[Slide 26]

bernard: PR #125 attempts to do that by referring explicitly to control of hardware acceleration
… this relates to discussion we had e.g. about events for hardware failure

Randell: what does "controlling" mean from a stack perspective?

Bernard: be able to control whether you specifically want hardware acceleration
… Web Codecs has a preferHardware switch
… We have an issue to raise events when fail over from hardware to software

Randell: notification seems useful, but that seems different from this specific requirement
… the requirement here seems to be on implementation

Bernard: no, it's an implementation on the API
… part of the issue is not knowing why the failover happened doesn't let developers determine whether they can fix the situation

Randell: the goal seems right, but the wording remains too fuzzy in terms of requirements

Henrik: "controlling" is the right word - this isn't just "prefering" for a given codec and leaving it to the UA to decide
… the app needs to know what's possible and make a decision based on the available hardware

randell: so exposing more information about current hardware support for streams that are running and potential hw support for streams that are offered

Bernard: there is also the big mess around receivable / sendable codecs

hta: control here is a keyword for "can you do this?", "do this and let me know if you can't"

jan-ivar: hw control is a requirement today, but it may not be the case tomorrow
… maybe we can specify the requirements in a way that makes it clear that it requires hw support today

randell: hw acceleration is mentioned as an example of a solution in the current language

jan-ivar: there will be concerns in terms of privacy review

bernard: please suggest changes to the PR

jan-ivar: how about "...acceleration if necessary"

henrik: SGTM

bernard: +1

Sun: PR #118 touches on other requirements for game streaming

[Slide 28]

Sun: we propose 2 kind of requirements: "fast recovery"

[Slide 29]

Sun: and 2 requirements for "consistent latency"

[Slide 30]

Sun: we discussed this during TPAC

Randell: I support this; when we first designed WebRTC, we purposefully didn't specify what happened in terms of decoding on frameloss
… Support for non-full I-frame collections is possible as well

Bernard: is there any API change surfaced to JS needed to support this?
… I could see this needed in the context of WebCodecs, but not sure for WebRTC

Sun: at minimum, the ability to set a flag on whether to enable fast recovery

Bernard: not just an SDP parameter then?

Peter: supporting this would require a large change to libwebrtc video jitter buffer - is that realistic?
… rtptransport has custom jitter buffer as a use case

Jan-Ivar: I'm unsure W3C needs to do something, vs having this being negotiated via SDP with user-agent deciding whether to support it

Randell: it could be done via SDP, but it feels a bit of a hack
… re implementation, having done this in the past, it would indeed require changes in the jitter buffer, but it doesn't seem like a major rewrite, more like a small or medium amount of work to continue decoding while waiting for an i-frame (with errors in the buffers)

Harald: if it doesn't change the bits on the wire, it shouldn't be done via SDP
… so in my mind, it does require an API

Henrik: if this was possible to do, would there be any reason not to have it on all the time? does it add overhead? does it need a control knob?

Bernard: there are probably applications that wouldn't want do this, e.g. in the context of a legal case

Dom: maybe we need a use case for the "off" mode before we say we need it

Sun: please bring more feedback on github

[Slide 31]

Bernard: if this is about how to do it, it probably needs discussion in IETF given discussions about how well RPSI works
… I'm not sure there is a WebRTC API issue vs implementation or IETF
… it's most complicated for conferences (compared to the one-to-one game streaming scenario)

Sun: so I guess we should remove this req based on that feedback

Bernard: +1 - let's take it up in IETF, it's an important problem

[Slide 32]

Bernard: which messages are you interested in getting the interval of? PLI? Receiver reports?

Shridhar: tranport@@@

HTA: can this be uplifted a bit? what you want is to have a definitive time limit by when you know if a packet was lost
… it might be good better to say that rather than speak of specific RTCP messages
… this would make a more concrete requirement that allows different implementations

Sun: we'll update the requirement in that direction

[Slide 33]

Randell: how much of this is talking about the API vs the implementation?

Sun: no API change required here

Randell: so more of a commentary of the implementation in libwebrtc
… not really in scope for the WG

Sun: thanks for the feedback

Jan-Ivar: there is an API for jitter buffer target

Randell: that's mostly to have more delay

jan-ivar: or less, in theory

Sun: we'll update the PR based on the discussions

RtpTransport 🎞︎

[Slide 37]

RtpTransport Reminder

Peter: 3 possible general directions I want to review today

Harald: another dimension is whether you can talk to existing end points
… you can't with the new datachannel - in which case, what's the advantage over using WebTransport or Quic
… the piecemeal approach doesn't need to be complex for the simple case - you just let through the things you don't want to modify
… as long as you stay within certain parameters, you can talk to people that are not you, which I like
… you can talk with endpoints that talk basic SRTP
… it provides for incremental deployments and incremental implementations, by focusing on the particular function we care about today

Peter: you're correct with regard to the new datachannel speaking to existing end points
… in terms of Quic - they're not P2P in the browser today; we could fix that
… I agree with your points re piecemeal

Jan-Ivar: in my slides, I'm not proposing an rtp data channel, I agree with the limitations you and Harald have pointed out
… I think we should only expose RTP as needed (maybe that points toward piecemeal)
… we need to understand better the scope of the problems we want to address

Peter: so hearing more support for piecemeal (bit by bit adding to the existing API, but with likely a low level API)
… I've done some exploration in this space

Bernard: dall-e did a good job representing the data channel proposal being an isolated island
… I would not favor that
… re piecemeal customization - if you can get the flexibility without the complexity, this would be ideal
… but we have to demonstrate it
… the incremental deployability is also appealing

Youenn: another vote for piecemeal - a web site should be able to override some parts of the pipeline, but with the UA taking over at some point
… complexity is an important consideration
… we've approached this by exposing injection points into the pipeline, we should continue with that approach
… we should compare with the approach we're exploring in encoded-transform or to bring web codecs in

Henrik: another benefit of piecemeal is in terms of shipping it from an implementation perspective; not enough expertise on RTP to say much more

Issue #9 Customizing piecemeal 🎞︎

[Slide 53]

Peter: if we go with piecemeal, this can be done - suggest following the taxonomy in rfc 7656, most importantly RtpStream
… building on top of that
… will flush this out more

Bernard: some of this would be around custom FEC/custom RTX scenarios?

Peter: right; I have explored a number of scenarios with that approach

Stefan: would you insert packets into these RtpStream objects?

Peter: capture what has been sent, allow it through if you're not customizing, or modifying it and injecting it back
… with lots of implication in terms of congestion control - it gets complex but I think it's feasible

Harald: once you allow to insert packets then you can generate packets yourself, in either direction

Peter: absolutely; this is a superset of what I presented to TPAC in that sense

Jan-Ivar: I'm worried by "full piecemeal customisation" - I don't think we need to go to the extreme low level of exposing everything of the RFC on RTP

Peter: to be clear, I'm only asserting something is feasible, not any API shape
… I'll come back with more specific API proposals, and others can make proposals as well

Henrik: I get similar vibes to encoded transform from this slide
… which has proved a powerful pattern

Issue #10: SDP Bumper lanes 🎞︎

[Slide 54]

Peter: if we go with piecemeal, I think we'll end up with lightly enforced bumper lanes

Bernard: Harald raised a concern with avoiding interferences with the existing peerconnection; this looks to be in the good direction

Peter: this may depend on whether we allow creating transports outside of a peerconnection

Harald: I think there should be an API for reserving things, e.g. to reserve an SSRC
… the API might expose names (e.g. extensions by URIs) rather than by numbers which avoids the issue of conflicts

Peter: +1; interesting idea on names vs numeric ids

Harald: I'm hitting this with fanout land where we're discussing using mime types rather than payload types

Issue #7: SRTP/cryptex support 🎞︎

[Slide 55]

Peter: with piecemeal, I think we could just rely on the outcome of the SDP negotiation

Bernard: we have a cryptex API in webrtc-extensions; if the other side supports cryptex, encrypting everything seems sufficient granularity, don't see value in per-packet control

Issue #13: Workers 🎞︎

[Slide 56]

Peter: if there is a e.g. an RtpStream interface, we would need to ensure it is transferable - that sounds doable

Harald: SGTM; remember that when you transfer, you transfer one end of the tube; what's at the other end stays attached to where it was originally, which may have implication on implementation

Bernard: we support transferable stream, but would RtpStream be a WHATWG stream?
… transfering individual objects may not be the right thing; e.g. for custom NACK needs both dealing with receiving Rtp and sending RTCP
… you need a combination of read and write streams
… this depends a lot on whether we use WHATWG streams or not

Peter: we need to figure the specific things that people would want to use in workers
… you're right that RTCP is tricky

Bernard: we also need to figure which workers we need to support, e.g. audio worklets

Jan-Ivar: time-check?

Peter: ok - we'll need to look at some of the other issues
… re WHATWG streams, I think this can be decided at a later stage once we have figured out the more fundamental question

WHATWG streams 🎞︎

(issue #8)

Harald: transform has proved a limiting model - a transform is combining a sending and receiving API together
… we should try to reduce the coupling between components, not increase it

Bernard: more work needed on this API; we talked about having a design team to work on this to advance more quickly on this in the general meetings
… and advance the explainer and specs into more fleshed out states
… next meeting next week

– DRAFT –
WebRTC December 5 2023 meeting

05 December 2023

Attendees

Meeting minutes

WebRTC-Encoded Transform: Describe data attribute PR #212 🎞︎

WebRTC Extensions 🎞︎

PR #175: Add RTCIceTransport method to remove pairs 🎞︎