See also: IRC log
[agenda bash]
Transceiver Part 2 slides from Peter
Peter: coming back on a couple of
new options for rollback
... Option I is a refinement of the F from yesterday
Peter: [description on slide 2]
Martin: there is 2 things that
create a transceiver: adding a track and receiving a remote
description
... when you do a rollback, you rollback only those created by
the remote description
Martin: if you have one that was created by the combination of these 2 things, you only reset the one created by the remote side of things
ekr: the ordinary rollback
behavior is to return the state you were before
... here, it's only half way backtracking
martin: we discussed this
yesterday and had an option to rollback everything since the
last stable state
... but that includes things that were done even before
reaching stable state
cullen: when we said rolling back
to 0 time, there are different states that need to be rolled
back
... some states are linked to the offer/answer, others
aren't
... rollback should be only about the former
... things that haven't been applied to the offer/answer state
machine shouldn't be impacted
martin: the problem of doing otherwise is that a rollback rewrites history even further back to stable state
ekr: with such an approach, do we actually need rollback?
martin: we need it, and we implement it that way
cullen: [missed comment about pranswer]
martin: we're unlikely to implement pranswer
peter: we have it now, but we haven't implemented transceiver yet
cullen: I think we're at the point where we need to discuss whether to remove it
peter: I'd love to remove it
cullen: if we're going to reopen these discussions, there are other decisions that would need to be revisited
peter: that probably needs to be
discussed separately in any case
... presenting the 2nd option, option J
... which is forbidding rollback after setting description and
calling addTrack after
ekr: the main reason of rollback is that sometimes you need to apply a description before you can decide whether it's good for you
hta: we discussed other cases
yesterday beyond that
... if we don't support rollback before or after addtrack (a
"no return" point), it means that some error cases will have to
be handled by tearing down the connection
ekr: which error cases?
hta: setRD is async, so things can happen between the call and the promise resolution that makes you think "I shouldn't have done that"
ekr: the Transceiver is created at the moment of the promise resolution of setRD
hta: the question is whether you can get in trouble between addTrack and setLD
ekr: I'd like to see a concrete set of events
juberti: if there is no
compelling use case, I think option J is nice and simple
... we could always come back to this if there are use cases
that require a more complex approach
ekr: do setRD, make sanity checks, do addTrack
martin: but sanity checks can be long and thus addTrack might need to be queued
cullen: the simplest scenario is: I send an offer, I add a track, the answerer doesn't understand my offer for any reason, and we need to rollback
ekr: but that's for setLD, not RD
juberti: I think for RD, you'll want to do your sanity checks before addTrack
hta: option J is looking good because it leaves us room for adding more stuff later if needed
cullen: this is deprecating rollback in a bunch of use cases
adamR: but we haven't identified any such use case
ekr: in the trivial case where I
have an outsdanding offer with voice and I add video, and the
audio call gets accepted, I'm now in onnegotiationneeded
because it doesn't match the negotiation state
... the question is what should I do if the offer is
rejected?
... how would you rollback?
... [describing an example on the whiteboard]
... A creates a datachannel, we exchange an offer/answer and
get back to stable state
... then I add track audio, send an offer
... and then I add an video track
... if instead of an answer, I'm getting a rollback, it would
be have to get to negotiationneeded state
... and this is addressed by option J
... but the other case of stuff happening after setRD and
addTrack doesn't seem very plausible
... but even if the plausible case I described, it's not even
clear how you can react
... stepping back a little, I'm not even sure what rollback is
useful for
cullen: adding/removing rejected streams is easy, modifying them is much harder
ekr: but recovering from a stream change will take a lot of logic
martin: the remote rollback case is harder to make
ekr: so we have a complex system
to signal situations that are complex to recover from
... not sure how useful that is
juberti: rollback is a way to
reject things that we immediately realize we don't want
... it's not clear that there is a value to use it for things
lower down in the chain
ekr: one way of dealing with avoiding loops of negotiations is to freeze the latest stable session in a super-stable / immutable state
peter: we need to distinguish remote and local rollback
cullen: I think we all agree we need local rollback
martin: local is (mostly) useful
for apply and reject
... option J will be enough if we deal with activateSender and
look at other state changes (at least removeTrack)
cullen: the only thing that rollback affects is what is part of the negotiation
ekr: jesus what a mess
adamR: either the app is in charge of keeping track of state changes, or the browser gives affordance to it
martin: option I was for the latter
juberti: but option J gives us more freedom to evolve on this
martin: it's a cop-out
... option I was rolling back just the receivers
hta: my suggestion is let's adopt J, and someone wanting to propose something else will need to provide a use case
martin: I don't think only handling addTrack is enough
peter: any other modification (e.g. activate or remove) is not surprising
cullen: another way to put it is: if you modify the transceiver created by the remote description and do a rollback, you'll get surprises
peter: the only thing that can change a transceiver is addTrack
martin: not if I addTrack before setRD, and then do a removeTrack
peter: this would not connect the two transceivers
martin: I think JSEP say they
would
... when you create an answer, you go through the m-lines in
the offer and find one that matches your track and reuse that
transceiver
ekr: it would be very surprising otherwise
peter: not to me
juberti: it would not be that
surprising
... it's not clear that the browser can make useful matching up
based on track types
martin: so you're backtracking on track matching in JSEP
juberti: we can move stuff from
JSEP to the application layer
... JSEP has some very complicated logic that is not
necessarily full enumerated
cullen: this matters a lot of legacy interop
adamR: this is revisiting multiple years of history and discussion
<mt> http://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1
juberti: I can tell you from experience that the matching up algorithm is extremely complex and the browser is not always in a good position to know what was meant
<mt> paragraph 5, http://rtcweb-wg.github.io/jsep/#rfc.section.5.3.1
juberti: we can maybe simply consider peter's case for the matching
martin: I suggest we simply stick
to JSEP's rules
... but that argues that option I is a better match to the JSEP
model
hta: so are we converging on
getting a pull request for Option I
... Peter will provide a pull request, and martin will
review/contribute
<vivien> Slide 4 Can we change MIDs on an m-line
Peter: looking at the other issue
we discussed: changing MIDs on an m-line
... Justin said making mids immutable would make JSEP much
simpler
cullen: this sounds good, but we
have to deal with the mid exposed in candidates
... we should have a note to that effect
<juberti> Candidates should get GCed when m= section is cleared
peter: this has only effect on JSEP
juberti: I think JSEP probably needs to understand the concept of transceiver to do the matching to SDP
RTCRtpSender support objects -> interfaces #350
martin: we have a bunch of
dictionaries returned by getParameters, and setParameters
accept these dictionaries
... when looking at setCodecPreferences, having to deal with
dictionaries being wrong vs requiring an interface, the latter
is much better
... since interfaces guarantees that the object is built by the
browser, you get much better control
... in general, domain objects are recommended to be built from
interfaces rather than dictionaries to avoid this type of
issume
(now looking at file modifications)
martin: in my pull request 350, I
distinguished between dictionaries that are under control of
the application, and interfaces that the browser need to
control to restrict input from the app
... going through the various existing usage in getParameters /
setParameters, most were cases where the browser should not
leave access to the developer
... going through this, it wasn't clear whether making ssrc
changeable is a good idea
... it was important beforehand, not sure it is useful
peter: I meant to be mark it as
readonly
... there are use cases where it would be useful, but there was
no consensus to do so
martin: I only focused on setParameters, but this probably affects other parts of the spec, especially stats
jan-ivar: we should not make too
much generalization either; the case for setParameters is clear
because you're getting objects from the browser before
resending it to it
... it's not clear that the same applies to stats
dom: I think the case is quite
clear for get/setParameters
... we would have to look at a proposal to evaluate stats
hta: speaking as stats editor,
it's important to have serialization — but we can get that for
interfaces too
... I wonder how that would play with ids
<Bernard> The question is whether interfaces will paint us into a corner - where the entire API will be unextendable.
<Bernard> These changes are being made based on the assumption that attributes that are readonly in exisrting objects will remain that way... or that this will be consistent between objects. That is not true in general.
<Bernard> Capability objects and Parameter objects are the similar or in some cases identical... dictionaries can serve both purposes but interfaces with restrictions cannot.
<Bernard> So effectively imterfaces are poisoning the object space in a way that will require massive changes later on... for no useful purpose.
shijun: dictionaries can vary, where interfaces can't
martin: if there are capabilities that browsers can or can't do, we should change the interfaces accordingly
varun: but for stats, there is data the browser may not be able to provide at some point
<Bernard> But the interface can't do multipile things at once....
<Bernard> The same attribute cannot be readonly and readwrite, for example....
shijun: I think we need to think more about this; at least the comparison with stats gives me pause
<Bernard> Yet that is exactly what is required in many cases if the same dictonary is used differently in different methods.
martin: I agree that the timing aspect of stats makes a good argument as to why to use dictionaries instead of interfaces
peter: if later one we decide to make ssrc writable?
martin: ... we add it to the dictionary
peter: so it would be fairly simple?
<Bernard> You'd have to change the attribute from readonly to readwrite - whereas with a dictionary you would not have to make that kind of change.
martin: yes; there are more complicated examples, e.g. when we want to change a sequence of interfaces to something else
jan-ivar: you could deal with that by making the attribute writable instead of readable (and still need to use setParameters)
<Bernard> There are cases where attributes are readonly within some methods and writeable in others. Interfaces cannot deal with this.
<Bernard> Yes.
dom: so this PR makes choices about things we would need to evolve later — are we comfortable with those?
martin: from my reading, the only constraints it creates are on things that are firmly under the browser control
peter: looking at the example, does this change requires this "copy(params)" call?
martin: yes
[discussion as to why the getParameters returns an immutable object]
martin: my assumption had been that setParameters only needed partial application
ekr: I'm still not sure what's
the point of having the readonly fields
... I'm confused
hta: I think it's clear that is no consensus on #350
martin: I think we're close to making progress
hta: I don't understand the claim that getting an object from an interface with non read-only parameters, you could not change them
martin: the issue is that there
are expectations that tweaking values from a platform objects
should have side effects
... in my model getParameters returns the browser view of its
state
... and then you have to provide your own view
ekr: but how does this help?
martin: I'm open to adopting the model of a modifiable platform object for setParameters
[consensus this would be better]
martin: I'll revise the pull request accordingly
Bug Bash for WebRTC-1.0 slides
harald: bug status
... there are still 52 bugs, (152 were closed)
... and the oldest bug is around 1 year old
... if you look at the categories of bugs
... you will find many (see slide) but we want to fix all of
them, except the functionality wishes
... of course, we might not deal with all of them today
.....
... as we did for media capture and stream, we first remove all
the editorial stuff
... first bug: #150 and #151: closing a peerconnection
... there is a question regarding what is the best practice for
promises? return error / failing, or not?
justin: I thought we had already decided that we would fail them.
jesup: there were a couple a
question regarding what happen after you close the peer
connection
... we don t really care anymore after a close. because
anything produced would not be usable.
fluffy: i do not say it s right, but right now we are collecting errors, and making statistics on them, so I am in favor of failing it.
martin: but you re calling close first, so you re biasing your stats
fluffy: right, i m just describing what we do right now.
mathieu: if we have an error, the application then get the option to deal with it or not.
ekr: can we make the promises cancelable?
martin: there are only two states right now, not a third cancelable state.
harald: our options are either fail or not fail. no cancel option
ekr: ok
martin: that s my rea
jan-ivar: the general consensus is that in cases where you shutdown or refresh, you don t want JS activity
justin: I think it s a different use case. I think failing would be nicer, if you want to do things like UI cleaning.
jib: <more comments on the close case>
ekr: if we don t fail, there is no option at all for the app. It seems to be a pretty strong argument in favor of resolving the promise. Of course, app can screw up, but hey, c est la vie. Plus it s already in the specs.
adam: bad user code is bad user code, we can t control it.
harald: plus PC will be closed at that time, so they can t shoot themselves in the foot anymore, really.
<juberti> +1 to leave it
jib: it looks like there is no case where close would be called by the browser. so I m ok with resolving the promise.
harald: show of hand?
all in favor or abstening
harald: ok , next problem
... #337 and #338: webrtc vs JSEP
... the suggested guideline are: if it s about what the SDP
looks like, defer to JSEP, if it s about the JS API, write it
in webrtc
harald: specific issues: how to reference JSEP in webrtc specs, how to reach agreement between teams, and what are the permitted changed to SDP in JS (between CreateOffer and SetLocalDescription)
peter: there is a sync issue, and sometimes we need to reference things that are not there yet.
fluffy: yes, like any sync problem, we'll deal with it. Eventually we will converge, it's ok. I'm happy to do whatever is needed here [as chair of IETF rtcweb group]
ekr: is there any reason why we
can cross reference symbols between webrtc and JSEP?
... I'm happy to help
dom: i'm happy to help too
fluffy: i don think we need to work too much on JSEP => WEBRTC, those are rarely needed
dom: one of the difficulty when i
started to review the spec, there is the question of
maintenaining referencing, but there are also implicit
references
... like handling ICE candidates
fluffy & dom: a scrum is in order
harald: dom and ekr are in charge of dealing with the leakage
harald: last bug
#334 consider using the stream API
back when we made the original decision, DC were modeled after web sockets,
however, streams API was not ready.
denis: revisiting it now, it
seems that streams API is still quite not ready
... proposing to leave it for later (close it now, mark it
"LATER")
... everybody OK with that?
all: OK
harald: next
harald: #257 ICE candidate should have accessors
harald: solved by <class
name>. Closed
... #263 Overriding default ICE servers
<different discussions about it being about turn, return, ICE servers ......>
ekr: let's separate the use cases. Here we are speaking about ICE servers, which are provided by the app. return, which behaves more like a proxy, that the application do not have access to today, is different.
mathieu: I'm not ok with browser overriding ICE servers (STUN, turn) provided by the app.
fluffy: I think this bug was
about to decide wether JS can decide to let the browser get
control over ICE servers.
... wether it is overriding one list (app) by another
(browser), or merge both lists.
... but this bug is only about overriding.
dom: what downside is there to use user provided server?
justin: if you use non-app-provided ICE server, you might bust th app.
adam: we can propose default servers, but make it explicit to the app with a GiveMeTheDefaultServersYouKnowAboutPlease() api available.
harald: here is a matrix of all
the possible cases.
... <drawing on the board>
justin: so where are we with this?
fluffy: summarizing: return and
proxy are not available to the app
... however in this bug, we propose an API that allows the app
to query the browser for known ICE servers, and do whatever
they want with it.
ekr: in terms of security, this is the same surface.
harald: so ... we do not override?
mathieu: yes, do not override.
dan: do not AUTOMATICALLY, silently, override.
harald: so we all agree on this?
martin: I have a pull request :-D
Harald: I feel ekr and martin are in favor ... ?
ekr: i m in favor of everything that close this conversation faster.
fluffy: it s something we wanted to do for a long time, we just never came around to decide HOW to do it
justin: I disagree.
<dom> [to clarify for my later self, the reason having this as a list of STUN servers rather than a a-priori-less-fingerprintable boolean "use configured stun" is that these STUN servers can be trivially exposed via stats after ICE gathering, without any user involvement]
harald: so we rejected the idea of letting the browser to inject ICE servers just because they feel like it.
adam: the other use case is return, and we do not need a JS API for that.
fluffy: agreed.
justin: in the case where you have multiple level of NAT, it s not clear people can handle it with ICE servers.
fluffy: someone report yesterday that the amount of <...> is pretty low
justin: i said that, right
ekr: you are misunderstanding return. It does not involve a normal TURN server, and in spirit it should be seen as a proxy.
fluffy: I mean this could work without return.
martin: the pull request is done, we could merge this right now.
harald: which interface shall we put it on
martin: PeerConnection (and it s static, which will also been fun to implement, but hey, my happy life)
harald: everybody satisfied with this?
fluffy: yes
justin: yes
ekr: fine
haralrd: ok, merged
... #265 add privacy/security review questions
... need volunteer for review and text proposals
ekr: is the rule obvious from this document, or will need further input from IETF docs?
harald: obvious.
ekr: hum, oh, uh ..... I guess I can do it .......
fluffy: why? it's an insane amount of work!
ekr: well, really? well, .. i'm not going to do it then :D
fluffy: the AI for you should be: read the document and decide if you want to do it.
ekr: agreed.
fluffy: we should close that bug anyway
alex: ekr reads it, decides, and if he decides NOT to do it, we reopen it.
ekr, fluffy, harald: OK
<adjourned, 1 hour for lunch>
<resuming the bug list>
harald: #305 (7 more bugs to
go)
... Describes what happen when media changes
... this is a remote variant of the "variable bitrate camera"
discussion.
... clarification: is the problem a sender side problem
(MSTrack violating constraints in the PC) or a receiver side
problem (incoming data violating constraints set on the remote
track) or both ?
mathieu: what is a constraint on the remote track ?
harald: nothing in the spec prevents you from applying constraint on the remote track.
mathieu: if you apply constraints
on a track, local or remote, the browser might do some
processing to make it fit the contsraints
... and I believe FF implemented ApplyConstraint (jib)
jib: currently we only support ApplyConstraint for local tracks
stefan: Spec says that you cannot act on constraints applied to remote tracks. Or that browser should ignore it....
mathieu: app can signal it to the remote side.
shijun: that s a protocol level thing
adam: at the application level, you do what you want, the specs only deals with local constraints.
harald: martin, can you elaborate on the bug
martin: sure, constraints at the session level, might be not overlapping with constraints on the track.
fluffy, ekr: it is already defined in JSEP.
matrin: right, I m just asking someone to explicit this case [in the webrtc spec]
ving: can you repeat the original question
matrin: you have (in the sdp) constraints both at t he session level and at the track level.
martin: the intersection might or might not be empty.
<ekr> https://tools.ietf.org/html/draft-ietf-rtcweb-jsep-12#section-3.5.2
martin: and each call to apply constraint can create a problem
ekr: i just pasted the corresponding text from JSEP. I think it should be in our spec as well.
harald: so the scaling policy is documented in JSEP.
ekr: that s correct, but we need to make a PR, because there are blank spots in the JSEP spec still.
<discussions about scaling up policies, and how to deal with errors>
harald: it seems that the text in JSEP is the right thing to import. Do you have a volunteer.
fluffy: I'll do it
harald: just import it from JSEP.
fluffy: wait, no, i don;t know where to put it!
<juberti> ...back...
<welcome back>
<discussion about wether we should handling scaling up on the Sender>
ekr: peter is saying, in an ORTC world, ....
peter: no, the RTPSender just sends whatever the size of the track is.
ekr: but for simulcast, you are forcing size in. ok, it s scaling down and not upscaling, but you still do not respect what the size of the track is.
fluffy: so where would you want
to put that (peter)
... ?
<juberti> seems like a sender property
martin: what is fundamentally wrong with putting in on the sender?
peter: i prefer it on the transceiver. But i m not going to fight to death on it. Would your solution lead to renegotiation?
fluffy: no.
harald: sense of the room, and alternatives ?
<juberti> sender
harald: senders has it.
stefan: who has the AI to make the pull?
fluffy: I'll take it.
harald: #308 RTCRTPSender.get/setParameters is underspecified
<vivien> RTCRtpSender.get/setParameters is underspecified #308
peter: I can't think of anything that would go outside the negotiated params.
adam, fluffy: we have several things that constraint the bitrate.
fluffy: if we can detect an error, please detect it.
martin: this is only one of the
issue in this bug.
... on the setting of parameters, shall we use getParameters?
Is it done as a promise resolve?
... I think that the other adam (B) has something for that, but
.... I'm not sure we can close this one today.
<juberti> sorry!
harald: so setting the parameters is immediate, but the action triggered by the setting can take some time
dom: so if you set and get, you get back the values you've just set?
peter: if we really wan tot know if things were done, we need to return a promise, but ....
martin: so what is the conclusion? The parameters are set right away, but taken into account only at a future (undefined) time?
xxxx: yes
peter: the question remains if we want a promise to be resolved at that undefined time.
martin: codec setting example
dan: so what happen if you make a set, and then a get, before the set finished?
adam: you get whatever is there at the get time, even if there is a pending operation.
martin, ekr: then you lost the ability to get the old parameters.
peter: that seems for a lot of complexity, while we have only one use case.
adam: the other problem is that it is an expending set.
peter: we are close to 1.0
...
... the set can't expand forever.
ekr: do we reject at set time? Why don t we make it async, and that s it.
harald: ok, set return a promise.
martin: i m not in favor of having another queue. I don t want get to return a promise.
harald: who is taking responsibility for this?
jib: note: you can t throw and return a promise
martin: i think we should ask Byron to identify what we missed.
peter: and also that we really do not have anything going out.
martin: I did answer that.
adam: I'll take care of the PR
harald: ok, next slide.
... #340 'RTCPriorityType not aligned with names in
rtcweb-transport
fluffy: so we re going with the tsvwg naming, right?
harald: yes.
... next #341, [FIPS-180-3] has been replaced by
[FIPS-180-4]
... next #343
RTCDegradationPrefence vs RTCQualityPreference
peter: thought we decided on Degradation
harald: ok, peter, you ll take
it
... next #345 #346 - create DataChannel param Error
shall we return n InvalidValue?
scribe: ok, next
... #358 Remove mid from sender receiver
peter: we talked about it yesterday, and then i realized it s easy, and there is no reason not to
harald: ok, done. next. last
slide.
... and that s the only thing i m actually sure about, this is
really the last slide.
<back to the agenda>
ekr: agenda proposal: focus on 1.0, keep NV for later if needed.
fluffy: proposing to put simulcast on the list since we made progress on it yesterday.
martin: yeah, most of the things we solved in the last hour we would have never solved on the mailing list.
<jesup> It I heard someone reference me, couldn't hear what was said. "lot of people happy..."
<vivien> Adding an accessor for the browser-configured ICE servers #365
<vivien> ^^ the issue that was just discussed
<dong_Cm> shalll we still have the section 'Introduction to ORTC', today?
<dom> dong_Cm, it's not clear yet — it depends on how much progress on the 3 additional topics we manage to make
<vivien> m-line match up doesn't work with RtpSender.mid or RtpTransceiver.mid #363
Peter: on issue #363
Fluffy: should set the mid after offer/answer
Peter: if transceiver has no mid,
how to match with the m-line?
... addTrack can lead to new transceiver or choose the existing
transceiver if mid match
MT: can check the transceivers and choose one without using mid
Peter: offer created by the transceiver reflects what is created already, so the mid should be set already when ceateOffer
MT: think #3 is okay
Peter: no need to create the
sender when addTrack, the transceiver can then be
replaced
... #4, remove mid and use a new transCeiver with the same
sender
... mid uniquely identified the transCeiver
Fluffy: why #2 won't work?
Peter: not deterministic, so will need more rules
Adam: based on the order the tranCeivers are created and the type of the transceiver (audio or video)
Fluffy: follow the JSEP definition, based on the order of the transceiver creation
Peter: rollback, mid get
reset?
... need time to think through the jsep matching rule, jsep has
no transceiver concept yet
... mid proposed at create offer, and set/final at
setlocal/remote.
MT: need to think through all corner cases
Peter: take the PR
Stefan and Fluffy added notes to the issue
Harald: PR #317 - data channel in workers
<vivien> Make RTCDataChannels Transferable #317
<jesup> :-)
Harald: marked as "later" already
Adam: easy to implement, easy to be compliant
Dom: plan to implement in FF?
Adam: yes even if not in the spec
Dom: that's what I wanted to know
Dom: worry about security aspect
Ekr: not think through yet
Stefan: assign to Ekr for security review
Dom: would it be in 1.0 if there is no security issue?
<jesup> This is a significant ask by DataCHannel users (see list posting by Faaborg(sp), and games users)
looks like the depedency on the security review
<jesup> "significant benefit to "
<jesup> ok
<jesup> thanks
<vivien> [Moving back to Simulcast discussion]
Fluffy: would like to discuss
simulcast issues, and decide what to include in 1.0, e.g.
framerate, maxbitrate, framesize, ...
... should start with framerate and bitrate
<vivien> RTP Payload Format Constraints draft-pthatcher-mmusic-rid-02
Justin: introduce frameratescale
Adam: or use an upper bound
Peter: framerate scale is better, a lot of cameras changes framerates
Justin: how this interact with the degradationperference?
<dom> [we're looking at the smallest set of properties for WebRTC 1.0, not dealing with all possible corner cases]
Bernard: for max framerate for each track, you can use separate clones
Dom: we are looking at the minimm set of encoding params for simulcast in 1.0. We don't need to look at all params at the point
Varun: smulcast should have streams with different properties, sender should choose which ones to send
Adam: adding a control to framerate is a good thing? then we can talk about what to put there
<jesup> framerate control makes sense
<vivien> +1 to jesup
<vr000m> +1 as well
Harald: if not converge, we should leave it out
<jesup> I generally happy with simple solutions, and if there are less-than-great corner cases, that's ok. In some cases, the app can push us out of them by noticing we're there from stats. In other cases we may have live with a corner case. I'm'm ok with that.
<vivien> from what I heard I think max-framerate is enough
Peter: should address the question from Justin on the degradation
<jesup> I'm pretty much ok with maxbitrate, and fill smaller layers first before higher layers. Send higher layer when there are "enough" bits to make sense to send it; up to the browser
Harald: Justin to help clarify the relationship with the degradationpreference, and will have discussion next week at IETF, Ekr and Justin will drive the decision
(coffee break, we resume at 16:00)
Media capture in Worker slides presented by Tzu-Hao
Tzu-Hao: introducing
mediacapture-worker
... the spec in in a GH repo: https://w3c.github.io/mediacapture-worker/
... the purpose of the proposal is to solve the following
problem:
... no efficient way to do video processing on the Web Platform
currently
[ showing a diagram of the current building blocks ]
Tzu-Hao: must use canvas element to modify, which is inefficient
Tzu-Hao: the problems: redundant canvas element, processing in the main thread, only pull-based mechanism
Martin: in FF, we implemented
capture in canvas
... canvas.captureString API
<jesup> http://mozilla.github.io/webrtc-landing/canvas_demo.html and http://mozilla.github.io/webrtc-landing/canvas_filter_demo.html
Martin: the current situation is
not pretty, I agree with the presenter
... we want a mechanism where we can quarantee frame
processing
... so that a frame is processed only once
... the third problem is that all the processing happens in the
main thread
... better have a mechanism to do video processing on the
worker thread
... core concepts:
... MediaStream with Worker
... for each frame a browser can dispatch an event at the
worker
... with design the problem is how a developer can access the
video frame data from the event dispatched at the worker
... the proposal is that in the event there are two properties:
inputBitmap and outputBitmap
... web developers can access the frame data through
inputBitmap
... the spec extends the ImageBitmap interface
... we want to get rid of the hidden canvas and ImageData in
the current processing pipeline
<jesup> can someone update us on what slide he's on as he goes? Thanks
Martin: only one copy of the real video source exists
ImageBitmapsFactories interface is extended
shijun: how can you quarantee you
get every frame only once?
... how can you ensure you can process the data in
realtime?
peterT: if the processing takes a really long time, e.g. face tracking, what happens?
Tzu-Hao: worker has a queue and if the queue is almost full the process will stop dispatch to the worker
Martin: the basic model is, you
take video frames from the underlying track, and you have an
object you mirror into worker, in which you can manipulate the
frame data
... advantages of canvas is that you have access to all canvas
APIs, so this means those optimization would not be
available
... you could do things like take the image rendered on canvas
and apply shaders there
mathieucitrix: you can draw to an offscreen canvas, and it stays in GPU
Tzu-Hao: use case: video
editing
... also camera effects, stream manipulation, computer vision,
depth
... demo
Cullen: how is this tied to the depth images
Related: Worklets, aka Processors aka Isolated Workers
https://drafts.css-houdini.org/worklets/
mathieucitrix: this is very important to me, I want to help improve the spec
[ asking if there is someone who does *not* care about the use cases presented ]
[ sees no hands raised ]
cullen: using processors is compelling, but having an issue for having invisible workers
Tzu-Hao: discussing open
issues
... that's a wrap
Martin: capture from element,
status:
... implemented in FF, close to complete
<jesup> http://mozilla.github.io/webrtc-landing/canvas_demo.html and http://mozilla.github.io/webrtc-landing/canvas_filter_demo.html are the demos
Martin: have one open issue,
thanks to Paul that's resolved soon
... next steps: consider progressing to CR
[ showing slides ]
Martin: unless someone else
implements, do not want to advance to CR
... not on a Chrome roadmap AFAIK
... Microsoft seems to have a plan to implement, not scheduled
yet
Harald: any test cases?
Martin: no w-p-t tests
... a compelling use case: put together an MCU in Firefox
<vivien> Media Capture from DOM Element Editor's draft
Martin: Firefox Nightly has this, and FF 43-ish
<vivien> Google said in https://code.google.com/p/chromium/issues/detail?id=524218#c3 "We started discussing possible implementations here: https://goo.gl/EtsTBG"
Martin: merge one PR to get a spec ready to publish an updated WD
Harald: how do capture from
canvas vs capture from <video> tag compare
... canvas is hugely useful for a variety of use cases
<vivien> Landed officially in Firefox 43 https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream#Browser_compatibility
[ presenting slides, screen sharing on slide 3]
<vivien> Media Capture Screen Sharing repository
Martin: last time we discussed
the distinction between window and application
... implemented this in the spec and figured our the
distinction is not much different from your window
Martin: REMOVE ALL constraints
<vivien> [ shijun give a thumb up ]
<vivien> [ discussion of the use case of enterprise products having an option that forbids sharing the whole screen, and a possible unsafe mode that would also include sharing the browser ]
cullen: requirement is if javascript can somehow indicate never to share unsafely
mt: others may not have the same
policy as cisco
... open an issue on this
cullen: when i ask for permission what do i get?
mt: browser is responsible for presenting ux, and contains complete list of browser, monitor, etc.
adam: if we went finer-grained about what we don't want to share we couldn't knock out all apps that way
cullen: you could get most for
most users
... so you want to ensure that user gets to control?
mt: virtually all the issues here
are related to selection constraints by apps
... we plan to include browers in the list but don't have all
the machinery yet
ekr: don't think filtering this
list will help at all because the app will suggest the user
allow everything
... is it okay to all of you that user has to do this
selection?
peter: yes, but not sure about distinctien between apps and tabs
shijun: want to be able to filter out certain resoultion sizes or crazy framerates. want to keep resolution from surface (no up or down scaling)
mt: you don't want to do resoluttion constraint, just keep what is provided. if someone asks for a restriction, then the track itself will be downscaled.
cullen: on the list of windows you might not want to display small windows
mt: that's the browser's responsibility
peter: you don't want app to even downscale at all?
cullen: good use case for this is in classrooms where teacher is getting many videos from all the students to make sure they stay on task
shijun: any dsp processing would damage the output, just give us the raw original stream
mt: but that would change the way mst works
hta: nice thing about keeping to the mst deifiniton is that then any downstream process expecting an mst can use it
shijun: might not implement rigth
away
... what about framerate
mt: won't offer a rate that browser can't handle
cullen: can you do less than 1 fps?
mt: yes, very important for some use cases
hta: i worry about taking away selection constraints because it makes the selection list too long unles the browser displays this intelligently
cullen: show me your UI. that will determine whether this is okay or not
mt: think we can do this well. have some mockups that are pretty good.
hta: other issues?
mt: permissions API adds a "screenshare" label with no extra parameters. Do not display allow state, only deny
shijun: what about notification?
mt: on screen indicators, easy revocation, if you need more raise an issue
cullen: if you capture app and it goes full screen, then what
mt: PPT case? Yes. Challenge was if you selected a window and then go full screen it generates a new window for that, so you lose it. You're sharing the old one, which might be gray. issue #29 is "powerpoint is special". we think the browser needs to just deal with this.
shijun: if you open PPT in full window it could even go to a different screen (projector) and that's what you want to get
mt: this may be simple to describe. monitor for new windows. if new window for app you're sharing is full screen that's it.
cullen: would also be nice in full screen sharing mode, sharing whole monitor, to be able to share the other monitor
mt: we have indicator for what is being shared. thinking about having that show a menu where you can select the source.
cullen: if you watch screen sharing failure, multi screen is the most interesting case
mt: may need some text on this in the doc
hta: what is implemenatiotn status?
mt: both ff and chrome have some sort of capability that doesn't match this API at all. No concrete plans to implement this. UX folks in particular thought this was complex, so still under discussion.
cullen: what about MSFT?
shijun: not this year. Next year I don't know (laughs). This is uunder my domain.
peter: I'd have to go ask at
Google. New owner for screencasting and will have to ask.
... will ask what is the plan for a non-extension and how does
the picker include all possibiliites and still remain
usable
mt: may want to do frequency sorting for list of apps so the most likely shows up at the top of the list
adam: yes, we have to deal with many many options
hta: main outstanding item seems to be UX design
[various jokes about listing PPT first ]
hta: time to adjourn?
cullen: one question: what to comumnicate to rtcweb chairs for next week's meeting?
hta: finish jsep!!!
cullen: authors, authors, authors
[and we are done!]