Media Capture Task Force Teleconference

<stefanh> proposed agenda http://lists.w3.org/Archives/Public/public-media-capture/2014Mar/0123.html

<dom> scribenick: mt

<stefanh> minutes http://lists.w3.org/Archives/Public/public-media-capture/2013Nov/0139.html

Minutes approval

stefanh: the minutes for approval

fluffy: these are the same minutes I had problems with?
... in which case I object

juberti: that might have been the previous meeting

stefanh: your objection was to webrtc minutes

fluffy: ah, no

ekr: proposes to postpone

Existing design of constraints

<burn> my slides: http://lists.w3.org/Archives/Public/public-media-capture/2014Mar/att-0158/constraints20140327.pdf

burn: will outline changes in the last year and the syntax that results
... one of the biggest changes is that the optional elements can now include multiple attribute and value pairs (as opposed to just one)
... this is the same value for "optional" as for "mandatory"
... there is a separate interface: Constrainable

<ekr> The problem is that the minutes are always bad

burn: ConstraintSet is the core property, in which you can have specific values, min/max, or enumerated
... the order of optional properties is significant
... applyConstraints and gUM are the two places that constraints (and Constrainable) is used

<fluffy> @ Stefan - you are right it was the WebRTC minutes I had a problem with. I have no idea if the minutes from last Media TF are right or wrong but I have no objections to them be approved if you feel like asking again at end of call

burn: applyConstraints is different from what is passed to gUM because applyConstraints is applied to a track - where you already know if it is audio or video
... all mandatory constraints MUST be satisfied; optional constraints are attempted, starting from the first
... this is also true when changing constraints

ekr: this is expanded as of previously, because you can specify sets of properties, instead of single properties

burn: correct
... an entirely constraint set must be satisfied together
... capabilities return multiple values (ranges or sets)
... settings (as applied) return a single value
... typo on slides s/1280/1080/

juberti: if you can't get 1080 exactly, but if you can get close, then you would fall back to 4:3.

burn: correct

juberti: would you get all values that a camera can do, or would you get a min-max range?

burn: min/max

juberti: why not specifics so that the application can choose directly?

<JimBarnett> it could be either - it depends on how the property is defined

burn: the complexity of returning multiple sets of values primarily
... combinatorial explosion

juberti: native applications manage
... just curious about the rationale

fluffy: one of the issues is that modern cameras have near infinite combinations

juberti: they have to be finite; the APIs provide enumeration capabilities

burn: fingerprinting concerns caused us to avoid providing specifics in the past
... we now have a process by which we can determine when it's safe to provide more surface area

juberti: how are unknown mandatory constraints handled? please summarize

<JimBarnett> unknown mandatory cnstraints must fail (per current spec)

burn: a mandatory property must be satisfied, else fail
... unclear on applyConstraints

ekr: this was my understanding and this is very important
... if we don't agree about this, we should stop here

<dom> "unknown property names will result in a ConstraintSet that can not be satisfied"

<ekr> that seems pretty clear

jib_: these are not dictionaries, right?

JimBarnett: no, they are not

jib_: where are the rules that define what is permitted and how to validate the values?

<juberti> I thought there was controversy on this point. Happy to hear that that is not the case

hta: the only reason we cannot use dictionary is that we can't check for unknown members on the object
... these are almost dictionaries

jib_: are these objects passed by value or reference?

hta: by-value

jib_: object is passed by reference

ekr: why is this an important question?
... is there a way to set a constraint and have the process fail if it is not met

fluffy: I think we already have agreement on that

juberti: wanted to clear up the point

<jib_> My slides: http://lists.w3.org/Archives/Public/public-media-capture/2014Mar/0142.html

Jan-Ivar's presentation

<juberti> http://lists.w3.org/Archives/Public/public-media-capture/2014Mar/att-0142/Constraints2014.pdf

jib: dictionaries don't allow for unknown or repeated members
... dictionaries are passed by value; IDL bindings do type checking
... objects are passed by reference in JS

<JimBarnett> NO! order doesn't matter within a constraint set

<JimBarnett> It's only when you list constraint sets that order comes in

<ekr> JimBarnett: are you going to make this point.

JImBarnett: order only matters if you have multiple constraint sets
... the entire set must be satisfied as a unit, if not you move on

jib_: prior to the latest change, there is ordering between constraints

burn: this was restricting us to a single property name-value, but it was always the case that those were independent and the optional array that provided the ordering

jib_: what is on slide 2 is not possible today

JimBarnett: then you would have 4 constraint sets in some order

burn: or more if you split min and max

jib_: this structure doesn't imply any order and allows the UA to choose

hta: the browser is allowed to do what it likes on any of these values

jib_: we might encourage the browser to pick the source that matches the most constraints

hta: so these are hints

adambe: agrees with Jan-Ivar that this is not possible today

<ekr> cullen… jan-ivar does have a syntax for that

<burn> adam, the behavior he describes is absolutely possible today, just with a different syntax than he shows

fluffy: the important point in this discussion is that being able to specify in the optional constraints which is most important to the app is the feature that is most important
... allowing the browser some latitude might be important, but there needs to be a way for the app to make some calls

<dom> burn, it seems to me that you can't have the exact equivalent since you *have* to provide an order in your optional constraint sets

burn: back in my day...

<adambe> burn: how? not by putting constraints in different elements or in the same element

burn: we had some browser freedom, but application developers wanted more control (something about mistrusting browsers)
... this is possible with the existing syntax

jib_: disagree. today you have to say that X is strictly more important than Y

<jesup> I would like to note that if the authors of the specs have to debate what is or is not even possible around fairly simple cases, we have developed an unusable spec. Developers will need a application just to build their constraint sets and be unable to know why they work or don't....

jib_: describes prefer and ideal extensions

burn: pass by value/reference is an irrelevant distinction; the current syntax is intended to work the same way

<juberti> ok, that was my question.

<dom> (I think the pass by value vs by ref is just another illustration of relying on WebIDL or our own micro-structure, not a specific argument)

jib_: objects are passed by reference; dictionaries are easier

JimBarnett: if the important thing is to get WebIDL, we could send a list of dictionaries and use the require keyword in addition

burn/JimBarnett: back and forth on some other proposal

<dom> [I'm not sure it's productive to build on the fly a WebIDL-compatible version of the current design of contraints]

jib_: Jim's proposal is to take Jan-Ivar's proposal and add it to an ordered list
... that's more complex, overkill

JimBarnett: wants expressiveness, complexity is justified based on that

jib_: where's the use case?

fluffy: difference is in the semantics that can be expressed, then syntax (and syntax is not important)
... these are largely converged, aside from having the same constraint multiple times (can't have separate optional and mandatory)
... sourceId is one place we might need multiples
... Jan-Ivar, would you be willing to consider things that cover these

jib_: concerned about it being webIDL

fluffy: that's easy to fix with a new syntax

jib_: for sourceId, you could provide a list/set of source IDs

<dom> dictionary MediaTrackConstraints : Constraints { … ConstrainDOMString sourceId; } with ConstrainDOMString defined as either a string or an array of strings

jesup: we have created a very complex language for specifying stuff, not quite Turing-complete
... expressivity might be useful for some, in which case, we've not done a good job of capturing requirements, because we can't evaluate proposals againstrequirements

<dom> +1 to all of what jesup says

jesup: the danger of defining the ur-solution is that it makes it harder to use
... it's not a great thing for the web if users need a helper app just to construct constraints

<burn> -1 to all of what jesup says

hta: I don't see these as especially complex; each proposal is a small purpose built language

<burn> agree with hta, this is not at all complex compared to what developers have to deal with when listing and specifying all device settings in detail in other languages

hta: we have a proposal for a reduction in functionality, on a basis that is not especially compelling
... what can we do that does the least amount of damage to what we have already discussed

<JimBarnett> +1 to hta

hta: Jan-Ivar's proposal is a reduction in espressiveness

fluffy: disagree with jesup and (dom?)
... these two things are almost the same
... argument from precedent

ekr: if/else if/else is useful, not sure if Jan-Ivar's proposal can address that

jib_: require addresses the mandatory problem, which is my main WebIDL beef
... tried to do both things, reducing expressiveness and fixing webIDL
... might look at the optional sequencing problem

fluffy: with the interaction with user prompts, not sure that we can avoid optional

(minute taker missed something here)

juberti: backwards compatibility story: both proposals require churn in implementations
... Jan-Ivar's proposal is more churn

<dom> [I'm not sure to understand why this causes more churn in implementation]

jesup: not saying either proposal is perfect, but there are requirements here that are not well captured and that makes it hard to evaluate
... we should have discussed these requirements
... I don't mind it being a language, but worry about complexity
... the question is: how many people are using syntax that would be affected by either proposal

burn: I think that the requirements are out there and have been discussed for a while already

<dom> [back in November, we said we should go back to listing requirements for constraints; this hasn't happened, and this seems to confirm this hadn't been done before]

burn: these capabilities are not just for local use, they might be for RTCPeerConnection; where changing network conditions might require the browser to act without application input
... effective constraints are dynamic, within the constraints specified by a developer

<jesup> burn: Congestion handling should be automatic, and not necessarily change the GUM resolution. That's a bad way to respond to congestion; changing scaling in PeerConnection is how to do that

<jesup> I suspect more developers will tend to overspecify when they try to use it and accidentally mess over users with configs they didn't expect

<dom> [another way to put it: reduced expressiveness is not an issue; sufficient expressiveness is what we should care about]

<JimBarnett> reduce expressiveness is not sufficient

<burn> jesup, changing scaling in PeerConnection will have an effect downstream, and that's exactly what I'm talking about

<JimBarnett> The current spec has the expressiveness we need

<dom> JimBarnett, I think the point is that the requirements of what is we need haven't laid down explicitly

<juberti> So far, I haven't seen anyone articulate a real-world use case requiring all the expressiveness possible here.

<juberti> Right now, people are using constraints to pick cameras, and choose HD vs VGA.

<JimBarnett> The one Dan gave has been there from the beginning

<juberti> That could be done with a less expressive proposal. ideal=1080p, min=480p

fluffy: this has been expressive in this way since 2012-08

<JimBarnett> +1 to Cullen

<JimBarnett> a different syntax with the same expressiveness is worth investigating

<JimBarnett> But I don't want to lose expressiveness

<juberti> I don't mind losing expressiveness and complexity that apps don't really need.

stefanh: we've talked about this, some people are concerned about complication, more people want expressiveness

<gmandyam> Will we be taking a vote on the proposal?

<juberti> especially if we can enumerate caps directly..

stefanh: we don't have any consensus for reducing expressiveness

<jesup> If I'm wrong about the requirements, great. Let's use that to evaluate them. But complexity is an issue as justin says

stefanh: would a more webIDL compatible API be worth pursuing?

JimBarnett: that would be good

burn: I would be fine with that, expressivity >> syntax

<jesup> I would be ok with reducing expressiveness (barring a good argument that it has to be maintained for important reasons), or to modify jib's proposal to add it

<juberti> likewise. the lack of conforming implementations at this point indicates that we won't have the desired expressiveness any time soon, even with the current proposal.

<jesup> I would prefer to simplify, but am ok if we have strong reasons for needing complexity

stefanh: will ask jib_, JimBarnett, burn to work something out

<gmandyam> Agree with jesup's previous comment on reducing expressiveness, or modifying jib's proposal to add it

<jesup> I'd like to see a definition of *what* expressiveness is needed, and roughly why

<gmandyam> We have a Use Cases/Requirements doc - we should be matching up requirements for expresiveness with that doc's use cases

<dom> expressiveness isn't free; it comes with a cost in complexity and implementation, which should be justified and balanced by the use cases they enable

<burn> dom, so should dramatically changing something that has largely been in the spec for a *very* long tim

<juberti> are we going to discuss Constrainable as well?

<gmandyam> My guess is that there are no use cases in the Use Case/Requirements doc that require expressiveness

fluffy: we want to be done, we need to ensure we don't lose anything

jesup: this is the first time it's really been implemented

stefanh: we'll give this group a few weeks to come up with something

fluffy: if we are going to change what semantics can be expressed, I'm not interested
... changing the syntax should be easy

dom: can we get use cases that require this broader expressiveness, even if we don't start from the position that we are changing expressiveness

burn: this is annoying, it will take ages to sort this out

dom: coming up with one use case would be simple

juberti: expressiveness isn't holding applications back
... or lack thereof

fluffy: we don't have *any* use cases
... driving from use cases is too slow

<dom> [hmm... I think it might be slow *because* we don't have use cases and requirements]

fluffy: fixing the syntax should be quick

<gmandyam> Just because we have not been actively revising https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/scenarios.html does not mean that the doc does not exist. It can be revised.

<jib_> see slide 10

<burn> fluffy's comment about use cases was that none of our features have a use case and requirements document. He then made clear that the use cases for the current level of expressivity *have* been sent to the mailing list multiple times over the years.

hta: proposes that we fix the current proposal, then talk about reducing expressiveness

<fluffy> +1

<juberti> But with Capabilities, we now have some alternate ways of dealing with this

<juberti> I think it would be nice if we could let JS control this stuff instead of the constraints mini-language

<burn> Justin, I suspect you would never get agreement with jib on direct control of device characteristics, esp. for devices shared across tabs, apps, etc.