Meeting minutes
MSE #306
Matt: Chrome implementation is based on Serializable.
Chris: Was hoping to be able to discuss today with other implementers
Matt: Getting feedback for another approach would be unfortunate at this point
… The implemention relies on WebCodecs VideoFrame. The hope is Serializable is not blockable. Approach is to initially keep the API narrow and restrict flexibility (to not "bake-in" API that allows too much only to restrict it later), then relax those if needed.
… If it needs to become Transferable, we can add that on later
Chris: Do we need meeting time?
Matt: We've tried a couple of times. Getting responses from Mozilla, but having Apple chime in would be good
Chris: I'll follow up with Jer
Media Session
Chris: Welcome Tommy and Youenn as co-editors
Tommy: I haven't sent a change to update the editors but will do soon
Chris: Current discussion on Media Session and WebRTC Capture Handle Actions
… Question about the appropriate scope for Media Session. In particular. whether next slide / previous slide actions should be in Media Session?
Tommy: In my mind, it doesn't include that, doesn't really match. But open to be convinced otherwise
Chris: Is that because the actions come from the page rather than browser chrome?
Tommy: Chrome sees Media Session being related to Audio Focus, so including slide decks doesn't really fit that
… On whether they should be sent between origins, I don't have strong feelings, but Chrome security folks may have
Chris: I don't block progress on the WebRTC side, so determining the scope would help unblock
Tommy: If they're of a similar opinion, I'd recommend WebRTC progress with its own solution
Chris: There's a number of open issues, for different considerations
… I'd like to review those to see which parts do fit Media Session
… Open issues at https://
Chris: What's the current thinking on Audio Focus, mentioned in #277
Tommy: I'll have to check
Chris: Is the Media Session spec clear about a video conference session vs a media playback session?
Tommy: It depends on the user agent, depending on what surface is used. We use some last focused logic to route actions
… Up to the user agent to decide which actions to route to which tab
Chris: Want to get to a proposed resolution for Capture Handle Actions and identify the appropriate integration points with Media Session
Jer: Doesn't seem different in Safari. Tab most recently played back with a user gesture will be those to receive Media Session commands
Chris: Routing of telephony related actions
Jer: That's missing from the web generally. Audio Focus, lets the page determine which kind of audio is happening. Media Session would be the first place to introduce that
… Doesn't seem to make sense. Audio Focus could be used by Media Session, useful outside of Media Session. It's just an explainer, not even in WICG
… We've had requests for using an audio element for a notification sound, to change how it ducks or interrupts other playing audio
… Media Session wouldn't help there. My opinion as an implementer, is if we want to add some signalling that audio playback isn't long form audio, Audio Focus would be a better place to do that, different kinds of generated audio
Tommy: That's useful, thanks for the historical context
Jer: I believe Becca was working on it, but don't know if someone else picked it up
Chris: Need for a video conferencing session concept, and where should that be specced. Is it in scope here?
Jer: A UA knows who's doing capturing and in what tab. It should also know if a page has registered capture related session actions
… Do we need any explicit signalling from the page? If no capture is happening (if you're the remote listener, recipient of frames), that's the one place that may not know there's a conference happening
Chris: Please also have a look at the open issues, 274 onwards
MSE #306 (revisited)
Matt: We have some feedback from Mozilla on MSE in workers using a handle
… I've landed the implementation described in spec. I have a response last night from Mozilla, make it Transferable for real, not use Serializable
… From VideoFrame, we found Serializable has the necessary hooks to apply the restrictions I need for the handle, one per MediaSource
… When you postMessage it, it can't be postMessaged further
… As soon as the handle is set on a srcObject attribute, it's marked as used. Otherwise There's a race condition. Once set, prevent serialization
… The underlying MediaSource can enforce it's only attached once, and once simultaneuosly
… Seems a no-brainer that it works with serialization. Issues with making it Transferable, harder to implement a transferable MediaSourceHandle
… Have a narrowly defined case where it works, can only use the handle once. If we need it to be transferable, make it narrow now, relax later
… Would like your input on Transferable vs Serializable. Folks are clamouring for this
Jer: What behaviour would be different between those?
Matt: Can serialize, which can throw exceptions, InvalidState. The app would use the handle and assign it to a srcObject attribute on a media element
… That triggers the resource selection. You don't get a synchronous load attempt, that happens later
… The approach I took to prevent surprises and interop issues is: when it's assiged to a srcObject, the handle cannot be serialized again
… If it's ever been attached. If we change it later, would be surprising. If we want multiple handles from a MediaSource, could do later
Jer: For the use cases where someone wants to swap sources on a media element, you can't use the same object URL twice?
Matt: Yes. Jer: Does MSE in Workers say you can get multiple handles?
Matt: No, you'd need a different MediaSource
Jer: There's a mechanism where websites do ad insertion, creating an ad source. Can do today using multiple object URLs
Matt: So you'd have to create a new MediaSource
… When you detach, all SourceBuffers are removed, goes into the closed state
… So we may have over-restricted, but that's not in contention now
Jer: It's related insofar as if you want to support the same behaviour, you'd need a spec to create multiple object handles
Matt: One object per media source should help apps. not like shared ownership of MediaSource. The at most one handle for the MediaSource also has a reference to it
… Helps implementers clarify, and developers understand what they need to free up
Jer: Works differently to setting srcObject null? There's a quirk in HTML about src with empty string, where you have to explicitly invoke the resource selection algorithm
Matt: May be a difference in how we implement. Any time the src is set we try to load it.
Jer: I'll have to check that quirk. We may have to change Safari's implemntation
… So the big difference between Serializable and Transferable is with Serializable we have to add logic
… Is there a page-visible difference between Serializable and Transferable
Matt: There are broadcast channels etc where there may be differences
Jer: Is this a spec argument, or how its used in practice?
Matt: It's a spec interop question. Want to avoid the pitfalls we hit in WebCodecs VideoFrame
Jer: That suggests there is a page-visible difference
Matt: When I say interop, I mean is it implementable in Chrome
Jer: I'm thinking of cross-browser interop
Matt: The way you do the transfer is signalled by the app. postMessage takes Transferables
… Another way is if it's Transferable and not Serializable, but disagreement or misunderstanding on that
Jer: It seems, if you're making something Serializable but same as Transferable, weird as an implementer to have Transferable Serializable
… I don't know if I care that much
Matt: Wanted to have move semantics for VideoFrame in WebCodecs, but this detail made it impossible to implement in practice
… You cannot practically implement move semantics
… So if you trap it at the serialization point, or if already assigned to a srcObject. Don't feel there's an issue on those constraints. Is Transferable practically going to be possible
Jer: A handle can exist in any context, so you'd need to be able to transfer to a shared worker and back
Matt: The interface is only on DedicatedWorker and Window. Would be interesting to know if the IDL bindings in Chrome prevent SharedWorker
… If you're not doing a postMessage it won't work, makes no sense to put a handle into a database
Jer: Frameworks do weird things, it's use cases someone might try
… I don't have a strong opinion. From a high level point of view, Transferable makes sense as the semantics match
… If it's unimplementable, then fine. I'm a little worried that we'll face resistance if we need to change it later, if handles need to be both Transferable and Serializable
… I could pull in some of the JavaScript people for input
TPAC planning
Chris: Planning Media WG for Friday afternoon, joint meetings with WebRTC, TTWG, MEIG on DataCue
Jer: There's also capabilities and CSS, not necessarily for a joint meeting
… Audio playback capabilities, channels supported by user agents, in Audio WG. Spatialization flag in MC API, if you plug and unplug headphones, could change
… CSS doesn't handle audio
Chris: There's the HDR video plane issue open with CSS. Could request time in their agenda rather than a joint meeting
… Suggestions welcome for how best to use our meeting time. Prioritised issues, new incubations (could be for the M&E IG meeting)
Jer: Those suggestions sound reasonable
Next meeting
Chris: July 12. Possibly another joint meeting before to follow up with WebRTC on Capture Handle Actions
[adjourned]