IRC log of mediawg on 2023-06-13

Timestamps are in UTC.

20:55:49 [RRSAgent]
RRSAgent has joined #mediawg
20:55:53 [RRSAgent]
logging to https://www.w3.org/2023/06/13-mediawg-irc
20:55:53 [Zakim]
Zakim has joined #mediawg
20:56:34 [cpn]
Meeting: Media WG meeting
20:57:17 [cpn]
Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2023-06-13-Media_Working_Group_Teleconference-agenda.md
21:01:38 [cpn]
present+ Chris_Needham, Jer_Noble, Sun_Shin, Peter_Thatcher, Bernard_Aboba, Greg_Freedman,
21:01:44 [cpn]
Chair: Jer_Noble, Chris_Needham
21:01:47 [cpn]
scribe+ cpn
21:04:04 [cpn]
present+ Jean-Yves_Avenard
21:05:43 [cpn]
Topic: Coding tool configuration
21:06:18 [cpn]
Bernard: For AV1 we added the ability to configure screen coding tools. We found an interesting gap, looking at other needs
21:06:44 [cpn]
... We thought WebCodecs would not need to consumer content hinds, for audio things like speech / music
21:07:19 [cpn]
... and similar for video. We noticed that it wouldn't work in WebRTC
21:07:39 [cpn]
... Looking at the Chromium source, this is how WebRTC optimises for screen sharing
21:08:02 [cpn]
... For h.264 it sets a screen content realtime mode, and changes the QP range
21:08:14 [cpn]
... We have Per-Frame QP in WebCodecs for AVC
21:08:41 [cpn]
... For AV1 it enables the palette control, and turns on screen content mode tuning
21:09:01 [cpn]
... Interesting is screen content realtime control. It's implementation dependent, it's in Open H264
21:09:16 [cpn]
... So can't put it in the registry as it's only in Open H264
21:09:52 [cpn]
... Does it make sense for WebCodecs to consume a content hint to set the screen content realtime control
21:09:53 [cpn]
... https://github.com/w3c/webcodecs/issues/478
21:10:14 [cpn]
... There's an analog for audio, a speech / music hint
21:10:26 [cpn]
Eugene: In libvpx there's tuning for certain content types
21:11:02 [cpn]
... So there's probably more. We couldn't put all the possible switches in the registry. They're not video codec specific, but specific to the specific encodig library
21:11:25 [cpn]
... If they're just hints, and so up to the UA to use or not, we should totally do this
21:11:50 [cpn]
Bernard: So we're agreeing that we don't change anything in the registries, so doesn't affect per-frame QP?
21:12:15 [cpn]
Eugene: Yes. It would just be a hint in the encoder config. If nothing interferes with the hint, the UA can use it as it thinks necessary
21:12:45 [cpn]
Bernard: Would be concerned if the content hint moved the per-frame QP, that would be weird
21:13:26 [cpn]
Jean-Yves: Why pass it to WebCodecs? Would it be internal to the WC implementation? Why pass the data if you've configured the encoder or decoder for the particular site?
21:13:27 [markw]
markw has joined #mediawg
21:13:59 [cpn]
Bernard: THe content hint is a property of the track. No equivalent property on the Video Frame, so there's no current way to pass it as an argument
21:14:14 [cpn]
... Eugene is proposing we have a way to pass it to the encoder config, as it can't be done to day
21:14:50 [cpn]
Jean-Yves: Is was talking about modifying the encoder itself, as this would be for real-time sharing for example
21:15:25 [cpn]
... You could have two encoders. You can implement with what there is today, an encoder for screen sharing or an encoder for non-screen share
21:16:00 [cpn]
... I can see this evolving, adding lots of config over time for particular use cases, could be endless
21:16:25 [cpn]
Eugene: To some extent, it's possible we'll add options if the WG thinks they're useful
21:16:53 [cpn]
... Not sure what you mean by multiple encoders. The website needs a way to signal to the UA to be prepared to encode text or screenshare, or to encode camera input
21:17:05 [cpn]
... This is a way for the site to signal to the UA what kind of encoder it needs
21:17:22 [cpn]
... So what you say doesn't contradict the proposal. IIUC you're describing an implementation detail
21:18:14 [cpn]
... WebCodecs has just a single VideoEncoder class, so pass in config
21:18:29 [cpn]
Jean-Yves: I see the problem now, OK
21:19:14 [cpn]
Peter: I think it's a good approach, I like that concrete things override this, this changes the default behaviour
21:19:50 [cpn]
... Makes sense as a higher level control rather than lots of little
21:20:06 [cpn]
Chris: Agree with Peter, higher level seems preferable
21:20:21 [cpn]
Bernard: We'd blow up the registries if we added every possible option
21:20:40 [cpn]
Eugene: I wondered if we need the AV1 content tools in the registry
21:21:14 [cpn]
Bernard: The question is whether the content hint is taken into account. screen content can be disruptive
21:22:02 [cpn]
Chris: To clarify, this would apply to all codecs, and just be applied if there's the right implementation available
21:23:12 [cpn]
Bernard: Yes, but with checks that audio hints don't go to the video encoder and vice verse
21:23:23 [cpn]
... And include also for audio?
21:23:29 [cpn]
Peter: Would make sense, yes
21:23:54 [cpn]
Chris: Hearing no objection, sounds like that's a conclusion
21:24:05 [cpn]
Topic: ManagedMediaSource
21:24:42 [cpn]
Chris: Nice presentation, Jean-Yves!
21:25:55 [cpn]
... Want to discuss Marks' questions, and then talk about PRs and editing
21:26:28 [cpn]
Jer: Start with an quick refresher on the proposal?
21:26:44 [cpn]
Jean-Yves: We wanted to have an ability to enable MSE on iPhone
21:27:06 [cpn]
... We had concerns about battery life in comparison to HLS, so we wanted to find a way to add hints to the player
21:27:18 [cpn]
... to give guidance so the player fetches media at specific and given times
21:27:40 [cpn]
... the most power hungry aspect of players is fetching data, turning on the 5G modem which would drain power
21:27:56 [cpn]
... Initially we thought we'd need to enforce only appending data at a given time
21:28:10 [cpn]
... But it turned out that wasn't necessary to achieve our power saving goal
21:28:40 [cpn]
... We try to provide hints on when to fetch data and when to stop, so the MediaSource is a bit less free-reigned, and reduce power usage
21:29:13 [cpn]
... We added ability to evict content at any time, in low memory situations, rather than killing the page
21:29:40 [cpn]
... ManagedMediaSource in WebKit, they're hints only
21:30:16 [cpn]
... Internally, when the streaming attributes is true, we tag all requests as "media". Turn on the 5G modem
21:30:41 [cpn]
... When we send the endstreaming event, we stop tagging reqeusts and the modem goes into low power state
21:31:10 [cpn]
... We looked at what's useful in HLS, try to provide hints so MSE would behave in a similar fashion
21:31:26 [cpn]
... Hints to request a particular quality could be a fingerprint issue, so may not work
21:31:59 [cpn]
... The original idea was how to make MSE more like HLS to make some manufacturers happy, including the one I work for
21:32:00 [cpn]
q?
21:32:52 [cpn]
Mark: Sounds cool, good to see MSE coming to iPhone
21:32:56 [cpn]
Chris: +1 Mark
21:33:24 [cpn]
Mark: My comments are exploring options. The buffer removed events, to give the site more info on what's in the buffer, seem independent and could be standalone
21:33:36 [cpn]
... Could they be added to MediaSource?
21:33:59 [cpn]
... For discovery, adding the event handler could be enough
21:34:06 [cpn]
... Would it make sense to separate this off?
21:34:26 [cpn]
Jean-Yves: From an implementation point of view, moving that event is not a technical challenge
21:34:46 [cpn]
... But just the event, not when the eviction can occur?
21:34:58 [cpn]
Mark: Yes, just events telling you when eviction has happened
21:35:23 [cpn]
Jean-Yves: Putting it inside ManagedMediaSource, makes it clear you must listen to the event or the player will stall
21:35:40 [cpn]
... With MSE you know when it will be removed
21:36:09 [cpn]
Mark: So as well as the events for data evicting, you're also adding the ability to evict at other times, not just during the two MSE operations
21:36:26 [cpn]
Jean-Yves: The site must listen to the event, otherwise it could stall
21:36:44 [cpn]
Mark: So in using ManagedMediaSource you're giving permission to the UA to do that. Makes sense
21:37:14 [cpn]
Jean-Yves: I believe it greatly helps the player. Every single library works on a time range, so needs to manage the buffer
21:37:25 [cpn]
... So moving it would lose that hint
21:37:46 [cpn]
Mark: So I don't have a strong opinion either way now, you could move it or not
21:38:16 [cpn]
Jean-Yves: Google at the time suggested it could be nice to get the site itself to evict. The issue there was there could be multiple players, e.g., in hidden or suspended tabs
21:38:38 [cpn]
... When there's memory pressure you have to react quickly, so needs to be in the UA to have that response time
21:39:33 [cpn]
Mark: The MSE design fully decouples downloading content from buffering. The design makes no assumptions about what you're downloading and when you're appending
21:40:08 [cpn]
... We've found that extremely useful at Netflix, lots of use cases enabled. So I get nervous that we might curtail sites ability to do those things
21:40:42 [cpn]
... So it occurred to me, that there's a duty cycle for using the expensive modem could be useful outside of streaming, so talk about it for downloads also
21:40:58 [cpn]
Bernard: RTCDataChannel and WebTransport are also used with MSE, so not just HTTP adaptive streaming
21:41:20 [cpn]
Jean-Yves: The assumtion with MMS was to have something very lightweight, replace the MS in the player with MMS
21:41:42 [cpn]
... All are hints, provide as little change over MSE as possible, everything event driven rather than use promises
21:42:16 [cpn]
... It's an MSE extension in some way. Your question makes sense, they way the proposal is phrased is having a way to say: if you don't follow hints we'll block you
21:42:28 [cpn]
... But it evolved in implementation that this wasn't necessary
21:43:05 [cpn]
... If you call appendBuffer and streaming is not allowed, we could throw. Would you still have the same reservation?
21:43:21 [cpn]
Mark: Blocking the append would make it problematic
21:43:40 [cpn]
... I see three distinct things going on, so could be useful to separate out. Those are:
21:44:13 [cpn]
... The start/stop streaming events, that tell you when you have more efficient transport. Those could go on Window, and time your downloads for any purpose
21:44:46 [cpn]
... There's some logic not in the proposal that's UA specific, to choose when to send those events (turn the radio on/off). That could take into account whether you've created a MMS and how much is buffered
21:45:41 [cpn]
... The site knows what it has buffered, could the site inform the UA more of the buffer state to inform when to turn the radio on/off
21:45:47 [cpn]
...The third thing is XXX
21:46:26 [cpn]
Jean-Yves: There's many ways we could skin the cat. The idea of having it all in one place in MMS makes it easier, rather than having multiple places
21:46:50 [cpn]
Mark: So why is it specific to streaming? Other applications would benefit
21:47:31 [cpn]
Jer: With implementer hat on, we found there are some cellular networks that give benefits when streaming media over 5G, e.g., counts against your data cap
21:47:46 [cpn]
... So we wanted to be careful to not waste battery life and not waste people's money
21:48:24 [cpn]
... Your ideas for efficient network traffic management aren't bad ones, but we don't have control over that in this WG
21:49:01 [cpn]
... It's worth bringing to the people who manage the fetch spec, but we can't do that here. Similar on WebTransport and WebRTC DataChannel
21:49:29 [cpn]
Mark: I see the motivation to get this working for streaming. So why not put the start/stop events on Window, to generalise it a bit?
21:50:18 [cpn]
Jer: Sure, this is very particular to the device we were developing on. Some things we're not in control of. We may not want to expose to the entire web when the modem is available or not
21:50:38 [cpn]
... So we had to ride a fine line in selecting a heuristic
21:51:24 [cpn]
Jean-Yves: There's a strong link between the managed media source buffer and the streaming value. So it made more sense to have it in MMS, those events are for when streaming with MMS and only used with MMS
21:51:50 [cpn]
Mark: Feels like a bit too much is left without guidance to the site about how it might work. Don't want to constrain how the UA would behave
21:52:27 [cpn]
... Need guidance for sites. If I append a lot of data, the start/stop streaming events will slow down. I need to know that in my client code, to balance my media vs other requests
21:52:39 [cpn]
... Give some heuristics or guidance on what to expect
21:52:55 [cpn]
Jer: Sounds like you want a non-normative note on how to implement?
21:53:26 [cpn]
Mark: That might be sufficient. May be we'd put all our requests int he start/stop region
21:54:24 [cpn]
Jer: One of the reasons for the MMS name, is a similar problem is faced in Apple WebKit, Chromium, and set top boxes. Much more limited buffers in those devices
21:54:42 [cpn]
... They'll let you do complicated things, such as lots of buffers in JS, but the available memory is much less
21:55:04 [cpn]
... The goal of the MMS design is to make it work on those kinds of devices too
21:55:47 [cpn]
Mark: Use cases where you dont' want to append right away. Example, reaching an end of episode and likely to want to watch the next episode rather than the end credits
21:56:01 [cpn]
... I have a bunch of buffers, haven't appended anything
21:56:57 [cpn]
... If the only effect is using more battery, that's fine, so long as its rare. Problematic would be if you're not giving the resource I need
21:57:12 [cpn]
Jer: You mentioned in the thread giving more info about what you're doing
21:57:56 [cpn]
... A scenario where it won't work is live video, where you append every 10 seconds. In that case the page won't be obeying the start/stop events, the UA could fall back to a more efficient transport mechanism
21:58:26 [cpn]
... Earlier we had an explicit signal to say the client would be doing live streaming, and use the more efficient modem up front, when loading small amounts of data more frequently
21:59:00 [cpn]
... In your example, your could signal that you're done. Those could be useful additions to MMS
21:59:48 [cpn]
Jean-Yves: With live streaming, whether we implemented or not, the heuristic was identical. No need for it
22:00:06 [cpn]
... With knowing when you won't append more data, can you signal that
22:00:29 [cpn]
Mark: But in this case you would still be appending
22:01:22 [cpn]
... Happy to do the editing if there's PR
22:01:47 [cpn]
Jean-Yves: I can volunteer to edit if Matt cannot
22:03:05 [cpn]
Chris: We also need to think about it, important use cases such as live stream, branching narratives, etc
22:03:31 [cpn]
Jean-Yves: Aim here was to give UA a bit more control, it's not an ultimate solution, could have different objects, but that wasn't our aim
22:04:01 [cpn]
... Our aim was to enable on low memory constrained devices, by extending MSE. I hope we've achieved this
22:04:52 [cpn]
Jer: I have some feedback from WAVE, so the people who work on STBs are aware and interesting
22:08:14 [cpn]
Chris: So with my MEIG chair hat on, let's set up a discussion so interested WAVE people can ask questions
22:08:18 [cpn]
Jer: Sounds good
22:09:01 [cpn]
... I also want to thank Mark for the productive discussions in the GitHub thread. Good suggestions that can lead to wider further improvement
22:09:06 [cpn]
Topic: Next meeting
22:09:41 [cpn]
Chris: Scheduled for July 11, we'll see how discussion in issues goes, can meet sooner if needed
22:09:49 [cpn]
[adjourned]
22:10:07 [cpn]
Regrets: Dale_Curtis, Francois_Daoust
22:10:17 [cpn]
rrsagent, draft minutes
22:10:18 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/06/13-mediawg-minutes.html cpn
22:10:25 [cpn]
rrsagent, make log public
22:17:10 [cpn]
present+ Eugene_Zemtsov
22:17:12 [cpn]
rrsagent, draft minutes
22:17:13 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/06/13-mediawg-minutes.html cpn
22:24:43 [cpn]
present+ Mark_Watson
22:24:45 [cpn]
rrsagent, draft minutes
22:24:47 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/06/13-mediawg-minutes.html cpn