20:55:49 RRSAgent has joined #mediawg 20:55:53 logging to https://www.w3.org/2023/06/13-mediawg-irc 20:55:53 Zakim has joined #mediawg 20:56:34 Meeting: Media WG meeting 20:57:17 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2023-06-13-Media_Working_Group_Teleconference-agenda.md 21:01:38 present+ Chris_Needham, Jer_Noble, Sun_Shin, Peter_Thatcher, Bernard_Aboba, Greg_Freedman, 21:01:44 Chair: Jer_Noble, Chris_Needham 21:01:47 scribe+ cpn 21:04:04 present+ Jean-Yves_Avenard 21:05:43 Topic: Coding tool configuration 21:06:18 Bernard: For AV1 we added the ability to configure screen coding tools. We found an interesting gap, looking at other needs 21:06:44 ... We thought WebCodecs would not need to consumer content hinds, for audio things like speech / music 21:07:19 ... and similar for video. We noticed that it wouldn't work in WebRTC 21:07:39 ... Looking at the Chromium source, this is how WebRTC optimises for screen sharing 21:08:02 ... For h.264 it sets a screen content realtime mode, and changes the QP range 21:08:14 ... We have Per-Frame QP in WebCodecs for AVC 21:08:41 ... For AV1 it enables the palette control, and turns on screen content mode tuning 21:09:01 ... Interesting is screen content realtime control. It's implementation dependent, it's in Open H264 21:09:16 ... So can't put it in the registry as it's only in Open H264 21:09:52 ... Does it make sense for WebCodecs to consume a content hint to set the screen content realtime control 21:09:53 ... https://github.com/w3c/webcodecs/issues/478 21:10:14 ... There's an analog for audio, a speech / music hint 21:10:26 Eugene: In libvpx there's tuning for certain content types 21:11:02 ... So there's probably more. We couldn't put all the possible switches in the registry. They're not video codec specific, but specific to the specific encodig library 21:11:25 ... If they're just hints, and so up to the UA to use or not, we should totally do this 21:11:50 Bernard: So we're agreeing that we don't change anything in the registries, so doesn't affect per-frame QP? 21:12:15 Eugene: Yes. It would just be a hint in the encoder config. If nothing interferes with the hint, the UA can use it as it thinks necessary 21:12:45 Bernard: Would be concerned if the content hint moved the per-frame QP, that would be weird 21:13:26 Jean-Yves: Why pass it to WebCodecs? Would it be internal to the WC implementation? Why pass the data if you've configured the encoder or decoder for the particular site? 21:13:27 markw has joined #mediawg 21:13:59 Bernard: THe content hint is a property of the track. No equivalent property on the Video Frame, so there's no current way to pass it as an argument 21:14:14 ... Eugene is proposing we have a way to pass it to the encoder config, as it can't be done to day 21:14:50 Jean-Yves: Is was talking about modifying the encoder itself, as this would be for real-time sharing for example 21:15:25 ... You could have two encoders. You can implement with what there is today, an encoder for screen sharing or an encoder for non-screen share 21:16:00 ... I can see this evolving, adding lots of config over time for particular use cases, could be endless 21:16:25 Eugene: To some extent, it's possible we'll add options if the WG thinks they're useful 21:16:53 ... Not sure what you mean by multiple encoders. The website needs a way to signal to the UA to be prepared to encode text or screenshare, or to encode camera input 21:17:05 ... This is a way for the site to signal to the UA what kind of encoder it needs 21:17:22 ... So what you say doesn't contradict the proposal. IIUC you're describing an implementation detail 21:18:14 ... WebCodecs has just a single VideoEncoder class, so pass in config 21:18:29 Jean-Yves: I see the problem now, OK 21:19:14 Peter: I think it's a good approach, I like that concrete things override this, this changes the default behaviour 21:19:50 ... Makes sense as a higher level control rather than lots of little 21:20:06 Chris: Agree with Peter, higher level seems preferable 21:20:21 Bernard: We'd blow up the registries if we added every possible option 21:20:40 Eugene: I wondered if we need the AV1 content tools in the registry 21:21:14 Bernard: The question is whether the content hint is taken into account. screen content can be disruptive 21:22:02 Chris: To clarify, this would apply to all codecs, and just be applied if there's the right implementation available 21:23:12 Bernard: Yes, but with checks that audio hints don't go to the video encoder and vice verse 21:23:23 ... And include also for audio? 21:23:29 Peter: Would make sense, yes 21:23:54 Chris: Hearing no objection, sounds like that's a conclusion 21:24:05 Topic: ManagedMediaSource 21:24:42 Chris: Nice presentation, Jean-Yves! 21:25:55 ... Want to discuss Marks' questions, and then talk about PRs and editing 21:26:28 Jer: Start with an quick refresher on the proposal? 21:26:44 Jean-Yves: We wanted to have an ability to enable MSE on iPhone 21:27:06 ... We had concerns about battery life in comparison to HLS, so we wanted to find a way to add hints to the player 21:27:18 ... to give guidance so the player fetches media at specific and given times 21:27:40 ... the most power hungry aspect of players is fetching data, turning on the 5G modem which would drain power 21:27:56 ... Initially we thought we'd need to enforce only appending data at a given time 21:28:10 ... But it turned out that wasn't necessary to achieve our power saving goal 21:28:40 ... We try to provide hints on when to fetch data and when to stop, so the MediaSource is a bit less free-reigned, and reduce power usage 21:29:13 ... We added ability to evict content at any time, in low memory situations, rather than killing the page 21:29:40 ... ManagedMediaSource in WebKit, they're hints only 21:30:16 ... Internally, when the streaming attributes is true, we tag all requests as "media". Turn on the 5G modem 21:30:41 ... When we send the endstreaming event, we stop tagging reqeusts and the modem goes into low power state 21:31:10 ... We looked at what's useful in HLS, try to provide hints so MSE would behave in a similar fashion 21:31:26 ... Hints to request a particular quality could be a fingerprint issue, so may not work 21:31:59 ... The original idea was how to make MSE more like HLS to make some manufacturers happy, including the one I work for 21:32:00 q? 21:32:52 Mark: Sounds cool, good to see MSE coming to iPhone 21:32:56 Chris: +1 Mark 21:33:24 Mark: My comments are exploring options. The buffer removed events, to give the site more info on what's in the buffer, seem independent and could be standalone 21:33:36 ... Could they be added to MediaSource? 21:33:59 ... For discovery, adding the event handler could be enough 21:34:06 ... Would it make sense to separate this off? 21:34:26 Jean-Yves: From an implementation point of view, moving that event is not a technical challenge 21:34:46 ... But just the event, not when the eviction can occur? 21:34:58 Mark: Yes, just events telling you when eviction has happened 21:35:23 Jean-Yves: Putting it inside ManagedMediaSource, makes it clear you must listen to the event or the player will stall 21:35:40 ... With MSE you know when it will be removed 21:36:09 Mark: So as well as the events for data evicting, you're also adding the ability to evict at other times, not just during the two MSE operations 21:36:26 Jean-Yves: The site must listen to the event, otherwise it could stall 21:36:44 Mark: So in using ManagedMediaSource you're giving permission to the UA to do that. Makes sense 21:37:14 Jean-Yves: I believe it greatly helps the player. Every single library works on a time range, so needs to manage the buffer 21:37:25 ... So moving it would lose that hint 21:37:46 Mark: So I don't have a strong opinion either way now, you could move it or not 21:38:16 Jean-Yves: Google at the time suggested it could be nice to get the site itself to evict. The issue there was there could be multiple players, e.g., in hidden or suspended tabs 21:38:38 ... When there's memory pressure you have to react quickly, so needs to be in the UA to have that response time 21:39:33 Mark: The MSE design fully decouples downloading content from buffering. The design makes no assumptions about what you're downloading and when you're appending 21:40:08 ... We've found that extremely useful at Netflix, lots of use cases enabled. So I get nervous that we might curtail sites ability to do those things 21:40:42 ... So it occurred to me, that there's a duty cycle for using the expensive modem could be useful outside of streaming, so talk about it for downloads also 21:40:58 Bernard: RTCDataChannel and WebTransport are also used with MSE, so not just HTTP adaptive streaming 21:41:20 Jean-Yves: The assumtion with MMS was to have something very lightweight, replace the MS in the player with MMS 21:41:42 ... All are hints, provide as little change over MSE as possible, everything event driven rather than use promises 21:42:16 ... It's an MSE extension in some way. Your question makes sense, they way the proposal is phrased is having a way to say: if you don't follow hints we'll block you 21:42:28 ... But it evolved in implementation that this wasn't necessary 21:43:05 ... If you call appendBuffer and streaming is not allowed, we could throw. Would you still have the same reservation? 21:43:21 Mark: Blocking the append would make it problematic 21:43:40 ... I see three distinct things going on, so could be useful to separate out. Those are: 21:44:13 ... The start/stop streaming events, that tell you when you have more efficient transport. Those could go on Window, and time your downloads for any purpose 21:44:46 ... There's some logic not in the proposal that's UA specific, to choose when to send those events (turn the radio on/off). That could take into account whether you've created a MMS and how much is buffered 21:45:41 ... The site knows what it has buffered, could the site inform the UA more of the buffer state to inform when to turn the radio on/off 21:45:47 ...The third thing is XXX 21:46:26 Jean-Yves: There's many ways we could skin the cat. The idea of having it all in one place in MMS makes it easier, rather than having multiple places 21:46:50 Mark: So why is it specific to streaming? Other applications would benefit 21:47:31 Jer: With implementer hat on, we found there are some cellular networks that give benefits when streaming media over 5G, e.g., counts against your data cap 21:47:46 ... So we wanted to be careful to not waste battery life and not waste people's money 21:48:24 ... Your ideas for efficient network traffic management aren't bad ones, but we don't have control over that in this WG 21:49:01 ... It's worth bringing to the people who manage the fetch spec, but we can't do that here. Similar on WebTransport and WebRTC DataChannel 21:49:29 Mark: I see the motivation to get this working for streaming. So why not put the start/stop events on Window, to generalise it a bit? 21:50:18 Jer: Sure, this is very particular to the device we were developing on. Some things we're not in control of. We may not want to expose to the entire web when the modem is available or not 21:50:38 ... So we had to ride a fine line in selecting a heuristic 21:51:24 Jean-Yves: There's a strong link between the managed media source buffer and the streaming value. So it made more sense to have it in MMS, those events are for when streaming with MMS and only used with MMS 21:51:50 Mark: Feels like a bit too much is left without guidance to the site about how it might work. Don't want to constrain how the UA would behave 21:52:27 ... Need guidance for sites. If I append a lot of data, the start/stop streaming events will slow down. I need to know that in my client code, to balance my media vs other requests 21:52:39 ... Give some heuristics or guidance on what to expect 21:52:55 Jer: Sounds like you want a non-normative note on how to implement? 21:53:26 Mark: That might be sufficient. May be we'd put all our requests int he start/stop region 21:54:24 Jer: One of the reasons for the MMS name, is a similar problem is faced in Apple WebKit, Chromium, and set top boxes. Much more limited buffers in those devices 21:54:42 ... They'll let you do complicated things, such as lots of buffers in JS, but the available memory is much less 21:55:04 ... The goal of the MMS design is to make it work on those kinds of devices too 21:55:47 Mark: Use cases where you dont' want to append right away. Example, reaching an end of episode and likely to want to watch the next episode rather than the end credits 21:56:01 ... I have a bunch of buffers, haven't appended anything 21:56:57 ... If the only effect is using more battery, that's fine, so long as its rare. Problematic would be if you're not giving the resource I need 21:57:12 Jer: You mentioned in the thread giving more info about what you're doing 21:57:56 ... A scenario where it won't work is live video, where you append every 10 seconds. In that case the page won't be obeying the start/stop events, the UA could fall back to a more efficient transport mechanism 21:58:26 ... Earlier we had an explicit signal to say the client would be doing live streaming, and use the more efficient modem up front, when loading small amounts of data more frequently 21:59:00 ... In your example, your could signal that you're done. Those could be useful additions to MMS 21:59:48 Jean-Yves: With live streaming, whether we implemented or not, the heuristic was identical. No need for it 22:00:06 ... With knowing when you won't append more data, can you signal that 22:00:29 Mark: But in this case you would still be appending 22:01:22 ... Happy to do the editing if there's PR 22:01:47 Jean-Yves: I can volunteer to edit if Matt cannot 22:03:05 Chris: We also need to think about it, important use cases such as live stream, branching narratives, etc 22:03:31 Jean-Yves: Aim here was to give UA a bit more control, it's not an ultimate solution, could have different objects, but that wasn't our aim 22:04:01 ... Our aim was to enable on low memory constrained devices, by extending MSE. I hope we've achieved this 22:04:52 Jer: I have some feedback from WAVE, so the people who work on STBs are aware and interesting 22:08:14 Chris: So with my MEIG chair hat on, let's set up a discussion so interested WAVE people can ask questions 22:08:18 Jer: Sounds good 22:09:01 ... I also want to thank Mark for the productive discussions in the GitHub thread. Good suggestions that can lead to wider further improvement 22:09:06 Topic: Next meeting 22:09:41 Chris: Scheduled for July 11, we'll see how discussion in issues goes, can meet sooner if needed 22:09:49 [adjourned] 22:10:07 Regrets: Dale_Curtis, Francois_Daoust 22:10:17 rrsagent, draft minutes 22:10:18 I have made the request to generate https://www.w3.org/2023/06/13-mediawg-minutes.html cpn 22:10:25 rrsagent, make log public 22:17:10 present+ Eugene_Zemtsov 22:17:12 rrsagent, draft minutes 22:17:13 I have made the request to generate https://www.w3.org/2023/06/13-mediawg-minutes.html cpn 22:24:43 present+ Mark_Watson 22:24:45 rrsagent, draft minutes 22:24:47 I have made the request to generate https://www.w3.org/2023/06/13-mediawg-minutes.html cpn