20:54:49 RRSAgent has joined #mediawg 20:54:53 logging to https://www.w3.org/2023/04/11-mediawg-irc 20:54:53 Zakim has joined #mediawg 20:54:56 RRSAgent, make logs public 20:55:08 Meeting: Media WG call 20:55:28 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2023-04-11-Media_Working_Group_Teleconference-agenda.md 20:58:23 cpn has joined #mediawg 21:01:20 present+ Chris_Needham, Francois_Daoust, Bernard_Aboba 21:03:33 jernoble has joined #mediawg 21:04:00 present+ Jer_Noble 21:04:13 present+ Dale_Curtis 21:04:39 present+ Eugene_Zemtsov 21:06:40 present+ Peter_Thatcher 21:06:47 repo: w3c/webcodecs 21:06:47 OK. 21:07:23 Topic: Allow decoder to ignore corrupted frames 21:07:23 scribe+ cpn 21:08:28 Slideset: @@ 21:08:36 [Slide 2] 21:08:47 [Slide 3] 21:08:48 Bernard: s/@@/https://docs.google.com/presentation/d/1KbOE1VtD1fzvjmsYl-SA18l2kpVOL-uFB_h4h7s2Pdw/edit/ 21:09:13 Bernard: WebCodecs encoder and decoder errors are fatal. It queue's a task to close the encoder/decoder 21:09:25 ... The issue is: Is there some way to not close? 21:09:28 #656 21:09:30 https://github.com/w3c/webcodecs/issues/656 -> Issue 656 Allow decoder to ignore corrupted frames (matanui159) agenda 21:09:41 [Slide 4] 21:09:57 ... Dale clarified that perhaps the text could clarify fatal vs non-fatal 21:10:14 ... The safest thing to do could be to close it 21:10:28 Dale: All errors are fatal, not sure why I wrote just software there 21:10:50 s|Bernard: s/@@/https://docs.google.com/presentation/d/1KbOE1VtD1fzvjmsYl-SA18l2kpVOL-uFB_h4h7s2Pdw/edit/|| 21:11:03 Bernard: It doesn't interfere with resiliance, FEC, redundant error coding, etc 21:11:03 s|@@|https://docs.google.com/presentation/d/1KbOE1VtD1fzvjmsYl-SA18l2kpVOL-uFB_h4h7s2Pdw/edit/ 21:11:18 ... Discussed if we could have more tests for sending errors 21:11:31 ... Chromium reports the wrong error type 21:11:40 Dale: We have a bug open to fix that 21:11:52 Bernard: Is it a bug? Is more info needed in the error? 21:12:09 ... Question: should all errors be fatal? Dale makes a good point why they should be 21:12:44 ... Potential issues with security team review if we don't make it fatal 21:13:00 Dale: If the author wants to handle the error and resume the decoding, could be for them to decide 21:13:07 Bernard: But they can't if it's closed 21:13:16 Dale: They can create a new decoder 21:13:19 s|https://docs.google.com/presentation/d/1KbOE1VtD1fzvjmsYl-SA18l2kpVOL-uFB_h4h7s2Pdw/edit/|https://lists.w3.org/Archives/Public/www-archive/2023Apr/att-0000/MEDIAWG-04-11-23.pdf 21:13:23 RRSAgent, draft minutes 21:13:25 I have made the request to generate https://www.w3.org/2023/04/11-mediawg-minutes.html tidoust 21:13:32 Bernard: Yes. That would require a keyframe 21:13:56 ... Second question: There are various reasons why you could get an error. Hardware decoders may error where a sofware decoder wouldn't 21:14:22 ... Hardware resources could be acquired by another app. Reconfigure with prefer-hardware could then fail, then you'd have to fall back to software 21:14:40 ... Paul asked what's the difference between reset and close, and impact on performance? 21:15:22 Bernard: Any opinions? I've had developers ask whether there's truly been a decoder error, or something else, such as a GPU crash 21:15:32 ... Does only having EncodingError provide enough info? 21:16:01 Dale: We're limited on the information available to us. Where a software decoder is more permissive it's in a way non-compliant to the spec 21:16:33 ... I though we decided among editors that it should be fatal 21:16:43 Bernard: Is there any objection in the WG to that? 21:16:47 (none) 21:17:08 Bernard: So what to do next? Reconfiguring with prefer-hardware could fail. 21:17:52 Dale: Developers would have to handle it either way. We could provide more docs to advise on use a more professional analysis tool 21:18:06 ... or ffmpeg 21:18:30 Bernard: Is EncodingError right? 21:18:56 ... Any other changes to the spec? 21:19:18 Dale: No spec change, just MDN documentation improvements. My team have been working on that 21:19:55 Eugene: Could add a new optional exception such as corrupted stream or corrupted chunk, to say it's something to do with the stream 21:20:07 ... and not some kind of infrastructure issue underneath 21:20:14 Bernard: That would be helpful though 21:20:48 ... Developers would appreciate that. Would it be an error message inside EncoderError? 21:21:07 Eugene: That error type doesn't sound right 21:22:10 Dale: I'm not sure why we didn't add that. Technically it's an error in the encoding... 21:22:25 Bernard: Not sure it's a requirement to change the type, but the extra info would be useful 21:22:47 i/Bernard: So what to do next?/[Slide 5] 21:22:58 Eugene: If we can see the data is noncompliant, and maybe an OperationError when it's a GPU crash or out of resources, would be useful distinction 21:23:18 ... Hardware decoders don't always give the reason for error though 21:23:35 Bernard: Next step would be to see if that's feasible and prepare a PR if so 21:23:40 Eugene: I can do that 21:24:02 Topic: Allow configuration of AV1 screen content coding tools 21:24:10 [Slide 6] 21:24:15 #646 21:24:16 https://github.com/w3c/webcodecs/issues/646 -> Issue 646 Support for Screen Content Coding (aboba) PR exists 21:24:16 Bernard: We've added a PR to initialise the AV1 quantizer 21:24:26 PR #662 21:24:27 https://github.com/w3c/webcodecs/issues/662 -> Pull Request 662 Enable configuration of AV1 screen content coding tools (aboba) 21:24:29 ... This PR adds a boolean for forceScreenContentTools. Default is false 21:24:56 ... when true, the AV1 spec sets a seek force screen content tools, then it uses the palette and block copy tools 21:25:09 [Slide 7] 21:25:15 ... The PR adds this boolean attribute, default false, with an explanation 21:25:30 Eugene: Difference with the other PR? 21:25:50 Bernard: I rebased it due to a merge conflict 21:26:30 Eugene: Another one proposed adding the flag to VideoEncodingConfig. The distinction is you configure it once, but now you'd set it per-frame 21:26:39 Bernard: Quantizer is per-frame as well 21:26:57 Eugene: So need to decide where it goes: per frame or not 21:27:25 Bernard: Should be per frame. Content could change during screen capture, e.g., go from slides to a video presentation where you'd disable screen content tools 21:27:48 ... I closed the other PR 21:28:05 Eugene: It belongs per-frame. I checked libaom, it does allow setting per-frame 21:28:25 Bernard: If can change, but how you know to change it is a different story... 21:29:05 Dale: Quantizer makes sense as an encoder, but the screen tools feels like a per-frame metadata thing 21:29:15 ... Then under the hood we'd automatically do the right thing 21:29:48 Bernard: WebRTC does it that way, by checking whether it comes from a screen - but could be a game or sports event which are not amenable to screen content tools 21:30:09 Eugene: IIRC, we'll have the same for VP9 21:30:35 Bernard: I guess so, the AV1 tools are more sophisticated. Would that use the same kind of parameter? 21:30:56 Eugene: As far as I know there's a global setting, not per frame 21:31:24 Bernard: HEVC has screen content tools, but it's hardware only, so doesn't make sense to add it, as it woulnd't be used 21:31:43 ... Looking at the WebRTC code, it mostly changed the quantizer 21:33:05 Chris: So proposed resolution is to add this per-frame for AV1, then consider VP9 separately. It's very much codec-specific 21:33:11 Topic: Extend EncodedVideoChunk metadata for SVC 21:33:24 [Slide 8] 21:33:37 #619 21:33:38 https://github.com/w3c/webcodecs/issues/619 -> Issue 619 Consistent SVC metadata between WebCodecs and Encoded Transform API (aboba) agenda, PR exists 21:33:40 PR #654 21:33:41 https://github.com/w3c/webcodecs/issues/654 -> Pull Request 654 Extend EncodedVideoChunkMetadata for Spatial SVC (aboba) 21:34:08 Bernard: For background: In WebCodecs we support temporal scalablity, and WebRTC supports temporal and spatial scalability 21:34:31 ... In the WebRTC encoded transform, we provide metadata for these encoded frames 21:35:00 ... WebCodecs has an encodeded metadata, but there's a mismatch - if you want temporal scalability you need to use the WebRTC API 21:35:16 ... Since it's in WebRTC, why not bring it also to WebCodecs? 21:35:19 [Slide 9] 21:35:51 ... First question is compare WebCodecs and WebRTC APIs. WebCodecs API is more structured, not just one big blob of stuff as in WebRTC 21:36:36 [Slide 10] 21:36:59 ... The SVC sub-dictionary design has a frame number (unsigned short), not the same as frame id 21:37:08 ... frameId is a globally unique id for the frame 21:37:51 ... It's something you want to serialize on the wire, so the sender and receiver could want the information. Frame number is a modulo 2^16 of the frame id, to use less space in the wire serialization 21:38:09 ... When you describe dependencies you are referencing a series of frame numbers 21:38:37 ... In real life you typically don't have 2^16 or 2^32 frame-ago dependencies 21:38:55 ... Different names compares to the WebRTC version 21:39:41 ... Decode targets and chain links. A forwarder keeps state, and the frame rate and spatial resolution for a particular client. This determines what layers I forward to that client 21:40:14 ... It's useful to have this from the encoder, as the forwarder has state about the target, and compare against the frame itself 21:40:39 ... It makes it possible forwarder to quickly decide whether to forward or not 21:41:22 ... Protect the WebCodecs decoder against things that would cause a decoder error 21:41:49 ... Chain links. If you get a frame as a receiver, with dependencies, is it true that if I submit it ot the WebCodecs decoder I should get an error? 21:42:00 ... The dependencies might have dependencies? So you'd still get a decoder error 21:42:17 ... Chain links look at the whole chain of dependencies, and see if you'd get an error 21:42:47 ... Easier for the encoder to send the data than for the receiver to calculate the dependency graph, and avoid duplicate work across each client 21:43:04 ... We're thinking this should go in the SVC dictionary 21:43:36 ... One thing is there's no unsigned short, just unsigned long. Is this headed in the right direction? 21:44:05 Eugene: As a superficial comment, we should everything as unsigned long. It wouldn't cost anything 21:44:08 Bernard: Agree 21:44:31 Eugene: I'm not sure when this would be implemented 21:45:06 Bernard: It's in the Chromium code base. The SVC modes and depedency descriptor is there 21:45:44 Dale: I think we'd need to check what our encoders produce. We should get one software encoder working before landing the spec change 21:46:19 [Slide 11] 21:46:22 [Slide 12] 21:46:24 [Slide 13] 21:46:31 Bernard: I think it's possible to enable it. Next steps? 21:46:44 Dale: Can you share a link to the Chromium code? 21:46:50 Bernard: Will do 21:47:37 Chris: Prototyping as Dale suggested? 21:47:59 Bernard: It's in WebRTC spec, concern about the alternative version showing up in WebRTC 21:48:27 Dale: I feel like having it working end to end, either WebRTC or WebCodecs, would be good 21:48:51 Bernard: If the WG approves the approach, could follow it up in WebRTC 21:49:03 Dale: Would prefer to see it working first though 21:49:51 Eugene: We only have the temporal layer ID, as that's the only thing implemented, and we can add more dictionary entries when we're happy 21:50:43 Bernard: We could submit a PR to remove from WebRTC, then when it's implemented add it in both places 21:51:51 ... It's dangeous for both to be out of sync 21:52:14 Chris: Has this shipped in WebRTC or is it only a spec change for now? 21:52:22 Bernard: Would have to check 21:52:35 Chris: Happy to organise joint meeting discussion between both groups 21:53:28 Chris: So check if shipped, get working end to end in either WebRTC and WebCodecs, then ensure specs are consistent across both 21:53:54 Topic: Media WG rechartering 21:54:04 scribe+ tidoust 21:54:16 repo: w3c/charter-media-wg 21:54:16 OK. 21:54:23 #38 21:54:24 https://github.com/w3c/charter-media-wg/issues/38 -> Issue 38 Document Picture-in-Picture (steimelchrome) 21:54:39 https://github.com/w3c/media-wg/pull/38 : Add webcodecs quantizer mode to agenda listing. 21:54:39 cpn: Open question related to rechartering and the Document Picture-in-Picture. 21:55:11 ... The Media WG was suggested in the TAG review as venue for Recommendation track progress. 21:55:58 ... Suggestion at this stage would not be that it become a WG deliverable but rather a potential normative deliverable as we've done in the past with the Audio Focus API. 21:56:20 ... With the goal being to avoid rechartering when spec is ready to enter the Recommendation track. 21:56:36 ... Question here is whether we should consider the document to our scope as a potential normative spec. 21:57:08 ... François pointed out that this would change the scope of the WG a little, because the group is focused on media related features. 21:57:22 ... Document PiP is broader in scope. 21:57:43 ... While it has interest from media companies to using it for media content, it's not restricted to that at all. 21:58:04 ... Question is: is the Media WG the right venue to continue work on the spec? 21:58:20 ... In favor: Picture-in-Picture is developed by the Media WG 21:58:43 ... I have a concern about the broader scope though. 21:59:29 jernoble: The original media element supported fullscreen mode. The Fullscreen API is a more general purpose API. I think that is a WHATWG deliverable and that there are lots of parallel there. 21:59:49 ... As an implementer, I don't know that working on the spec in the Media WG makes sense. 22:00:14 cpn: I think François suggested an alternate ohme for the spec, WebApps WG. 22:00:27 ... I don't think there's a barrier to landing that on the Recommendation track somewhere. 22:00:46 ... I'll just start an email thread with Tommy and chairs to thrash this out. 22:01:22 ... We don't want to hold up too much on that because draft charter is mostly ready otherwise. 22:01:57 Dale: Most use cases are media-related but argument and comparison with Fullscreen makes sense to me. 22:02:47 jernoble: If it becomes necessary to integrate with other specs, such as CSS, etc. it also makes sense to use a group that's more used to doing that. 22:03:01 cpn: I agree. My proposal would be not to do it here. 22:03:21 jernoble: Even more importantly, I think it would be even more successful in another group. 22:03:33 Topic: TPAC 2023 joint meetings 22:04:13 cpn: I think what I'm going to propose for our group is similar to last year: full morning or full afternoon to go through discussions. The other thing I'm interested in exploring is joint meetings. 22:05:13 Bernard: Ongoing poll to figure out who's going to show up in-person. If not enough people, my sense is that W3C guidelines are not to request TPAC time. 22:05:22 ... I'll be remote. 22:05:30 cpn: I'm planning to be there in-person. 22:06:07 tidoust: I wouldn't worry too much about the requirement for number of people. The venue is large enough 22:07:07 Bernard: Joint meeting with WebRTC will be useful. Lots of ongoing discussions. 22:07:21 cpn: I'll follow-up via email on the specifics of that. 22:07:35 RRSAgent, draft minutes 22:07:36 I have made the request to generate https://www.w3.org/2023/04/11-mediawg-minutes.html tidoust 23:25:33 Zakim has left #mediawg