16:46:56 RRSAgent has joined #mediawg 16:46:56 logging to https://www.w3.org/2022/12/13-mediawg-irc 16:47:02 Zakim has joined #mediawg 16:47:13 Meeting: Media WG 16:47:28 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2022-12-13-Media_Working_Group_Teleconference-agenda.md 16:54:14 alwu has joined #mediawg 17:01:12 Matt_Wolenetz has joined #mediawg 17:01:18 present+ 17:03:28 jernoble has joined #mediawg 17:04:10 youenn has joined #mediawg 17:04:14 Hi Jer :) - I don't see anyone else on webex yet. 17:04:23 Zakim, prepare the meeting 17:04:23 RRSAgent, make logs Public 17:04:24 please title this meeting ("meeting: ..."), jernoble 17:04:43 meeting: Media WG Teleconference - 2022-12-13 17:04:51 likewise 17:04:52 * Same 17:05:18 tidoust has joined #mediawg 17:06:17 present+ 17:06:46 present+ Chris_Needham 17:06:49 scribe+ cpn 17:07:10 present+ 17:07:32 RRSAgent, draft mintes 17:07:32 I'm logging. I don't understand 'draft mintes', tidoust. Try /msg RRSAgent help 17:07:34 RRSAgent, draft minutes 17:07:34 I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust 17:08:53 hta has joined #mediawg 17:09:08 Present+ Francois_Daoust, Dale_Curtis, Alastor_Wu, Peter_Thatcher, Harald_Alverstrand, Frank_Liberato, Youenn_Fablet, Eric_Carlson 17:09:10 scribe+ 17:09:21 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2022-12-13-Media_Working_Group_Teleconference-agenda.md#agenda 17:09:44 Topic: ITU-T SG16 Liaison statement on WebCodecs 17:09:46 youenn_ has joined #mediawg 17:09:50 ericc has joined #mediawg 17:10:01 cpn: We received an incoming liaison statement from ITU-T SG16. 17:10:09 https://github.com/w3c/media-wg/blob/main/liaisons/2022-10-28-itu-t-sg16.md <- Draft reply 17:10:14 ... Around WebCodecs, and also around new VVC codec. 17:10:50 ... I drafted a reply, which describes WebCodecs, the use cases, a few indications about our own plans such as current work on VideoFrame metadata registry. 17:11:11 ... I shared this. Got a thumbs up from Bernard, Jer, Paul. 17:11:30 ... I want to make sure that everything we write here is representative. 17:11:39 ... I was hoping to get this out before the Christmas break. 17:11:58 ... If you haven't had a chance to look at it yet, now would be a good time. 17:12:12 Present+ Bernard_Aboba 17:12:52 youenn: I like the fact that you state that the group would be open to add a registration provided there was support from implementors. 17:13:00 ... I assume that means user agent implementors? 17:13:32 cpn: That's a question for the group perhaps. H.263 comes to mind for instance. 17:13:57 Dale_Curtis: I don't think that we want to be gatekeepers of what the registry contains, even though there isn't support in web browsers per se. 17:14:09 ... We'd still want some technical constraints to be met. 17:14:28 cpn: Right. That would apply to any future registration as well. 17:14:47 Topic: WebKit update on Audio focus/audio session API 17:15:10 Slideset: https://docs.google.com/presentation/d/1jICP-gAT5FydEEi_fUR0kXFHIYcbxlCo2J54rIxBCeg/edit 17:15:52 scribe+ cpn 17:16:10 Youenn: We received reports that audio handling on iOS isn't easy, e.g., VC applications 17:16:24 ... The intent of the application may not match our heuristics for setting up the audio pipeline 17:16:33 ... So a new API may be appropriate 17:16:51 ... You might remember the Audio Focus API, initially in Media Session, then split out from that 17:17:00 ... There's an explainer, linked from the slides 17:17:16 ... The overall goal is to get feedback, is the scope right, next steps? 17:17:41 ... Compared to the original Audio Focus API, we wanted to reduce scope, for the iOS platform 17:18:04 ... We focused on the audio session category, and interruptions 17:18:20 ... The API should support future features such as requesting or abandoning audio focus 17:18:27 ... Handling audio providers as a group 17:18:35 ... We wrote an explainer, and a prototype in WebKit 17:18:49 [slide 3] 17:18:50 ... Some examples: setting the audio session category, you can open the demo in iOS 17:19:08 ... playAudio and capture functions, for microphone input 17:19:36 s|Slideset: https://docs.google.com/presentation/d/1jICP-gAT5FydEEi_fUR0kXFHIYcbxlCo2J54rIxBCeg/edit|Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Dec/att-0000/AudioSession_API.pdf| 17:19:44 ... If you call playAudio initially, then capture, it's disruptive in iOS. The reason is that when you play using Web Audio, it's ambient 17:20:03 ... Two different audio levels when going from ambient to play & record. Something we want to avoid 17:20:15 i/Youenn: We received/[Slide 2] 17:20:20 The setCategory function allows you to set the category to play & record, don't use ambient 17:20:28 s/The/... The/ 17:20:42 [slide 4] 17:21:09 Youenn: On interruption, when you're in a video call, you might receive a phone call, which is higher priority, and the website is interrupted, capture stopped, audio or video elements may be stopped 17:21:17 ... But the website may not know that 17:21:34 ... It's also not clear to the website whether to restart audio after the phone call 17:21:56 ... Providing the concept of an audio session, which can go between active and interrupted, allows the website to change what is visible to the user 17:22:25 ... On an interruption, it could show a UI, or UI to allow the user to restart capture 17:22:39 [slide 5] 17:23:03 ... We tried to keep the API small. There's an audio session state and audio session type. Then we added an AudioSession interface, which we though was clearer 17:23:29 ... Use that to say it's ambient (mix with others), or play & record, so the UA can set the audio pipeline accordingly 17:23:55 ... There are event handlers, no constructor. For simple use cases, a getter on navigator to get the audio session 17:24:12 ... A default global audio session. Use this object to query or tailor it 17:24:16 [slide 6] 17:24:28 ... My main interest is not to go into specific issues. More issues are welcome 17:24:55 ... Question: is this of interest, is it going in the right direction? Any thoughts on potential next steps? 17:25:37 Dale: From a Chrome point of view, Mounir and Becca worked on it. At a glance, seems reasonable. There might be worry about duplication between Media Session and Audio Session, but no specific thoughts on that 17:26:03 Youenn: The API shape is different, there might be only one Media Session in a page, but only one Audio Session 17:26:16 ... The call to split the two things in the past is OK 17:26:43 ... We decided to delay the grabbing and releasing of audio focus. There might be other things to consider, e.g., auto play 17:27:26 q? 17:27:50 ... A question I have, is it's not yet submitted in the WG. Is it already in scope? 17:28:35 cpn: Looking at the charter, Audio Focus API is in the list of potential normative deliverables 17:28:53 ... We just need to run a call for consensus to adopt the spec to the Media WG 17:29:23 Sushanth: How to handle audio from multiple tabs? 17:29:39 Youenn: This would help with that 17:30:04 present+ Sushanth_Rajasankar 17:30:04 Sushanth: If the audio type requested by one browser is playback, and from another is ambient, only one can exist at a time 17:30:47 Youenn: You'd mimic what two native applications would do. One session with playback would probably not be interrupted by another that requests ambient 17:31:05 cpn: At what point would we be ready to run a call for consensus on this? 17:31:24 youenn: If there's already consensus in this call, we'd be interested to run it as soon as possible. 17:31:33 ... No particular hurry, but the sooner the better. 17:31:48 ... If there's no consensus, we'd like to know what to work on. 17:32:12 cpn: Just worried about support from other browser vendors. 17:32:32 youenn: We talked a bit with Mozilla. I can check with them and get back to you. 17:33:03 alwu: From Mozilla Firefox perspective, that's an API we'd be interested in supporting as well. 17:33:33 Dale: And no reason to hold off calling for consensus while we figure things out internally. 17:34:10 jernoble: In the meantime, feedback on existing issues is welcome. 17:34:34 cpn: So proposed resolution is to run a CfC. 17:34:47 Topic: Consistent SVC metadata between WebCodecs and Encoded Transform API 17:38:16 Bernard: [going through slides]. Sequence of unsigned long dependencies. There's also some missing information. 17:38:33 ... We're essentially re-inventing WebCodecs in another spec, perhaps not the right way to go. 17:38:47 ... Two different SVC metadata dictionaries could be avoided. 17:39:21 ... Temporal may be shipping in Safari, but spatial is not shipping anywhere. 17:39:32 Dale: I'm in favor of unifying what we can. 17:39:56 Bernard: Proposal is for a few of us to get together and prepare a PR to harmonize things 17:40:04 ... This would at least avoid future issues. 17:40:33 ... We made some progress in the last couple of days, and Youenn prepared a bunch of PRs that solved a number of type mismatches. 17:41:57 cpn: Is this something for the WebCodecs spec itself or the metadata registry? 17:42:10 Bernard: This is for encoded metadata for which we don't have a registry. 17:43:34 Topic: Media Pipeline architecture - Media WG input and WebRTC collaboration planning 17:43:58 cpn: Back at TPAC, we identified several places where we may benefit from coordination between groups. 17:44:08 ... This is picking up on where we're at with this. 17:44:28 Bernard: We created a Media Pipeline architecture repo following discussions. 17:44:50 ... Issues and pointers to sample code covering integration of next generation web media apis. 17:45:20 ... Also to go beyond just the specs we mentioned already, e.g. WebTransport which could be used to transport media. 17:45:49 ... From time to time, it's hard to undertand whether there are performance issues in the specs, implementations or in the code sample. 17:47:04 Bernard: When I started of, I was thinking about capture with Media Capture and Streams Extensions, then encode/decode with WebCodecs (and also MSE v2 to some extent), Transport (WebTransport, WebRTC data channels in workers), and Frameworks (WHATWG streams, WASM) 17:47:35 Bernard: The pipeline model is based on WHATWG Streams, through TransformStreams piped together. 17:47:54 ... When you're sending frames, you have a several options, e.g. reliable/unreliable, etc. 17:48:14 ... To stream these pipelines together, you have to use all of these APIs together. Does it all make sense? 17:48:31 ... I don't know that many developers who understand all of these APIs. 17:48:48 ... Some issues already created in the repo. 17:49:04 -> https://github.com/w3c/media-pipeline-arch/ Media Pipeline architecture repo 17:49:12 ... A lot of the issues are focused on transport. 17:49:35 ... There are a few things that are worth discussing here. 17:50:16 ... E.g. rendering and timing. Media Capture Transform is an interesting API. Does VideoTrackGenerator have a jitter buffer? Does it not? 17:50:25 ... That is not particularly well defined in the spec. 17:50:48 ... We have two samples at the momen. One is a WebCodecs encode/decode in worker in the WebCodecs repo. 17:51:20 ... The second one adds WebTransport to that. This one took more work to optimize the transport. It adds serialization/deserialization. 17:51:40 ... We use frame/stream transport. That's not exactly RTP but it's close. 17:51:52 ... We're using SVC at baseline and partial reliability. 17:52:04 ... Overall, it's working surprisingly well. 17:52:26 ... I had to do a reorder buffer but still not a full jitter buffer. 17:52:43 Bernard: Here are some of the things that you can play with. 17:53:24 ... You can play with this stuff. At the end, it generates a Frame RTT graph. That does not really give you glass to glass measurements. 17:53:39 ... Performances are pretty reasonable now after some work. 17:53:56 ... Slide shows an example with AV1 at full-HD. 17:54:46 ... What's interesting is that key frames can be transmitted within a single congestion window. 17:55:07 ... General question is what do we do with this? 17:56:06 cpn: That's really great to get that practical feedback from building things. 17:57:02 Bernard: Yes, we're seeing a lot of stuff. Similarly, there are a few things where I don't know enough of the internals to understand what needs to be done. 17:57:39 ... You have to be cautious of await calls with WHATWG Streams, since they are going to block. Debugging is also hard. 17:58:13 youenn: Note you may use JS implementations or ReadableStream and WritableStream to ease debugging. 17:58:54 Bernard: Good idea. You can get a dozen stages and you don't really know where things are in the different queues. It's not easy to figure out what happens. The code is fairly small though. 17:59:24 cpn: Immediate next step? 18:00:02 Bernard: Adding APIs in multiple groups adds question. It's worthwhile checking in on this periodically. 18:00:12 ... I don't want to act like I have a handle on this. 18:01:04 cpn: OK, we'll talk more about how to improve that cross-group collaboration. 18:01:45 cpn: Our next meeting will be on the new year. Happy Christmas and looking forward to seeing you next year! 18:02:45 RRSAgent, draft minutes 18:02:46 I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust 18:31:09 i|Bernard: [going through slides]|Slideset: https://lists.w3.org/Archives/Public/www-archive/2022Dec/att-0002/MEDIAWG-12-13-2022.pdf 18:32:11 i|Bernard: [going through slides]|[Slide 2] 18:32:16 RRSAgent, draft minutes 18:32:16 I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust 18:32:49 i|Bernard: [going through slides]|[Slide 3] 18:32:52 i|Bernard: [going through slides]|[Slide 4] 18:33:01 i|Bernard: [going through slides]|[Slide 5] 18:33:46 i|Bernard: We created a Media Pipeline architecture repo|[slide 6] 18:34:09 i|Bernard: When I started of|[Slide 7] 18:34:29 i|Bernard: The pipeline model is based on WHATWG Streams|[Slide 8] 18:35:07 i|... Some issues already created in the repo.|[Slide 9] 18:35:42 i|... We have two samples at the momen|[Slide 10] 18:36:01 i|Bernard: Here are some of the things|[Slide 11] 18:36:29 i|... Slide shows an example with AV1 at full-HD.|[Slide 12] 18:36:34 RRSAgent, draft minutes 18:36:34 I have made the request to generate https://www.w3.org/2022/12/13-mediawg-minutes.html tidoust 18:39:27 RRSAgent, bye 18:39:27 I see no action items