06:58:02 RRSAgent has joined #webcodecs-serialization 06:58:06 logging to https://www.w3.org/2023/09/13-webcodecs-serialization-irc 06:58:06 RRSAgent, do not leave 06:58:06 RRSAgent, make logs public 06:58:07 Meeting: WebCodecs Serialization Format 06:58:07 Chair: Peter Thatcher 06:58:08 Agenda: https://github.com/w3c/tpac2023-breakouts/issues/78 06:58:10 clear agenda 06:58:12 agenda+ Pick a scribe 06:58:14 agenda+ Reminders: code of conduct, health policies, recorded session policy 06:58:16 agenda+ Goal of this session 06:58:20 agenda+ Discussion 06:58:22 agenda+ Next steps / where discussion continues 06:58:31 Ian has left #webcodecs-serialization 07:15:38 Ian has joined #webcodecs-serialization 07:15:42 Ian has left #webcodecs-serialization 13:16:47 RRSAgent has joined #webcodecs-serialization 13:16:47 logging to https://www.w3.org/2023/09/13-webcodecs-serialization-irc 14:00:40 cpn has joined #webcodecs-serialization 14:01:17 Meeting: WebCodecs Serialization Format 14:01:23 scribe+ cpn 14:03:10 Present+ Peter_Thatcher, Eugene_Zemtsov, Harald_Alvestrand, Xiaohan_Wang, Eric_Carlson, Youenn_Fablet, Paul_Adenot 14:03:14 rrsagent, draft minutes 14:03:15 I have made the request to generate https://www.w3.org/2023/09/13-webcodecs-serialization-minutes.html cpn 14:03:19 rrsagent, make log public 14:03:58 present+ Rijubrata_Bhaumik 14:04:23 Peter: In the IETF MoQ group, there's interest in sending media over QUIC 14:04:38 ... How do you serialize the media into bytes so you can send over QUIC stream or datagrams 14:04:56 ... Involves attaching metadata. Looked at different codecs, which have different modes 14:04:58 Xiaohan has joined #webcodecs-serialization 14:05:04 ... WebCodecs registry is a place we can point to 14:05:22 ... People will want to make MoQ endpoints using web clients, using WebCodecs and WebTransport 14:05:37 Riju has joined #webcodecs-serialization 14:05:43 ... Want to take the output from WebCodecs, serialize, then send to a decoder on the other side, which may be WebCodecs 14:06:10 ... I did something similar in Second Screen WG, on Open Screen Protocol 14:06:23 ... At that time we chose CBOR, a kind of binary JSON 14:06:59 ... What is the interest from others? What would you want to do with the serialization, store it, send over QUIC? 14:07:30 Harald: IIUC this is about making sure we have both encoded data and metadata for a frame on the other side of the connection 14:08:06 ... So it's like the RTP encapsulation format 14:08:13 ... We don't have a 1:1 mapping of it 14:08:13 ... Which parts of this should be different from RTP formats, and why? 14:08:23 Bernard: This would include things normally in RTP extensions as well 14:08:59 ... Applies to the RTP packet, timestamp, things related to SVC in IETF are always packeted 14:09:16 ... Would become patent encumbered in IETF 14:09:36 Peter: Different questions: where should this happen? How different should this be to work already done? 14:09:45 ... IETF looking at how to do RTP over QUIC 14:10:15 ??: I thought the output of the encoder was already serialized and you could plug it into the decoder. What additional metadata is needed? 14:11:46 Peter: For example, the output of the encoder includes EncodedVideoChunk, which includes the bytes, timestamp, keyframe flag, metadata such as how SVC is structured 14:11:46 ... Information about size, color space, and which codec it is 14:12:09 ... In header extensions, things like frame dependencies. In RTP payload header, there's metadata 14:12:24 ... There's metadata tied to the unencoded media, e.g., audio level 14:12:41 ... So if you look at RTP metadata, it's the same kind of metadata 14:12:59 ??: Is the metadata already exposed to JS, just needs packaging, or are there internal pieces needed? 14:13:19 Peter: Want it done in a standardised way, but they don't want to use RTP 14:14:05 ... There could be internal metadata, but a good start is to take everything in the API and serialize that 14:14:24 Youenn: Are you thinking of extensions? 14:14:59 ??: It's whether the problem can be solved in JavaScript, it sounds like I could convert to JSON and parse on the other side. Would that work? 14:15:24 Peter: You could, yes. Make your own non-standard format. The question is what interest is there in having a standard format? 14:15:31 ... There's interest in the MoQ WG for that 14:15:46 ... Is there interest here? 14:16:05 Bernard: The standards interest comes from the caching. MoQ uses a cache, relays need metadata to do forwarding 14:16:14 padenot_ has joined #webcodecs-serialization 14:16:20 s/??/Henrik/ 14:16:34 Peter: Relays are designed to be agnostic to the media format, and some metadata is opaque to the relay 14:17:02 Bernard: Not sure that's true, there's preferential forwarding, may be ready by caches 14:17:19 Peter: That would have to be defined by MoQ WG, as it's MoQ specific 14:17:37 ... The serialization can be done in a separate doc, as it's opaque to relays, that's my understanding 14:18:15 ... Are people interested in being able to serialize outside of MoQ, for other use cases 14:18:27 Bernard: There have been issues raised about support for DRM 14:18:49 Xiaohan: How is that related to this question for a need to serialize WebCodecs? 14:19:25 Harald: We have a proposal to define a generic packetizing format for RTP so you can send any content you want over the wire 14:20:27 cpn_ has joined #webcodecs-serialization 14:20:32 scribenick: cpn_ 14:20:50 Peter: Content of S frames are opaque 14:21:12 Harald: This came up in a side discussion about SDP munging for WebRTC encoded transform 14:21:50 Peter: One way to think about it is, if we were to start from scratch and solve the same thing, serialize the payload, and accumulate all the header extensions, what would we do? 14:22:33 ... Similar question in the Second Screen group, we used CBOR and came up with types. It was straightforward, easier to use a CBOR parser 14:23:36 ... Let's come to Bernard's question about where this should be worked on 14:23:36 Riju has joined #webcodecs-serialization 14:23:38 ... There's IPR issues for some of the metadata 14:24:01 Bernard: The concern would be if you take the WebCodecs spec you might start encumbering WebCodecs, that would be awful 14:24:24 Peter: What about trying to define the serialization format in W3C, would there be enough people interested in working on it to do it here? 14:25:09 Bernard: Media WG already has an ISO BMFF format for MSE 14:25:14 Harald: A note of caution on IPR. IETF runs on the principle that every participant should disclose their IPR, but doesn't force anyone to give their rights to anything 14:25:24 ... At W3C all participants make a commitment when joining the WG 14:25:37 ... If you don't join the WG, no promises made 14:26:49 ... So unless all contributors are in the W3C WG and agree to remain members when the work item is adopted, that's what's required for W3C 14:27:21 ... Getting all people who're asserting IPR to join might be the issue 14:27:26 Youenn: I'd tend to pick the venue where more people are interested 14:28:04 Peter: Gauging level of interest is the purpose of this 14:28:25 Xiaohan: What's the use case you're imagining? 14:29:01 Peter: It could be used for any purpose. For the MoQ WG, it's for real time and streaming use cases, e.g., live broadcast video conference 14:29:49 Xiaohan: There are already solutions not using WebCodecs out there. If it's for streaming then CMAF can be the format, but too heavy for real time 14:29:58 ... Why not adopt an existing format? 14:30:21 Peter: MoQ WG separated the two, so the format can be CMAF based 14:31:03 ... Others who want to do things web oriented, you don't want to containerise to CMAF, more straightforward to do what JSON does, but more efficient, hence CBOR 14:31:40 ... For realtime or very low latency streaming it would be a better DX to do something new, which is why there's interest in MoQ WG for a low overhead container 14:32:02 Xiaohan: Format of MediaStream in WebRTC? 14:32:28 Peter: That uses RTP, as Harald described. It could be done over QUIC, there's work in IETF, but like CMAF it carries baggage 14:33:52 Chris: Wouldn't MoQ WG need to define a format anyway, absent any work here? 14:34:13 Peter: Yes, they're working on the CMAF based format, but the question is about the low latency format 14:34:55 Bernard: The Media WG meeting heard about the IETF 117 WebCodecs presentation 14:35:47 ... https://datatracker.ietf.org/meeting/117/materials/slides-117-moq-webcodes-container-00 14:36:42 Peter: WebCodecs already has the metadata defined that we'd want to serialize 14:36:57 ... What we define might be usable in different situations 14:37:52 ... Continue in IETF or bring to W3C 14:38:44 Chris: Interest expressed at IETF to work on it? 14:38:54 Peter: Yes, three of us at least are interested, and others 14:42:20 Chris: We are developing a browser based video editor, have our own format, so we have the general need 14:42:29 Peter: Need for interop around that? 14:42:38 Chris: I'd have to follow up to get precise requirements 14:43:02 Bernard: A frequent request we get on WebCodecs is a standard API for container formats 14:43:50 Peter: I have a similar situation, where I have my own non-standard serialization, there might be potential for a standard solution 14:44:23 Peter: So if not this, what are people interested in? 14:44:46 Henrik: Using WebCodecs with the RTP transport in WebRTC, don't know if packetization needs to be standardised or not 14:45:58 Peter: So if you were to use WebCodecs with RTP transport, how would you use it? 14:46:35 Henrik: I see value in coupling app-specific metadata to video frames. Haven't done enough homework to know more 14:47:13 Youenn: MoQ is trying to do real time for QUIC, server to client. Why not using the MoQ approach, over WebTransport 14:47:53 Harald: Ability to include non-standard or proprietary metadata. That might be an interesting feature to explore further 14:48:09 ... How would you do that, for the interoperable web? 14:48:43 Peter: Youenn said it would be to use the same format over MoQ or other channel 14:48:56 ... Then some interest in having P2P QUIC come back 14:49:37 ... If you model it that way, if you add app-specific metadata, you would rather put it in the payload than metadata extensions, treating RTP transport as a generic transport than RTP specific 14:50:32 ... To Harald's question about experimental metadata, it would be easy to put what you want in the format. CBOR is very flexible, key/value, you can have custom keys for custom values 14:50:40 s/metadata extensions/header extensions/ 14:50:53 ... So we could do similar to header extensions with this 14:51:54 Bernard: Things useful for evaluating experiments, e.g., capture time, you wouldn't send on the wire but useful for experiments 14:51:56 Harald: We also use capture time for reasons 14:52:03 Peter: Since P2P and QUIC were brought up, is anyone interested in a P2P QUIC API? 14:52:38 Randall: P2P QUIC would be more useful for WebRTC like things 14:53:31 ... Why was it dropped? 14:53:31 Peter: To focus on client/server initially, then come to it later 14:54:23 ... Anything else to discuss? 14:55:06 Harald: I was hoping to learn new use cases. I'll monitor what MoQ is doing 14:55:48 Henrik: I was hoping to see if the serialization use case can be done, and it sounds like it's possible to do today 14:55:52 Peter: Yes, it is possible 14:56:11 ... And some are already doing it, so question is about need for interop 14:56:42 Henrik: If the format and API is flexible enough it may be preferable to doing it yourself 14:57:01 ... But risk if there's anything you can't do in the proposed format, people would do their own thing anyway 14:57:51 Xiaohan: There's lots of container formats, so depends on use cases. There's a profile Peter mentioned. If requires super low latency, what should be done? 14:58:20 Peter: If it's standardised and pick up and use, it might be a default people would reach for rather than start from scratch 14:59:01 Xiaohan: If we get to the point where the browser can do the muxing. Is the CMAF concern to do with the size of the generated stream, or compexity? 14:59:17 Peter: Size, if you're sending each as a separate message 14:59:40 ... At last for depacketizing, you hand a CMAF chunk to MSE 15:02:01 Chris: Interest in WebCodecs for MSE. We're in effect polyfilling the video element with buffering and a canvas 15:03:44 Peter: You can hand over decoded VideoFrames to MSE 15:03:55 ... Integration with MSE a good thing to think about 15:04:18 Youenn: Would be interesting to know what's missing in MSE for you to migrate to that, what features we could add 15:04:42 Peter: Thanks for coming. If you have renewed interest or use case, let us know 15:04:45 [adjourned] 15:04:47 rrsagent, draft minutes 15:04:49 I have made the request to generate https://www.w3.org/2023/09/13-webcodecs-serialization-minutes.html cpn_ 15:16:21 xueyuan has joined #webcodecs-serialization 15:16:59 xueyuan has left #webcodecs-serialization 15:19:43 Xiaohan has left #webcodecs-serialization