14:58:48 RRSAgent has joined #mediawg 14:58:53 logging to https://www.w3.org/2023/05/30-mediawg-irc 14:58:53 Zakim has joined #mediawg 14:58:57 Meeting: Media WG meeting 14:59:08 Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2023-05-30-Media_Working_Group_Teleconference-agenda.md 15:01:38 present: Chris_Needham, Bernard_Aboba, Dan_Sanders, Youenn_Fablet 15:01:52 RRSAgent, make logs public 15:02:08 present+ Francois 15:02:17 Chair: Chris 15:02:25 eugene has joined #mediawg 15:02:41 present+ padenot 15:02:58 present+ eugene 15:04:28 sandersd has joined #mediawg 15:05:30 in the agenda, the Detach codec inputs link is wrong, the issue is: https://github.com/w3c/webcodecs/issues/104 15:07:32 Topic: Detach Codec Inputs 15:07:51 scribe+ cpn 15:08:08 Eugene: https://github.com/w3c/webcodecs/issues/104, there's a PR that adds a transfer list to avoid a copy 15:08:26 ... The spec change is minimal but allows saving a copy when the frame is constructed 15:08:45 ... PR is https://github.com/w3c/webcodecs/pull/676 15:09:11 ... Can also consider it for audio and ImageDecoder, but this was a starting point 15:09:25 ... If we agree on the approach for video, it'll be trivial to do those also 15:09:58 ... We add one member to VideoFrameBufferInit. It's a list, potentially it might be a single item, as we never need to transfer more than one arraybuffer 15:10:10 ... But for consistence with scoped clone and postMessage in workers, I made it a list 15:10:42 ... In future could support separate array buffers for planes, so to future-proof it a bit, transfer list is an array 15:11:00 ... Paul looked at the spec wording, I tried to incorporate his feedback 15:11:24 ... We say what goes in the transfer list, we test if can be detached, if not we throw. Then detach everything in the transfer list 15:11:55 Youenn: What if there's something in the transfer list unrelated to the VideoFrame 15:12:45 Eugene: Everything gets detached, but we don't use the pixels. We give structured clone an object, and all buffers in the list get transfered. If they're in the object that's cloned they get transfered, otherwise detached 15:12:47 RRSAgent, draft minutes 15:12:49 I have made the request to generate https://www.w3.org/2023/05/30-mediawg-minutes.html tidoust 15:13:00 ... It's not the most obvious behaviour, but consistent with other places on the web 15:13:26 Youenn: Developers could make a mistake using it, but if consistent with postMessage and other places, it's OK I guess 15:13:57 Eugene: Any objections from anyone? If we're all happy, we intend to figure out wording, and merge. I have a prototype in Chromium, will send intent to ship? 15:13:59 present+ Jer_Noble, Peter_Thatcher, Eric_Carlson 15:14:24 Dan: Are GPU buffers that are mapped transferreable? 15:14:27 Eugene: They're not 15:15:17 Paul: My comments are resolved. Two comments still open, for the first the sentence isn't useful, as we're transferring the data and not copying - line 3449 in the latest version 15:15:35 Eugene: "Use visible rectangle and layout" 15:15:47 Paul: You can do that, but not useful if you're copying 15:16:15 Eugene: I think we still need it, you might have an arraybuffer with gaps, and layout would tell you where to look for the planes. I took the wording from the copy case 15:16:30 ... If we need to know it for when to copy from, we also need it in this case 15:17:09 Paul: The idea of the prose is to make a smaller copy. You have all the info about the layout so doesn't need normative text, it can be omitted so we don't say the same thing multiple times 15:17:13 Eugene: I'll drop it 15:17:31 Paul: The other comment is just a remark, nothing to do 15:17:55 Dan: If we remove that prose, it might remove ability to purposely change those things, e.g., change the visible rect to 0,0 15:18:16 ... Might be useful to preserve for transfer. We'd transfer the same backing 15:18:35 ... It's not required to produce the same visible rect as output 15:18:46 I want browsers to have the option to do it 15:19:08 Paul: If we think there's a compatibility issue, it needs to be in the spec 15:19:34 Dan: I also want to refrain from forcing browsers to not copy. E.g., the odd nv12 case, Chrome expands the size outwards 15:19:58 Paul: So if you specify a buffer in the transfer list, but the implemenation says it's cannot be done, would you prefer it to throw or silently copy? 15:20:12 Eugene: Think we need to silently copy, a browser might not implement this part of the spec 15:20:47 ... We might have a buffer that doesn't work like this, e.g., a GPU backed buffer in WebGPU, or other corner cases where we can't guarantee transfer 15:21:27 Paul: Throwing is fine. Detach is an operation that can cause issues. In general I prefer to be explicit, but postMessage as precedent - or generally, transfer lists 15:21:58 Eugene: Another thing we can't transfer is SharedArrayBuffer. That might be preserved by a worker, and we don't want pixels to be changed from under the VideoFrame 15:22:05 Paul: Can make it throw in that case 15:22:35 Eugene: Dan, if you have more comments, please add to the PR 15:22:48 ... I'll add detail on the errors throw, in what cases 15:23:01 Paul: I approved the PR, add a non-normative note 15:23:31 Youenn: Would be good to check edge cases around structured clone, e.g., if you have the same arraybuffer twice in the transferlist 15:23:48 ... Ideally we'd just call structured clone, but good to check all possible cases 15:24:27 Eugene: I'll do this. We can't use structured clone as it makes an arraybuffer out of another arraybuffer, and we need to move it to a VideoFrame. But we're aiming to be like structured clone 15:24:35 Paul: Why can't we use structured clone? 15:25:17 Eugene: It takes an object and clones or detaches. For us it's tricky as there's no visible arraybuffer after moving to the VideoFrame, it'll be an implementation detail 15:25:58 Paul: It's an abstract operation, take what you need from it and give it to the VideoFrame object. Call the structured serialize transfer from the HTML spec 15:26:30 ... Get the return value, an object with properties, then use those as part of the constructor. We call the operation IsDetachedBuffer, could be something higher level rather than reimplementing 15:26:47 Eugene: Good idea, I'll look into it and answer in the PR 15:26:48 eugene (IRC): for reference: https://html.spec.whatwg.org/multipage/structured-data.html#structuredserializewithtransfer this is the operation 15:27:18 Chris: Anything else on this item? 15:27:40 Paul: An extension, once you've encoded your image, for example a video frame, the memory can be acquired again 15:27:58 ... So give a 4K frame to the video encoder, it optionally gives you back the memory of the 4K frame 15:28:26 ... Ideally we can reuse the texture or buffer cycle. There's an API proposal linked in the issue, an argument in the callback 15:29:18 ... You take a regular arraybuffer, give to encoder, it gets detached, the browser might collect it eventually. The frame is encoded so it's useless 15:29:46 ... Proposal is to give it back to JS to it can reuse it, if often have frame of same size 15:30:06 .. VideoFrame can have an arraybuffer inside it. It works for software, not so much for hardware 15:30:18 Eugene: What if we encode the same frame with another VideoEncoder? 15:30:25 Paul: Only works if you transfer 15:31:14 ... So give the frame, so the implementation becomes the owner. Saves a large number of allocation of the same size, to speed up the transcoding case 15:31:45 Eugene: This would work nicely with this PR, as we'll take an arraybuffer, transfer to VideoFrame, then transfer to encode, then VideoFrame gives the buffer back. Could consider adding a new callback to give the buffers back 15:31:59 https://github.com/w3c/webcodecs/issues/212 15:31:59 https://github.com/w3c/webcodecs/issues/212#issuecomment-832075352 15:32:18 Dan: I have implementation concerns, due to how frames are refcounted, we may not know when to return them 15:32:37 Paul: This was commented by someone, link in the minutes 15:33:16 ChrisN: Youenn, any view on this? 15:33:31 Youenn: Not sure, maybe copy can be done in one place? 15:34:15 Paul: Might want to zero the memory before handing it back, it'll be touch a lot, costly to send back to the pool. If you can just send the original frame without touching it at all you save reallocation or somewhere in the JS engine would have to zero it out 15:34:43 ... As a point of reference, if you're attempting to do realtime 4K encoding, just zeroing the memory makes you over budget with DDR4 15:34:50 Bernard: We've seen that with gaming 15:35:00 Youenn: What about closing? 15:35:32 Paul: It's OK to give JS back the memory it knows about. But hard to have new data created from script with data from previous use 15:36:06 ... Proposal to make it bounded, rather than reuse the same backing buffer somehow as part of another API call. If we're forced to do that, creates security or performacne concerns 15:36:17 ... It's all commented in the issue. The actual IDL change is small 15:36:54 ... In a realtime scenario with high frame rate and fast software encoder, time durations are annoying 15:37:08 ... It also busts the memory busts, so slows down other CPU operations 15:37:22 Youenn: In general we prefer hardware encoders 15:37:30 ... I'll look at the issue 15:37:42 present+ Yuhao_Fu 15:38:06 Topic: Orientation Metadata 15:38:10 Chris: https://github.com/w3c/webcodecs/issues/351 15:38:52 Dan: This is a hole in the spec for a while. We have frames with orientation metadata internally, eg., in the container 15:39:09 ... Rendered in video element it looks as expected, but copyTo has no way to signal 15:39:30 ... Could hide it, so copyTo acts like the frame is rotated, so developers don't need to know about orientation 15:39:46 ... Other way is to expose orientation metadata in the APIs 15:40:13 ... Orientated frames will be rare, capture from a rotated camera, we're not getting feedback about that 15:40:49 ... Want to feed it into decoders. Is it OK to hide this, or expose the metadata which is what low level video developers are used to 15:41:13 Youenn: It's not that rare, in Safari iOS. We don't expose VideoFrame there yet. It's important and will happen a lot 15:41:38 ... Might be difficult for developers to get it right. Two browsers on same device, one might handle differently to the other 15:42:00 ... If you create a VideoFrame by hand, isn't it useful to provide the orientation metadata somehow? 15:42:11 ... Thoughts on potential usage of metadata outside of this? 15:42:32 Dan: If you know the frame orientation you can configure the encoder to use it, can have savings 15:42:51 ... So there are some reasons to expose it, but things we could get to later as people need them 15:43:02 Youenn: What would be the default? 15:43:24 Dan: Apply orientation metadata first, then encode - unless we add future API controls to change that 15:43:28 Youenn: OK 15:43:56 Paul: How would it work, to not apply the transformation? 15:44:17 Dan: If the encoder config matches the incoming video frame, you wouldn't need to transform 15:45:05 Paul: So you can't do pixel data repack, send to canvas, you lose the orientation at this point, for modifying the frame? You go outside the realm of WebCodecs so the metadata is lost 15:45:15 Dan: Yes. There's CSS properties, but it's a mess 15:45:39 Paul: Could be useful to have implicit but know the orientation anyway. I think I agree with you 15:46:10 Youenn: So there'd be a need for an option to say don't orient before encoding? Do not apply transformation at all 15:46:43 Dan: Can change every frame. Would these back and forth switches be common in practice? 15:47:02 Youenn: Can be quite a bit on change of phone orientation. Size of video frame is the same, so we don't reconfigure 15:47:58 Dan: We'd have to output a new config, a new concept, don't currently have a non-keyframe config change config. Looking at that for color anyway 15:48:34 Paul: An exposed implementation detail proposal is workable 15:48:53 Dan: Worry that per-spec we'd treat as a size change, not what you want 15:49:05 Paul: What is shipped currently in Chrome? 15:49:28 Dan: Curently encoders don't know about orientation, so they'd not see the orientation change, so they'd visually rotate 15:49:48 Paul: The video wouldn't change dimensions, just rotate suddenly 15:50:22 Dan: Something to think about. As people don't complain already, maybe we're making a copy 15:50:29 Bernard: Dont' rely on people complaining! 15:51:03 Dan: Sounds like there's general support, but have extensions in the future? 15:51:20 Paul: Thinking about if we don't expose, but what if you use some other API than WebCodecs? 15:51:51 Bernard: I think it'll be a problem, create something like a CVO header extension to keep the orientation 15:52:14 Youenn: Also background blur and suddenly you have to do orientiation, could it be a performance issue as well? 15:52:21 Bernard: Yes 15:53:13 Chris: Seems we're tending toward wanting to expose something? 15:53:23 Dan: Debate on whether to include in first version or not 15:53:34 ... I expect we'll implement eventually 15:54:13 Topic: Custom Error Types 15:54:17 ChrisN: https://github.com/w3c/webcodecs/issues/669 15:54:36 Eugene: I was looking at how WebCodecs handles different error cases. We have a list of DOMException reasons from the existing spec 15:54:51 ... Those reasons are pretty vague, don't cover all our needs 15:55:42 ... WebGPU introduces a new sublass of DOMException, GPUPipelineError. Maybe WebCodecs could indicate its error cases through its own exception subclass, to indicate errors with more precision 15:55:55 ... As a subclass of DOMException, can be done in a backwards compatible way 15:56:23 Paul: In Firefox we use the standard errors but also have a mechanism to log errors so developers have info without introducing fingerprinting 15:56:56 Bernard: We have almost no useful error info today. An example is hardware resources not being allocated, another is a data issue. Distinguishing those would be better than where we are now 15:57:02 ... Another is GPU crashes, etc 15:57:13 Paul: We should do a survey of what we could do 15:57:32 Bernard: Chrome doesn't do what the spec says, also doesn't distinguish data and resource issues 15:57:59 Eugene: Do you think it makes sense to introduce a subclass to convey more info, or go with the DOMException reason names 15:58:32 ... We can indicate in this or that case, which DOMException name is used. Doesn't matter if name matches precisely, so long as people can determine how to at onit 15:58:43 ... Might be fingerprinting surface 15:58:53 Paul: Doesn't have to though 15:59:04 Bernard: If you get into a lot of detail in the errors, it might be 15:59:05 [FWIW, interfaces/specs that extend DOMException: GPUPipelineError, OverconstrainedError, RTCError, WebTransportError, see https://dontcallmedom.github.io/webidlpedia/names/DOMException.html ] 15:59:22 ... My concern is we have nothing now 15:59:57 Dan: We originally wanted to add recoverable and non-recoverable error path. I'd support hHaving an error type that distingushing those 16:00:10 Bernard: Distinguish data from resource errors 16:00:30 ... Could also use WebGPU errors, e.g., if there's a GPU pipeline error 16:00:43 Eugene: Those errors are very WebGPU specific, dont' want to go there 16:01:13 ... So use OperatingError for resources and EncodingError for data 16:01:21 Bernard: sounds good 16:03:21 Eugene: So don't add a subclass at this stage, just clarify the names 16:03:31 Dan: Could be useful to add it in the future 16:04:49 RRSAgent, draft minutes 16:04:50 I have made the request to generate https://www.w3.org/2023/05/30-mediawg-minutes.html tidoust 16:05:27 Topic: Next meeting 16:05:32 Chris: Meeting in two weeks, scheduled for later in the day, let me know if you prefer this earlier time instead 16:05:34 [adjourned] 16:05:53 rrsagent, draft minutes 16:05:54 I have made the request to generate https://www.w3.org/2023/05/30-mediawg-minutes.html cpn 17:51:15 Zakim has left #mediawg