IRC log of mediawg on 2023-05-30

Timestamps are in UTC.

14:58:48 [RRSAgent]
RRSAgent has joined #mediawg
14:58:53 [RRSAgent]
logging to https://www.w3.org/2023/05/30-mediawg-irc
14:58:53 [Zakim]
Zakim has joined #mediawg
14:58:57 [cpn]
Meeting: Media WG meeting
14:59:08 [cpn]
Agenda: https://github.com/w3c/media-wg/blob/main/meetings/2023-05-30-Media_Working_Group_Teleconference-agenda.md
15:01:38 [cpn]
present: Chris_Needham, Bernard_Aboba, Dan_Sanders, Youenn_Fablet
15:01:52 [tidoust]
RRSAgent, make logs public
15:02:08 [tidoust]
present+ Francois
15:02:17 [cpn]
Chair: Chris
15:02:25 [eugene]
eugene has joined #mediawg
15:02:41 [padenot]
present+ padenot
15:02:58 [eugene]
present+ eugene
15:04:28 [sandersd]
sandersd has joined #mediawg
15:05:30 [padenot]
in the agenda, the Detach codec inputs link is wrong, the issue is: https://github.com/w3c/webcodecs/issues/104
15:07:32 [cpn]
Topic: Detach Codec Inputs
15:07:51 [cpn]
scribe+ cpn
15:08:08 [cpn]
Eugene: https://github.com/w3c/webcodecs/issues/104, there's a PR that adds a transfer list to avoid a copy
15:08:26 [cpn]
... The spec change is minimal but allows saving a copy when the frame is constructed
15:08:45 [cpn]
... PR is https://github.com/w3c/webcodecs/pull/676
15:09:11 [cpn]
... Can also consider it for audio and ImageDecoder, but this was a starting point
15:09:25 [cpn]
... If we agree on the approach for video, it'll be trivial to do those also
15:09:58 [cpn]
... We add one member to VideoFrameBufferInit. It's a list, potentially it might be a single item, as we never need to transfer more than one arraybuffer
15:10:10 [cpn]
... But for consistence with scoped clone and postMessage in workers, I made it a list
15:10:42 [cpn]
... In future could support separate array buffers for planes, so to future-proof it a bit, transfer list is an array
15:11:00 [cpn]
... Paul looked at the spec wording, I tried to incorporate his feedback
15:11:24 [cpn]
... We say what goes in the transfer list, we test if can be detached, if not we throw. Then detach everything in the transfer list
15:11:55 [cpn]
Youenn: What if there's something in the transfer list unrelated to the VideoFrame
15:12:45 [cpn]
Eugene: Everything gets detached, but we don't use the pixels. We give structured clone an object, and all buffers in the list get transfered. If they're in the object that's cloned they get transfered, otherwise detached
15:12:47 [tidoust]
RRSAgent, draft minutes
15:12:49 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/05/30-mediawg-minutes.html tidoust
15:13:00 [cpn]
... It's not the most obvious behaviour, but consistent with other places on the web
15:13:26 [cpn]
Youenn: Developers could make a mistake using it, but if consistent with postMessage and other places, it's OK I guess
15:13:57 [cpn]
Eugene: Any objections from anyone? If we're all happy, we intend to figure out wording, and merge. I have a prototype in Chromium, will send intent to ship?
15:13:59 [tidoust]
present+ Jer_Noble, Peter_Thatcher, Eric_Carlson
15:14:24 [cpn]
Dan: Are GPU buffers that are mapped transferreable?
15:14:27 [cpn]
Eugene: They're not
15:15:17 [cpn]
Paul: My comments are resolved. Two comments still open, for the first the sentence isn't useful, as we're transferring the data and not copying - line 3449 in the latest version
15:15:35 [cpn]
Eugene: "Use visible rectangle and layout"
15:15:47 [cpn]
Paul: You can do that, but not useful if you're copying
15:16:15 [cpn]
Eugene: I think we still need it, you might have an arraybuffer with gaps, and layout would tell you where to look for the planes. I took the wording from the copy case
15:16:30 [cpn]
... If we need to know it for when to copy from, we also need it in this case
15:17:09 [cpn]
Paul: The idea of the prose is to make a smaller copy. You have all the info about the layout so doesn't need normative text, it can be omitted so we don't say the same thing multiple times
15:17:13 [cpn]
Eugene: I'll drop it
15:17:31 [cpn]
Paul: The other comment is just a remark, nothing to do
15:17:55 [cpn]
Dan: If we remove that prose, it might remove ability to purposely change those things, e.g., change the visible rect to 0,0
15:18:16 [cpn]
... Might be useful to preserve for transfer. We'd transfer the same backing
15:18:35 [cpn]
... It's not required to produce the same visible rect as output
15:18:46 [cpn]
I want browsers to have the option to do it
15:19:08 [cpn]
Paul: If we think there's a compatibility issue, it needs to be in the spec
15:19:34 [cpn]
Dan: I also want to refrain from forcing browsers to not copy. E.g., the odd nv12 case, Chrome expands the size outwards
15:19:58 [cpn]
Paul: So if you specify a buffer in the transfer list, but the implemenation says it's cannot be done, would you prefer it to throw or silently copy?
15:20:12 [cpn]
Eugene: Think we need to silently copy, a browser might not implement this part of the spec
15:20:47 [cpn]
... We might have a buffer that doesn't work like this, e.g., a GPU backed buffer in WebGPU, or other corner cases where we can't guarantee transfer
15:21:27 [cpn]
Paul: Throwing is fine. Detach is an operation that can cause issues. In general I prefer to be explicit, but postMessage as precedent - or generally, transfer lists
15:21:58 [cpn]
Eugene: Another thing we can't transfer is SharedArrayBuffer. That might be preserved by a worker, and we don't want pixels to be changed from under the VideoFrame
15:22:05 [cpn]
Paul: Can make it throw in that case
15:22:35 [cpn]
Eugene: Dan, if you have more comments, please add to the PR
15:22:48 [cpn]
... I'll add detail on the errors throw, in what cases
15:23:01 [cpn]
Paul: I approved the PR, add a non-normative note
15:23:31 [cpn]
Youenn: Would be good to check edge cases around structured clone, e.g., if you have the same arraybuffer twice in the transferlist
15:23:48 [cpn]
... Ideally we'd just call structured clone, but good to check all possible cases
15:24:27 [cpn]
Eugene: I'll do this. We can't use structured clone as it makes an arraybuffer out of another arraybuffer, and we need to move it to a VideoFrame. But we're aiming to be like structured clone
15:24:35 [cpn]
Paul: Why can't we use structured clone?
15:25:17 [cpn]
Eugene: It takes an object and clones or detaches. For us it's tricky as there's no visible arraybuffer after moving to the VideoFrame, it'll be an implementation detail
15:25:58 [cpn]
Paul: It's an abstract operation, take what you need from it and give it to the VideoFrame object. Call the structured serialize transfer from the HTML spec
15:26:30 [cpn]
... Get the return value, an object with properties, then use those as part of the constructor. We call the operation IsDetachedBuffer, could be something higher level rather than reimplementing
15:26:47 [cpn]
Eugene: Good idea, I'll look into it and answer in the PR
15:26:48 [padenot]
eugene (IRC): for reference: https://html.spec.whatwg.org/multipage/structured-data.html#structuredserializewithtransfer this is the operation
15:27:18 [cpn]
Chris: Anything else on this item?
15:27:40 [cpn]
Paul: An extension, once you've encoded your image, for example a video frame, the memory can be acquired again
15:27:58 [cpn]
... So give a 4K frame to the video encoder, it optionally gives you back the memory of the 4K frame
15:28:26 [cpn]
... Ideally we can reuse the texture or buffer cycle. There's an API proposal linked in the issue, an argument in the callback
15:29:18 [cpn]
... You take a regular arraybuffer, give to encoder, it gets detached, the browser might collect it eventually. The frame is encoded so it's useless
15:29:46 [cpn]
... Proposal is to give it back to JS to it can reuse it, if often have frame of same size
15:30:06 [cpn]
.. VideoFrame can have an arraybuffer inside it. It works for software, not so much for hardware
15:30:18 [cpn]
Eugene: What if we encode the same frame with another VideoEncoder?
15:30:25 [cpn]
Paul: Only works if you transfer
15:31:14 [cpn]
... So give the frame, so the implementation becomes the owner. Saves a large number of allocation of the same size, to speed up the transcoding case
15:31:45 [cpn]
Eugene: This would work nicely with this PR, as we'll take an arraybuffer, transfer to VideoFrame, then transfer to encode, then VideoFrame gives the buffer back. Could consider adding a new callback to give the buffers back
15:31:59 [tidoust]
https://github.com/w3c/webcodecs/issues/212
15:31:59 [padenot]
https://github.com/w3c/webcodecs/issues/212#issuecomment-832075352
15:32:18 [cpn]
Dan: I have implementation concerns, due to how frames are refcounted, we may not know when to return them
15:32:37 [cpn]
Paul: This was commented by someone, link in the minutes
15:33:16 [cpn]
ChrisN: Youenn, any view on this?
15:33:31 [cpn]
Youenn: Not sure, maybe copy can be done in one place?
15:34:15 [cpn]
Paul: Might want to zero the memory before handing it back, it'll be touch a lot, costly to send back to the pool. If you can just send the original frame without touching it at all you save reallocation or somewhere in the JS engine would have to zero it out
15:34:43 [cpn]
... As a point of reference, if you're attempting to do realtime 4K encoding, just zeroing the memory makes you over budget with DDR4
15:34:50 [cpn]
Bernard: We've seen that with gaming
15:35:00 [cpn]
Youenn: What about closing?
15:35:32 [cpn]
Paul: It's OK to give JS back the memory it knows about. But hard to have new data created from script with data from previous use
15:36:06 [cpn]
... Proposal to make it bounded, rather than reuse the same backing buffer somehow as part of another API call. If we're forced to do that, creates security or performacne concerns
15:36:17 [cpn]
... It's all commented in the issue. The actual IDL change is small
15:36:54 [cpn]
... In a realtime scenario with high frame rate and fast software encoder, time durations are annoying
15:37:08 [cpn]
... It also busts the memory busts, so slows down other CPU operations
15:37:22 [cpn]
Youenn: In general we prefer hardware encoders
15:37:30 [cpn]
... I'll look at the issue
15:37:42 [tidoust]
present+ Yuhao_Fu
15:38:06 [cpn]
Topic: Orientation Metadata
15:38:10 [cpn]
Chris: https://github.com/w3c/webcodecs/issues/351
15:38:52 [cpn]
Dan: This is a hole in the spec for a while. We have frames with orientation metadata internally, eg., in the container
15:39:09 [cpn]
... Rendered in video element it looks as expected, but copyTo has no way to signal
15:39:30 [cpn]
... Could hide it, so copyTo acts like the frame is rotated, so developers don't need to know about orientation
15:39:46 [cpn]
... Other way is to expose orientation metadata in the APIs
15:40:13 [cpn]
... Orientated frames will be rare, capture from a rotated camera, we're not getting feedback about that
15:40:49 [cpn]
... Want to feed it into decoders. Is it OK to hide this, or expose the metadata which is what low level video developers are used to
15:41:13 [cpn]
Youenn: It's not that rare, in Safari iOS. We don't expose VideoFrame there yet. It's important and will happen a lot
15:41:38 [cpn]
... Might be difficult for developers to get it right. Two browsers on same device, one might handle differently to the other
15:42:00 [cpn]
... If you create a VideoFrame by hand, isn't it useful to provide the orientation metadata somehow?
15:42:11 [cpn]
... Thoughts on potential usage of metadata outside of this?
15:42:32 [cpn]
Dan: If you know the frame orientation you can configure the encoder to use it, can have savings
15:42:51 [cpn]
... So there are some reasons to expose it, but things we could get to later as people need them
15:43:02 [cpn]
Youenn: What would be the default?
15:43:24 [cpn]
Dan: Apply orientation metadata first, then encode - unless we add future API controls to change that
15:43:28 [cpn]
Youenn: OK
15:43:56 [cpn]
Paul: How would it work, to not apply the transformation?
15:44:17 [cpn]
Dan: If the encoder config matches the incoming video frame, you wouldn't need to transform
15:45:05 [cpn]
Paul: So you can't do pixel data repack, send to canvas, you lose the orientation at this point, for modifying the frame? You go outside the realm of WebCodecs so the metadata is lost
15:45:15 [cpn]
Dan: Yes. There's CSS properties, but it's a mess
15:45:39 [cpn]
Paul: Could be useful to have implicit but know the orientation anyway. I think I agree with you
15:46:10 [cpn]
Youenn: So there'd be a need for an option to say don't orient before encoding? Do not apply transformation at all
15:46:43 [cpn]
Dan: Can change every frame. Would these back and forth switches be common in practice?
15:47:02 [cpn]
Youenn: Can be quite a bit on change of phone orientation. Size of video frame is the same, so we don't reconfigure
15:47:58 [cpn]
Dan: We'd have to output a new config, a new concept, don't currently have a non-keyframe config change config. Looking at that for color anyway
15:48:34 [cpn]
Paul: An exposed implementation detail proposal is workable
15:48:53 [cpn]
Dan: Worry that per-spec we'd treat as a size change, not what you want
15:49:05 [cpn]
Paul: What is shipped currently in Chrome?
15:49:28 [cpn]
Dan: Curently encoders don't know about orientation, so they'd not see the orientation change, so they'd visually rotate
15:49:48 [cpn]
Paul: The video wouldn't change dimensions, just rotate suddenly
15:50:22 [cpn]
Dan: Something to think about. As people don't complain already, maybe we're making a copy
15:50:29 [cpn]
Bernard: Dont' rely on people complaining!
15:51:03 [cpn]
Dan: Sounds like there's general support, but have extensions in the future?
15:51:20 [cpn]
Paul: Thinking about if we don't expose, but what if you use some other API than WebCodecs?
15:51:51 [cpn]
Bernard: I think it'll be a problem, create something like a CVO header extension to keep the orientation
15:52:14 [cpn]
Youenn: Also background blur and suddenly you have to do orientiation, could it be a performance issue as well?
15:52:21 [cpn]
Bernard: Yes
15:53:13 [cpn]
Chris: Seems we're tending toward wanting to expose something?
15:53:23 [cpn]
Dan: Debate on whether to include in first version or not
15:53:34 [cpn]
... I expect we'll implement eventually
15:54:13 [cpn]
Topic: Custom Error Types
15:54:17 [cpn]
ChrisN: https://github.com/w3c/webcodecs/issues/669
15:54:36 [cpn]
Eugene: I was looking at how WebCodecs handles different error cases. We have a list of DOMException reasons from the existing spec
15:54:51 [cpn]
... Those reasons are pretty vague, don't cover all our needs
15:55:42 [cpn]
... WebGPU introduces a new sublass of DOMException, GPUPipelineError. Maybe WebCodecs could indicate its error cases through its own exception subclass, to indicate errors with more precision
15:55:55 [cpn]
... As a subclass of DOMException, can be done in a backwards compatible way
15:56:23 [cpn]
Paul: In Firefox we use the standard errors but also have a mechanism to log errors so developers have info without introducing fingerprinting
15:56:56 [cpn]
Bernard: We have almost no useful error info today. An example is hardware resources not being allocated, another is a data issue. Distinguishing those would be better than where we are now
15:57:02 [cpn]
... Another is GPU crashes, etc
15:57:13 [cpn]
Paul: We should do a survey of what we could do
15:57:32 [cpn]
Bernard: Chrome doesn't do what the spec says, also doesn't distinguish data and resource issues
15:57:59 [cpn]
Eugene: Do you think it makes sense to introduce a subclass to convey more info, or go with the DOMException reason names
15:58:32 [cpn]
... We can indicate in this or that case, which DOMException name is used. Doesn't matter if name matches precisely, so long as people can determine how to at onit
15:58:43 [cpn]
... Might be fingerprinting surface
15:58:53 [cpn]
Paul: Doesn't have to though
15:59:04 [cpn]
Bernard: If you get into a lot of detail in the errors, it might be
15:59:05 [tidoust]
[FWIW, interfaces/specs that extend DOMException: GPUPipelineError, OverconstrainedError, RTCError, WebTransportError, see https://dontcallmedom.github.io/webidlpedia/names/DOMException.html ]
15:59:22 [cpn]
... My concern is we have nothing now
15:59:57 [cpn]
Dan: We originally wanted to add recoverable and non-recoverable error path. I'd support hHaving an error type that distingushing those
16:00:10 [cpn]
Bernard: Distinguish data from resource errors
16:00:30 [cpn]
... Could also use WebGPU errors, e.g., if there's a GPU pipeline error
16:00:43 [cpn]
Eugene: Those errors are very WebGPU specific, dont' want to go there
16:01:13 [cpn]
... So use OperatingError for resources and EncodingError for data
16:01:21 [cpn]
Bernard: sounds good
16:03:21 [cpn]
Eugene: So don't add a subclass at this stage, just clarify the names
16:03:31 [cpn]
Dan: Could be useful to add it in the future
16:04:49 [tidoust]
RRSAgent, draft minutes
16:04:50 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/05/30-mediawg-minutes.html tidoust
16:05:27 [cpn]
Topic: Next meeting
16:05:32 [cpn]
Chris: Meeting in two weeks, scheduled for later in the day, let me know if you prefer this earlier time instead
16:05:34 [cpn]
[adjourned]
16:05:53 [cpn]
rrsagent, draft minutes
16:05:54 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/05/30-mediawg-minutes.html cpn
17:51:15 [Zakim]
Zakim has left #mediawg