15:55:08 RRSAgent has joined #me 15:55:08 logging to https://www.w3.org/2022/02/21-me-irc 15:55:12 Zakim has joined #me 15:55:22 Meeting: Media Timed Events TF 15:55:28 scribe+ cpn 15:55:58 Agenda: https://www.w3.org/events/meetings/c336d9ef-1b72-4116-839f-322586916d0f 15:58:07 takio has joined #me 16:00:29 present+ Fuqiao_Xue, Takio_Yamaoka, Chris_Needham 16:01:29 nigel has joined #me 16:01:51 present+ Yuhao_Fu, Nigel_Megitt, Xabier_Rodriguez_Calvar 16:01:58 Present+ Nigel_Megitt_BBC 16:02:03 Present- Nigel_Megitt_BBC 16:02:05 present+ Rob_Smith 16:02:18 tidoust has joined #me 16:02:20 RobSmith has joined #me 16:02:26 calvaris has joined #me 16:03:47 present+ Kaz_Ashimura 16:03:55 zakim, who is on the call? 16:03:55 Present: Fuqiao_Xue, Takio_Yamaoka, Chris_Needham, Yuhao_Fu, Nigel_Megitt, Xabier_Rodriguez_Calvar, Rob_Smith, Kaz_Ashimura 16:04:05 present+ Francois_Daoust 16:04:40 rrsagent, make log public 16:04:44 rrsagent, draft minutes 16:04:44 I have made the request to generate https://www.w3.org/2022/02/21-me-minutes.html kaz 16:04:52 Agenda: https://www.w3.org/events/meetings/c336d9ef-1b72-4116-839f-322586916d0f 16:05:35 Topic: DataCue status update 16:06:08 Chris: Work has slowed recently, not clear what to do next 16:06:55 ... Two main parts to the proposal: one DataCue API for timed metadata events, generic, store any object and trigger events during media playback 16:07:20 ... Second part was for surfacing DASH emsg events through DataCue 16:07:56 ... IF people want to progress the emsg part then we need additional contributors 16:08:14 q+ 16:08:29 ... The work started in MEIG, with a presentation from Giri with ATSC and 3GPP requirements 16:08:50 ... Needs input to develop the technical proposal 16:09:13 ... Can't and shouldn't do this myself 16:09:47 Karen has joined #ME 16:10:08 Kaz: Can you share the existing resources? 16:10:21 Chris: The documents are linked from GitHub page: https://github.com/WICG/datacue 16:10:30 -> https://github.com/WICG/datacue/issues/21 issue 21 16:10:37 -> https://github.com/WICG/datacue/blob/main/explainer.md Explainer 16:10:56 -> https://w3c.github.io/me-media-timed-events/ Requirements 16:12:02 scribe+ 16:12:04 Kaz: would we like to get an alternative or additional moderator? 16:12:08 Kaz: We might want to look for additional moderator? 16:12:15 s/Kaz: would we like to get an alternative or additional moderator?// 16:12:48 Chris: Yes, also an editor who can help write documents, specifically the emsg integration with MSE 16:13:23 ... But also for DataCue itself. Has value separate from emsg support 16:13:36 q? 16:13:38 ... Would like to progress that by itself 16:13:38 ack 16:13:42 s/ack// 16:13:43 ack k 16:14:09 q+ 16:14:19 ... Easier proposition, but needs editorial help 16:14:38 Kaz: So you can continue as moderator? 16:14:46 Chris: Yes 16:16:04 ... We don't have many companies pushing for emsg support, so want to confirm the need for this 16:16:25 Kaz: Go back to the use cases and requirements and ask people their interest 16:16:31 Chris: Yes 16:16:37 q? 16:16:39 -> https://w3c.github.io/me-media-timed-events/#use-cases Use Cases 16:16:42 ack kaz 16:17:20 Francois: Also update the GitHub issues to say progress is blocked? 16:17:29 Chris: Could be a good thing to do next 16:17:34 -> https://github.com/WICG/datacue/issues datacue issues 16:18:01 Chris: Will also talk with people at DASH-IF, they're working on interop issues around timed events 16:18:29 Topic: SEI events 16:18:59 Chris: Thank you to all who replied to the GitHub issues 16:19:18 ... https://github.com/leonardoFu/video-sei-event 16:19:34 -> https://github.com/w3c/media-and-entertainment/issues/82 Issue 82 - Video SEI events 16:19:39 ... Explainer: https://github.com/leonardoFu/video-sei-event/blob/main/explainer.md 16:19:52 ... and two open issues https://github.com/leonardoFu/video-sei-event/issues 16:20:21 ... Let's discuss the open issues and decide how to update the explainer 16:20:56 ... The goal is understand the use cases for SEI events and turn that into a technical API proposal 16:21:40 ... One thing I would like to understand is if this is a new API proposal, or if it aligns with DataCue 16:21:43 -> https://github.com/leonardoFu/video-sei-event/issues/2 leonardo's repo - issue 2 - Interaction with Encrypted Media Extensions 16:21:47 -> https://github.com/leonardoFu/video-sei-event/issues/3 16:22:15 s|issues/3|issues/3 leonardo's repo - issue 3 - Timing accuracy and decode/presentation ordering 16:22:30 subtopic: issue 2 16:22:42 -> https://github.com/leonardoFu/video-sei-event/issues/2 leonardo's repo - issue 2 - Interaction with Encrypted Media Extensions 16:22:53 Chris: On issue 2 (EME integration). This was mostly a clarification question, are the SEI events part of the encrypted bit stream? 16:23:13 .. It seems so 16:23:23 q+ to agree this must be the right answer 16:23:40 ... So would be good to describe in the explainer 16:23:55 Nigel: I agree, sounds like the only reasonable answer 16:23:58 ack n 16:23:58 nigel, you wanted to agree this must be the right answer 16:24:34 Chris: I recommend describing that in the explainer, to set the scope of the solution proposal 16:25:38 ... So with EME it may not be possible to surface EME events, so worth clarifying in the explainer 16:25:53 q? 16:27:11 Yuhao: Why can't we get the information. After decryption we have raw video frames 16:28:58 ... After EME the video frame is handled by the rendering. So in this case maybe we can't get the information 16:29:28 ... When using WebCodecs, how to handle? Does WebCodecs work with EME? 16:29:37 Chris: I don't think it does, so we don't have the same limitation 16:29:40 q+ 16:30:01 Yuhao: If we use WebCodecs, we need another way to do decryption, WASM or JS 16:30:53 Chris: With WebCodecs you would parse the video bitstream from the container in WASM or JS, then pass the video bitstream to WebCodecs 16:31:19 ... If you're using encryption, that would have to be after parsing the container 16:32:12 Kaz: Let's clarify each use case, and what kind of framework and mechanism should be applied to each part for the expected service 16:32:43 ... What kind of extension is expected here? 16:33:06 Chris: Yes, encryption is a point of detail really 16:34:03 Yuhao: Should we look at the media handling to see if we can get the video bitstream? 16:34:26 Chris: It's worth looking, yes. But unsure it's possible 16:35:27 Yuhao: I can spend some time on that. Suppose we can get the information after the EME pipeline, and we can still get SEI information? 16:36:06 Chris: Let's update issue #2 with that information, then use this to write a short description into the explainer 16:36:58 Subtopic: Issue 3 (timing accuracy and ordering) 16:37:02 Chris: https://github.com/leonardoFu/video-sei-event/issues/3 16:37:31 Chris: I notice that we said we want to have access to SEI events in decode order rather than presentation order 16:37:51 ... Looking at WebCodecs issue https://github.com/w3c/webcodecs/issues/198 16:38:16 ... This is the related issue for WebCodecs 16:38:34 ... We should bring our requirements to this GitHub issue 16:39:06 ... The WebCodecs spec describes that VideoFrames are output in presentation order https://w3c.github.io/webcodecs/#videodecoder-methods 16:39:43 ... So if want the SEI events in decode order, how does that affect the API we propose? 16:40:04 ... If we want SEI events in presentation order, we can propose to attach them to VideoFrame 16:40:24 i/Kaz: We might want/scribenick: cpn/ 16:40:31 rrsagent, make log public 16:40:35 rrsagent, draft minutes 16:40:35 I have made the request to generate https://www.w3.org/2022/02/21-me-minutes.html kaz 16:40:41 ... But if we want them in decode order, we may need an event handler so the VideoDecoder can surface the events earlier 16:41:27 ... The explainer doesn't describe WebCodecs currently 16:41:28 ... https://github.com/leonardoFu/video-sei-event/blob/main/explainer.md 16:41:58 ... So should we add it? Or should we focus on the video element? 16:43:16 Yuhao: I worked with WebCodecs recently. In some cases it's a better way to get the information from the media stream 16:43:31 ... And it works well with WebRTC, so it should be considered 16:43:32 q+ 16:43:38 ack k 16:43:46 ack n 16:44:05 Nigel: Would it make sense to require WebCodecs to fire these events if it sees them? 16:44:25 ... It would know how to fire them in the right order. If WebCodecs decodes the video it should provide this functionality 16:44:50 ... You want anything that's decoding video to do it, somehow. But may be harder to define in the general case 16:45:42 Chris: Do we need to write something in the explainer, or do we simply add our input to https://github.com/w3c/webcodecs/issues/198 16:46:16 ... The explainer could reference the GitHub issue 16:46:34 q? 16:46:48 Francois: What I'd like to see is more a consideration of the synchronisation needs that the use cases have 16:46:49 q+ 16:47:14 ... If we're talking about frame accuracy, are events a good approach? 16:47:37 ... The explainer has AI based subtitles for a live stream, which might not require frame accuracy 16:48:06 ... If you use SEI metadata for volumetric video where you need strong synchronization between SEI metadata and video, then events aren't going to work 16:48:50 q? 16:49:05 rrsagent, draft minutes 16:49:05 I have made the request to generate https://www.w3.org/2022/02/21-me-minutes.html kaz 16:49:21 Chris: Issue 3 talks about timing requirements: https://github.com/leonardoFu/video-sei-event/issues/3 16:49:44 Francois: Events could be good enough for 100 ms 16:49:49 +1 to tidoust 16:50:19 Chris: Do we have a use case that requires more timing accuracy? 16:51:03 Yuhao: In my use case, don't need to render the SEI information on the exact frame where the SEI is payloaded. It can stand a 100ms tolerance 16:51:29 ... If we really want to synchronise the frame with SEI, we can use WebCodecs with Canvas, to really control the frame to render 16:52:10 ... The SEI can be a property on the VideoFrame 16:52:38 ... I can see use cases such as video editing, where the need for accuracy is high 16:53:09 ... Seeking a video element isn't as accurate as needed. So use WebCodecs and Canvas to really control rendering 16:53:46 ... But SEI event may not be for this case 16:54:37 ... The proposal is intended to be easy to use. Calculate the end to end latency. So I don't need to match SEI with the video frame 16:55:20 ... Just need as exact as possible, also the presentation and decode order may not be so important 16:55:30 q? 16:55:45 q+ 16:56:20 Francois: It could be useful to describe that the explainer is for the loose synchronization use cases, and WebCodecs is for high accurate synchronization 16:56:39 q+ to ask about sync of message vs eventual correctness 16:56:46 Kaz: Agree to work on expectations for use cases and actual need from services, then look at potential application later 16:57:10 Karen has joined #ME 16:57:21 ... One potential use case that might require precise synchronization could be interactive multimodal avatars 16:57:43 ... Responding to speech in a realtime manner, which needs low latency 16:57:56 ... Clarify if we need to handle such advanced use cases 16:57:57 q? 16:58:01 fack k 16:58:06 ack k 16:58:07 s/fack k// 16:58:31 Rob: The video editor is interesting, no real time element, but you want precision to associate with video frames 16:59:01 ... Regarding loose vs tight sync, we're looking at this in WebVMT. Location is a low sync use case, but orientation is a tight sync use case 16:59:34 ... If you turn the camera around, the video frames change very quickly. AR use cases. Not sure what solutions are being used in AR 16:59:37 q? 16:59:42 ack Rob 17:00:03 ... A mobile phone streaming video, with a website that overlays information using AR 17:00:15 ack n 17:00:15 nigel, you wanted to ask about sync of message vs eventual correctness 17:00:55 Nigel: With applications that are doing something with metadata in realtime, human sensitivity to the timing may be higher than doing video editing 17:01:34 ... In video editing you need to know the timing. When you pause you want the correct state for the video 17:01:49 ... frame that you're paused on 17:01:54 Accuracy vs latency 17:02:22 ... Synchronisation of when messages are fired, so how quickly you can respond, then how quickly the state is updated 17:02:59 ... For example if events are fired 120 ms behind live playback, then you pause. You want the events to come out correctly, so that the view you end up with is the correct one for that frame 17:04:26 Chris: Do we have that captured in a doc already? If not we should add to this explainer 17:05:07 Nigel: In a real world application for authoring captions and subtitles 17:05:35 q+ 17:06:02 Chris: Helpful to put application description into the explainer 17:06:28 s/may be higher than doing video editing/may be higher when doing video editing 17:06:47 RobSmith: I created a demo with video on two smartphones, then used WebVMT to sync them, then had to deal with latency in the browser 17:07:03 ... Multiple cameras observing the same scene 17:08:26 Chris: For the explainer, describe the synchronization goals 17:09:28 ... If that's loose sync, we should look at potential alignment with DataCue, which would already gives loose sync 17:09:47 ... Applications can use VTTCue today, so can be a way to prototype 17:10:24 WebVMT video sync demo: https://webvmt.org/webvmt/blog#20201229 17:10:31 ... I'll add a GitHub issue about DataCue, let's discuss there 17:12:04 ... Use an existing mechanism if we can, or avoid proposing multiple solutions to similar problems 17:12:35 q? 17:12:42 ack Rob 17:13:12 q+ 17:13:39 Kaz: Will we try to generate some concrete use case description, or continue discussing the explainer 17:14:09 ack k 17:14:12 Chris: I think the use cases could have some more detail 17:14:38 q+ 17:16:03 Takio: There other mechanisms such as ID3, video.js can use that. Recommend clarifying use cases, then do a gap analysis 17:16:18 Chris: I agree 17:16:53 Topic: Next meeting 17:17:07 Chris: Next planned meeting is March 21. Should we meet earlier? 17:18:28 ... Can talk about plan, once we have the explainer ready 17:18:50 ack t 17:19:15 rrsagent, draft minutes 17:19:15 I have made the request to generate https://www.w3.org/2022/02/21-me-minutes.html kaz 17:19:37 chair: ChrisN 17:19:39 rrsagent, draft minutes 17:19:39 I have made the request to generate https://www.w3.org/2022/02/21-me-minutes.html kaz 17:20:40 rrsagent, make log public 17:21:06 [adjourned] 17:21:08 rrsagent, draft minutes 17:21:08 I have made the request to generate https://www.w3.org/2022/02/21-me-minutes.html kaz 17:38:38 nigel has joined #me 17:43:41 tidoust has joined #me 17:58:59 Karen has joined #ME 18:25:06 nigel has joined #me 18:29:19 Karen has joined #ME 18:45:11 nigel has joined #me 18:59:11 Karen has joined #ME 19:43:48 nigel has joined #me 20:18:15 tidoust has joined #me 20:44:38 nigel has joined #me 21:22:32 Zakim has left #me 21:52:10 Karen has joined #ME 22:47:54 nigel has joined #me 23:45:27 nigel has joined #me