IRC log of mediaprocessing on 2019-09-18
Timestamps are in UTC.
- 01:43:15 [RRSAgent]
- RRSAgent has joined #mediaprocessing
- 01:43:15 [RRSAgent]
- logging to https://www.w3.org/2019/09/18-mediaprocessing-irc
- 01:43:18 [wseltzer]
- rrsagent, make logs public
- 01:43:23 [wseltzer]
- wseltzer has changed the topic to: Efficient audio/video processing
- 01:46:12 [wseltzer]
- wseltzer has left #mediaprocessing
- 01:53:45 [koalie]
- koalie has joined #mediaprocessing
- 01:53:51 [koalie]
- koalie has changed the topic to: https://w3c.github.io/tpac-breakouts/sessions.html
- 01:53:55 [koalie]
- koalie has left #mediaprocessing
- 04:14:15 [takayud]
- takayud has joined #mediaprocessing
- 04:21:38 [tidoust]
- tidoust has joined #mediaprocessing
- 04:22:53 [tidoust]
- RRSAgent, draft minutes
- 04:22:53 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html tidoust
- 04:23:08 [tidoust]
- RRSAgent, make logs public
- 04:23:19 [riju]
- riju has joined #mediaprocessing
- 04:23:33 [tidoust]
- Meeting: Efficient media processing - TPAC breakout
- 04:23:40 [tidoust]
- Chair: Francois Daoust
- 04:23:48 [tidoust]
- RRSagent, draft minutes
- 04:23:48 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html tidoust
- 04:29:26 [yos]
- yos has joined #mediaprocessing
- 04:30:57 [anssik]
- anssik has joined #mediaprocessing
- 04:31:09 [takio]
- takio has joined #mediaprocessing
- 04:31:11 [anssik]
- Present+ Anssi_Kostiainen
- 04:31:13 [dom]
- dom has joined #mediaprocessing
- 04:31:16 [gkatsev]
- gkatsev has joined #mediaprocessing
- 04:31:29 [thomas]
- thomas has joined #mediaprocessing
- 04:31:35 [dom]
- Present+
- 04:31:43 [ericc_]
- ericc_ has joined #mediaprocessing
- 04:31:51 [kimwooglae]
- kimwooglae has joined #mediaprocessing
- 04:31:59 [Kiyoshi]
- Kiyoshi has joined #mediaprocessing
- 04:32:00 [riju]
- riju: Present+
- 04:32:03 [jib]
- jib has joined #mediaprocessing
- 04:32:03 [cpn]
- cpn has joined #mediaprocessing
- 04:32:08 [gkatsev]
- present+
- 04:32:09 [Joshue108]
- Joshue108 has joined #mediaprocessing
- 04:32:10 [cpn]
- present+ Chris_Needham
- 04:32:13 [dom]
- ScribeNick: anssik
- 04:32:16 [hhan]
- hhan has joined #mediaprocessing
- 04:32:16 [Joshue108]
- present+
- 04:32:25 [Jared]
- Jared has joined #mediaprocessing
- 04:32:31 [wonsuk]
- wonsuk has joined #mediaprocessing
- 04:32:42 [Gooroomee]
- Gooroomee has joined #mediaprocessing
- 04:32:46 [stepsteg]
- stepsteg has joined #mediaprocessing
- 04:32:51 [mdrimages]
- mdrimages has joined #mediaprocessing
- 04:32:55 [stepsteg]
- present+
- 04:33:07 [ken]
- ken has joined #mediaprocessing
- 04:33:07 [wonsuk]
- present+
- 04:33:10 [anssik]
- tidoust: looking at media activities across W3C, and interest in ensuring video and audio can be processed efficiently
- 04:33:35 [horiuchi]
- horiuchi has joined #mediaprocessing
- 04:33:37 [anssik]
- ... we have a number of group in this space, WebRTC, Machine Learning, etc. getting folks together to find shared interests
- 04:33:49 [dom]
- -> https://www.w3.org/2019/Talks/TPAC/efficient-media-processing/# Slides for Efficient audio/video processing intro
- 04:34:38 [anssik]
- ... use cases we've considered include e.g. media stream analysis, barcode reading, emotion analysis, speech recognition etc.
- 04:35:05 [anssik]
- ... custom codecs in JS or Wasm one emerging area of interest
- 04:35:34 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 04:35:57 [anssik]
- ... common need is to be able to process media somehow, on CPU, GPU, workers, again JS or Wasm, perhaps dedicated chipset in the device, or ML algos, or dedicated API such as face recognition as a higher-level abstraction
- 04:36:21 [anssik]
- ... what do I mean with efficiency? avoid duplication of bytes in memory to use that resource efficiently
- 04:36:42 [anssik]
- ... how to run multiple operations per stream or frame(?)
- 04:37:03 [anssik]
- ... media stream track and stream are the primitives we deal with from WebRTC
- 04:37:18 [anssik]
- ... do we need to define the mapping between the two?
- 04:37:33 [anssik]
- ... how to hook into media from Wasm context?
- 04:38:04 [anssik]
- ... also, what do we want to see standardized?
- 04:38:14 [niwamoto]
- niwamoto has joined #mediaprocessing
- 04:38:20 [disoul]
- disoul has joined #mediaprocessing
- 04:38:28 [wonsuk_]
- wonsuk_ has joined #mediaprocessing
- 04:38:39 [anssik]
- ... some of these use cases we'll see today can be already done today, perhaps not in the most efficient manner
- 04:38:53 [anssik]
- ... that's for the introduction
- 04:39:16 [anssik]
- ... riju from Intel will show some demos of use cases that can be implemented today
- 04:39:46 [anssik]
- ... paul from Mozilla and harald from Google will also chime in with their perspectives
- 04:40:06 [anssik]
- [riju showing slides]
- 04:40:22 [igarashi]
- igarashi has joined #mediaprocessing
- 04:40:26 [anssik]
- riju: 70% of internet traffic is media, so processing it efficiently is important
- 04:40:50 [anssik]
- ... HDR in browsers is one such experiment, want to capture, process and display photo in browser
- 04:41:18 [anssik]
- ... had to add exposure time feature to getUserMedia, use OpenCV.js to create the HDR photo
- 04:41:39 [anssik]
- ... to show power of media streams, everything you can do with native cameras, you can do on the web
- 04:41:45 [anssik]
- ... using SIMD and workers
- 04:42:15 [anssik]
- ... Facebook, Instragram are using native apps for media-related apps, but web can also do that today
- 04:42:47 [anssik]
- ... looking at fps we'll seeing 15 fps, real-time use cases require a bit higher performance for reasonable experience
- 04:43:17 [anssik]
- ... in these demos, no ML on the client, all processing on the CPU
- 04:43:53 [anssik]
- ... Facebook messenger funny hats ported to the web was one of the experiments, 15-20 fps on a typical laptop
- 04:44:49 [anssik]
- ... another use case was scanning, barcode reading, also possible without hw-accelerated implementation per our experiments
- 04:45:36 [anssik]
- ... yet another use case, camera pan and tilt
- 04:46:26 [anssik]
- ... specialized hardware for depth sensing has been enabled for the web via getUserMedia extension, use cases e.g. background segmentation
- 04:47:13 [anssik]
- ... less CPU compute needed when processing offloaded to specialized hardware
- 04:48:00 [anssik]
- ... experiments show Wasm with SIMD gave use 5-10x speedup in multiple use cases
- 04:48:14 [anssik]
- ???: which resolution is used in these demos?
- 04:48:38 [anssik]
- ????: impressive, what is the laptop specifically the demos were run on?
- 04:48:51 [tidoust]
- -> https://www.w3.org/2019/Talks/TPAC/efficient-media-processing/ Introduction slides
- 04:48:53 [anssik]
- riju: 10 fps on modern phones
- 04:49:10 [riju]
- https://github.com/riju/WebCamera
- 04:49:27 [anssik]
- paul: so I'm from Mozilla, back in the days, Firefox project did not have access to native APIs
- 04:49:37 [riju]
- On your chrome, try out : https://riju.github.io/WebCamera/samples/
- 04:50:07 [anssik]
- ... how to do competitive real-time image processing on the web was looked into at Mozilla around 2012-14
- 04:50:56 [cabanier]
- cabanier has joined #mediaprocessing
- 04:50:57 [ningxin_hu]
- ningxin_hu has joined #mediaprocessing
- 04:50:59 [anssik]
- ... findings were CPU on mobile was working but not working well enough, so investigated GPU path and found gaps that we were able to fix with a proprietary APIs only
- 04:51:18 [anssik]
- ... running CPU clocked on screen refresh(?)
- 04:51:55 [anssik]
- ... processing model of stacked filters is the most important part
- 04:52:15 [anssik]
- ... not compositing, but stacking within the same group
- 04:52:31 [anssik]
- ... we stopped Firefox OS and now we have access to better technology
- 04:53:04 [anssik]
- ... more recently, we started a Web Codec effort with Google, separate breakout 16:30 today
- 04:53:09 [Gooroomee2]
- Gooroomee2 has joined #mediaprocessing
- 04:53:52 [anssik]
- ... would like to coordinate across these two efforts
- 04:54:33 [anssik]
- [missed peter's comment re Web Codec]
- 04:54:53 [plamb_mozilla]
- plamb_mozilla has joined #mediaprocessing
- 04:55:36 [anssik]
- dom: any other feedback from Audio work that could apply to video processing
- 04:56:21 [anssik]
- paul: your API must be programmable
- 04:56:39 [tidoust]
- s/[missed peter's comment re Web Codec]/Peter: Mapping between Streams and MediaStreamTrack is in the WebCodecs proposal, but it doesn't have to be there. Could be done elsewhere for use cases that need it but don't need WebCodecs/
- 04:56:45 [anssik]
- ... Web Audio started as a node composition data flow pairing
- 04:56:57 [anssik]
- ... we had lots of nodes, that became an issue
- 04:57:15 [plamb_mozilla]
- plamb_mozilla has left #mediaprocessing
- 04:57:34 [anssik]
- ... let's also do a model understanding images are large and we cannot afford to do redundant copies
- 04:57:55 [anssik]
- [harald showing slides]
- 04:58:17 [anssik]
- harald: trying to make life easier for people who use RTCPeerConnection
- 04:58:32 [anssik]
- ... if I want to solve one problem only, then this is important
- 04:59:09 [anssik]
- [harald showing requirements, scribe assumes slides will be made available, so won't scribe the slide content]
- 04:59:55 [anssik]
- harald: we have a solution that works, as was demonstrated: use media and canvas elements
- 05:00:11 [anssik]
- ... we should be able to do better, using those primitives browsers cannot optimize
- 05:00:28 [anssik]
- ... existing apps that want to add image processing step, you should not need to redesign everything
- 05:00:48 [anssik]
- ... we believe composable objects are the future on the web
- 05:01:01 [anssik]
- ... when we see pieces going to the right direction, we should harmonize
- 05:01:32 [anssik]
- [harald showing drawing of WebRTC Streams flow]
- 05:02:07 [anssik]
- harald: let's give web apps ability to insert processing between getUserMedia and PeerConnection blocks
- 05:02:35 [anssik]
- ... Web Codec proposal says, encoder is in Stream
- 05:03:29 [anssik]
- ... WHATWG Streams are a concept that can be piped
- 05:04:01 [anssik]
- ... this is exactly what we need, how can I put this onto the Web with PeerConnection
- 05:04:41 [anssik]
- ... let's do that at insert-time with a Factory pattern
- 05:05:53 [tpk]
- tpk has joined #mediaprocessing
- 05:05:53 [anssik]
- [harald explaining a code example with encoderFactory]
- 05:06:36 [anssik]
- harald: JS has _some_ properties done right
- 05:07:23 [anssik]
- ... processing step can be offloaded to a worker that can read/write stream
- 05:08:46 [anssik]
- ... the goal of the project is to say: given a Stream add this translation into it -- seems like magic?
- 05:09:28 [dom]
- q?
- 05:09:29 [anssik]
- ... the end
- 05:09:37 [Zakim]
- Zakim has joined #mediaprocessing
- 05:09:56 [anssik]
- chris: processing in media element, is there a similar solution for that, consider with MSE?
- 05:10:09 [anssik]
- harald: this only for media that arrives with PeerConnection
- 05:10:35 [Maru]
- Maru has joined #mediaprocessing
- 05:10:41 [anssik]
- ... similar type of thinking could be applied there, not exactly this solution
- 05:10:46 [Steve_Anton]
- Steve_Anton has joined #mediaprocessing
- 05:11:22 [anssik]
- [scribe asks tidoust to scribe his comment]
- 05:12:33 [anssik]
- ?????: instead of putting this in sinks, common denominator MediaSteamTracks(?)
- 05:13:00 [tidoust]
- s/[scribe asks tidoust to scribe his comment]/Francois: To reformulate, the proposal is to create a hook into PeerConnection to plug WebCodecs. A similar hook is needed in MSE and other APIs./
- 05:13:14 [anssik]
- ... audio worklets need their own context running off the main thread
- 05:13:14 [tidoust]
- s/?????:/Jan-Ivar:/
- 05:13:36 [anssik]
- harald: this should be part of the worker, to solve that architectural issue
- 05:14:05 [anssik]
- ... what does PeerConnection actually does with Workers? A bit complex
- 05:14:12 [tpk]
- tpk has left #mediaprocessing
- 05:14:38 [anssik]
- ... APIs might allow codec switching on the fly
- 05:15:32 [anssik]
- ??????: is this specific to WebRTC? Is MediaRecorder in scope of the proposal?
- 05:15:53 [anssik]
- harald: when working with PeerConnection, porting over to MediaRecorder et al. doable
- 05:16:08 [dom]
- s/??????/Youenn_Fablet_Apple/
- 05:16:40 [anssik]
- Youenn_Fablet_Apple: funny hats would probably happen pre-encoding
- 05:17:38 [anssik]
- ... WebRTC questions, between PeerConnection and codec, what happens on the network level, two params depending on network connection, how to deal with that?
- 05:18:12 [riju]
- Here are the slides for use-cases for Media processing on Web platform that I presented on ImageProcessing+WASM+ImageCapture : http://bit.ly/rijuMediaTPAC
- 05:18:24 [anssik]
- harald: PeerConnection framework would same the messages, if it needs to modify things it can
- 05:18:58 [anssik]
- ??????_Google: changing format after encoding?
- 05:19:19 [disooo]
- disooo has joined #mediaprocessing
- 05:19:21 [anssik]
- harald: we haven't defined the format yet, so can do anything
- 05:19:53 [Steve_Anton]
- s/??????_Google/Emad_Omara_Google/
- 05:20:53 [anssik]
- Jan-Ivar: a different API like VideoWorklet, would it be a better abstraction?
- 05:21:48 [anssik]
- Youenn_Fablet_Apple: Stream piped was a great model, similar to audio graph, any learning from that?
- 05:22:46 [anssik]
- paul: cloning an area of optimization, separable filters
- 05:22:49 [dom]
- RRSAgent, draft minutes
- 05:22:49 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html dom
- 05:22:53 [kiyoshi]
- kiyoshi has joined #mediaprocessing
- 05:23:08 [anssik]
- tidoust: where can we continue this discussion, 2 minutes left!
- 05:23:31 [anssik]
- ... Web Codecs are part of the solution, generic solution on different level, not discussed exposing GPU
- 05:24:06 [anssik]
- ... no real group now, WebRTC, Audio, Media groups available but split attention, do we want a dedicated CG for incubating this specific issue on the table?
- 05:24:46 [anssik]
- harald: we have a point problem and point solution, we have a field of problems that need a field of solutions
- 05:25:02 [anssik]
- tidoust: any suggestions on a venue?
- 05:25:21 [anssik]
- ... Media WG could be it, but not in scope, maybe M&E IG?
- 05:25:36 [anssik]
- chris: we could consider to reformulate the IG to be the place
- 05:26:02 [anssik]
- tidoust: most stakeholders in this room are in Media WG
- 05:26:03 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 05:26:08 [anssik]
- ... thank you all!
- 05:26:13 [dom]
- RRSAgent, draft minutes
- 05:26:13 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html dom
- 05:26:15 [anssik]
- ... thanks Anssi for scribing!!!
- 05:26:25 [Maru]
- Maru has left #mediaprocessing
- 05:26:35 [tidoust]
- RRSAgent, draft minutes
- 05:26:35 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html tidoust
- 05:27:20 [NJ___]
- NJ___ has joined #mediaprocessing
- 05:27:53 [ericc]
- ericc has joined #mediaprocessing
- 05:30:55 [horiuchi]
- horiuchi has joined #mediaprocessing
- 05:31:19 [tidoust]
- tidoust has joined #mediaprocessing
- 05:31:53 [horiuchi]
- horiuchi has joined #mediaprocessing
- 05:39:12 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 05:39:52 [plamb_mozilla]
- plamb_mozilla has joined #mediaprocessing
- 05:41:13 [plamb_mozilla]
- plamb_mozilla has left #mediaprocessing
- 06:32:05 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 06:35:42 [horiuchi]
- horiuchi has joined #mediaprocessing
- 06:44:06 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 07:13:29 [horiuchi]
- horiuchi has joined #mediaprocessing
- 07:18:47 [tidoust]
- tidoust has joined #mediaprocessing
- 07:20:10 [horiuchi]
- horiuchi has joined #mediaprocessing
- 07:23:18 [dom]
- dom has joined #mediaprocessing
- 07:25:50 [Zakim]
- Zakim has left #mediaprocessing
- 07:26:44 [horiuchi_]
- horiuchi_ has joined #mediaprocessing
- 07:27:36 [ericc]
- ericc has joined #mediaprocessing
- 07:28:14 [dom]
- dom has left #mediaprocessing
- 07:31:19 [horiuchi]
- horiuchi has joined #mediaprocessing
- 07:31:52 [tidoust_]
- tidoust_ has joined #mediaprocessing
- 07:47:17 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 07:51:57 [tidoust_]
- tidoust_ has joined #mediaprocessing
- 07:59:56 [tidoust]
- tidoust has joined #mediaprocessing
- 08:23:31 [horiuchi]
- horiuchi has joined #mediaprocessing
- 08:31:24 [tidoust]
- tidoust has joined #mediaprocessing
- 08:33:22 [jib]
- jib has joined #mediaprocessing
- 08:34:44 [ericc]
- ericc has joined #mediaprocessing
- 08:37:41 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 08:38:14 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 09:33:21 [MasayaIk_]
- MasayaIk_ has joined #mediaprocessing
- 09:34:16 [horiuchi]
- horiuchi has joined #mediaprocessing
- 10:06:29 [MasayaIkeo]
- MasayaIkeo has joined #mediaprocessing
- 11:55:32 [tidoust]
- tidoust has joined #mediaprocessing