01:43:15 <RRSAgent> RRSAgent has joined #mediaprocessing
01:43:15 <RRSAgent> logging to https://www.w3.org/2019/09/18-mediaprocessing-irc
01:43:18 <wseltzer> rrsagent, make logs public
01:43:23 <wseltzer> wseltzer has changed the topic to: Efficient audio/video processing
01:46:12 <wseltzer> wseltzer has left #mediaprocessing
01:53:45 <koalie> koalie has joined #mediaprocessing
01:53:51 <koalie> koalie has changed the topic to: https://w3c.github.io/tpac-breakouts/sessions.html
01:53:55 <koalie> koalie has left #mediaprocessing
04:14:15 <takayud> takayud has joined #mediaprocessing
04:21:38 <tidoust> tidoust has joined #mediaprocessing
04:22:53 <tidoust> RRSAgent, draft minutes
04:22:53 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html tidoust
04:23:08 <tidoust> RRSAgent, make logs public
04:23:19 <riju> riju has joined #mediaprocessing
04:23:33 <tidoust> Meeting: Efficient media processing - TPAC breakout
04:23:40 <tidoust> Chair: Francois Daoust
04:23:48 <tidoust> RRSagent, draft minutes
04:23:48 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html tidoust
04:29:26 <yos> yos has joined #mediaprocessing
04:30:57 <anssik> anssik has joined #mediaprocessing
04:31:09 <takio> takio has joined #mediaprocessing
04:31:11 <anssik> Present+ Anssi_Kostiainen
04:31:13 <dom> dom has joined #mediaprocessing
04:31:16 <gkatsev> gkatsev has joined #mediaprocessing
04:31:29 <thomas> thomas has joined #mediaprocessing
04:31:35 <dom> Present+
04:31:43 <ericc_> ericc_ has joined #mediaprocessing
04:31:51 <kimwooglae> kimwooglae has joined #mediaprocessing
04:31:59 <Kiyoshi> Kiyoshi has joined #mediaprocessing
04:32:00 <riju> riju: Present+
04:32:03 <jib> jib has joined #mediaprocessing
04:32:03 <cpn> cpn has joined #mediaprocessing
04:32:08 <gkatsev> present+
04:32:09 <Joshue108> Joshue108 has joined #mediaprocessing
04:32:10 <cpn> present+ Chris_Needham
04:32:13 <dom> ScribeNick: anssik
04:32:16 <hhan> hhan has joined #mediaprocessing
04:32:16 <Joshue108> present+
04:32:25 <Jared> Jared has joined #mediaprocessing
04:32:31 <wonsuk> wonsuk has joined #mediaprocessing
04:32:42 <Gooroomee> Gooroomee has joined #mediaprocessing
04:32:46 <stepsteg> stepsteg has joined #mediaprocessing
04:32:51 <mdrimages> mdrimages has joined #mediaprocessing
04:32:55 <stepsteg> present+
04:33:07 <ken> ken has joined #mediaprocessing
04:33:07 <wonsuk> present+
04:33:10 <anssik> tidoust: looking at media activities across W3C, and interest in ensuring video and audio can be processed efficiently
04:33:35 <horiuchi> horiuchi has joined #mediaprocessing
04:33:37 <anssik> ... we have a number of group in this space, WebRTC, Machine Learning, etc. getting folks together to find shared interests
04:33:49 <dom> -> https://www.w3.org/2019/Talks/TPAC/efficient-media-processing/# Slides for Efficient audio/video processing intro
04:34:38 <anssik> ... use cases we've considered include e.g. media stream analysis, barcode reading, emotion analysis, speech recognition etc.
04:35:05 <anssik> ... custom codecs in JS or Wasm one emerging area of interest
04:35:34 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
04:35:57 <anssik> ... common need is to be able to process media somehow, on CPU, GPU, workers, again JS or Wasm, perhaps dedicated chipset in the device, or ML algos, or dedicated API such as face recognition as a higher-level abstraction
04:36:21 <anssik> ... what do I mean with efficiency? avoid duplication of bytes in memory to use that resource efficiently
04:36:42 <anssik> ... how to run multiple operations per stream or frame(?)
04:37:03 <anssik> ... media stream track and stream are the primitives we deal with from WebRTC
04:37:18 <anssik> ... do we need to define the mapping between the two?
04:37:33 <anssik> ... how to hook into media from Wasm context?
04:38:04 <anssik> ... also, what do we want to see standardized?
04:38:14 <niwamoto> niwamoto has joined #mediaprocessing
04:38:20 <disoul> disoul has joined #mediaprocessing
04:38:28 <wonsuk_> wonsuk_ has joined #mediaprocessing
04:38:39 <anssik> ... some of these use cases we'll see today can be already done today, perhaps not in the most efficient manner
04:38:53 <anssik> ... that's for the introduction
04:39:16 <anssik> ... riju from Intel will show some demos of use cases that can be implemented today
04:39:46 <anssik> ... paul from Mozilla and harald from Google will also chime in with their perspectives
04:40:06 <anssik> [riju showing slides]
04:40:22 <igarashi> igarashi has joined #mediaprocessing
04:40:26 <anssik> riju: 70% of internet traffic is media, so processing it efficiently is important
04:40:50 <anssik> ... HDR in browsers is one such experiment, want to capture, process and display photo in browser
04:41:18 <anssik> ... had to add exposure time feature to getUserMedia, use OpenCV.js to create the HDR photo
04:41:39 <anssik> ... to show power of media streams, everything you can do with native cameras, you can do on the web
04:41:45 <anssik> ... using SIMD and workers
04:42:15 <anssik> ... Facebook, Instragram are using native apps for media-related apps, but web can also do that today
04:42:47 <anssik> ... looking at fps we'll seeing 15 fps, real-time use cases require a bit higher performance for reasonable experience
04:43:17 <anssik> ... in these demos, no ML on the client, all processing on the CPU
04:43:53 <anssik> ... Facebook messenger funny hats ported to the web was one of the experiments, 15-20 fps on a typical laptop
04:44:49 <anssik> ... another use case was scanning, barcode reading, also possible without hw-accelerated implementation per our  experiments
04:45:36 <anssik> ... yet another use case, camera pan and tilt
04:46:26 <anssik> ... specialized hardware for depth sensing has been enabled for the web via getUserMedia extension, use cases e.g. background segmentation
04:47:13 <anssik> ... less CPU compute needed when processing offloaded to specialized hardware
04:48:00 <anssik> ... experiments show Wasm with SIMD gave use 5-10x speedup in multiple use cases
04:48:14 <anssik> ???: which resolution is used in these demos?
04:48:38 <anssik> ????: impressive, what is the laptop specifically the demos were run on?
04:48:51 <tidoust> -> https://www.w3.org/2019/Talks/TPAC/efficient-media-processing/ Introduction slides
04:48:53 <anssik> riju: 10 fps on modern phones
04:49:10 <riju> https://github.com/riju/WebCamera
04:49:27 <anssik> paul: so I'm from Mozilla, back in the days, Firefox project did not have access to native APIs
04:49:37 <riju> On your chrome, try out : https://riju.github.io/WebCamera/samples/
04:50:07 <anssik> ... how to do competitive real-time image processing on the web was looked into at Mozilla around 2012-14
04:50:56 <cabanier> cabanier has joined #mediaprocessing
04:50:57 <ningxin_hu> ningxin_hu has joined #mediaprocessing
04:50:59 <anssik> ... findings were CPU on mobile was working but not working well enough, so investigated GPU path and found gaps that we were able to fix with a proprietary APIs only
04:51:18 <anssik> ... running CPU clocked on screen refresh(?)
04:51:55 <anssik> ... processing model of stacked filters is the most important part
04:52:15 <anssik> ... not compositing, but stacking within the same group
04:52:31 <anssik> ... we stopped Firefox OS and now we have access to better technology
04:53:04 <anssik> ... more recently, we started a Web Codec effort with Google, separate breakout 16:30 today
04:53:09 <Gooroomee2> Gooroomee2 has joined #mediaprocessing
04:53:52 <anssik> ... would like to coordinate across these two efforts
04:54:33 <anssik> [missed peter's comment re Web Codec]
04:54:53 <plamb_mozilla> plamb_mozilla has joined #mediaprocessing
04:55:36 <anssik> dom: any other feedback from Audio work that could apply to video processing
04:56:21 <anssik> paul: your API must be programmable
04:56:39 <tidoust> s/[missed peter's comment re Web Codec]/Peter: Mapping between Streams and MediaStreamTrack is in the WebCodecs proposal, but it doesn't have to be there. Could be done elsewhere for use cases that need it but don't need WebCodecs/
04:56:45 <anssik> ... Web Audio started as a node composition data flow pairing
04:56:57 <anssik> ... we had lots of nodes, that became an issue
04:57:15 <plamb_mozilla> plamb_mozilla has left #mediaprocessing
04:57:34 <anssik> ... let's also do a model understanding images are large and we cannot afford to do redundant copies
04:57:55 <anssik> [harald showing slides]
04:58:17 <anssik> harald: trying to make life easier for people who use RTCPeerConnection
04:58:32 <anssik> ... if I want to solve one problem only, then this is important
04:59:09 <anssik> [harald showing requirements, scribe assumes slides will be made available, so won't scribe the slide content]
04:59:55 <anssik> harald: we have a solution that works, as was demonstrated: use media and canvas elements
05:00:11 <anssik> ... we should be able to do better, using those primitives browsers cannot optimize
05:00:28 <anssik> ... existing apps that want to add image processing step, you should not need to redesign everything
05:00:48 <anssik> ... we believe composable objects are the future on the web
05:01:01 <anssik> ... when we see pieces going to the right direction, we should harmonize
05:01:32 <anssik> [harald showing drawing of WebRTC Streams flow]
05:02:07 <anssik> harald: let's give web apps ability to insert processing between getUserMedia and PeerConnection blocks
05:02:35 <anssik> ... Web Codec proposal says, encoder is in Stream
05:03:29 <anssik> ... WHATWG Streams are a concept that can be piped
05:04:01 <anssik> ... this is exactly what we need, how can I put this onto the Web with PeerConnection
05:04:41 <anssik> ... let's do that at insert-time with a Factory pattern
05:05:53 <tpk> tpk has joined #mediaprocessing
05:05:53 <anssik> [harald explaining a code example with encoderFactory]
05:06:36 <anssik> harald: JS has _some_ properties done right
05:07:23 <anssik> ... processing step can be offloaded to a worker that can read/write stream
05:08:46 <anssik> ... the goal of the project is to say: given a Stream add this translation into it -- seems like magic?
05:09:28 <dom> q?
05:09:29 <anssik> ... the end
05:09:37 <Zakim> Zakim has joined #mediaprocessing
05:09:56 <anssik> chris: processing in media element, is there a similar solution for that, consider with MSE?
05:10:09 <anssik> harald: this only for media that arrives with PeerConnection
05:10:35 <Maru> Maru has joined #mediaprocessing
05:10:41 <anssik> ... similar type of thinking could be applied there, not exactly this solution
05:10:46 <Steve_Anton> Steve_Anton has joined #mediaprocessing
05:11:22 <anssik> [scribe asks tidoust to scribe his comment]
05:12:33 <anssik> ?????: instead of putting this in sinks, common denominator MediaSteamTracks(?)
05:13:00 <tidoust> s/[scribe asks tidoust to scribe his comment]/Francois: To reformulate, the proposal is to create a hook into PeerConnection to plug WebCodecs. A similar hook is needed in MSE and other APIs./
05:13:14 <anssik> ... audio worklets need their own context running off the main thread
05:13:14 <tidoust> s/?????:/Jan-Ivar:/
05:13:36 <anssik> harald: this should be part of the worker, to solve that architectural issue
05:14:05 <anssik> ... what does PeerConnection actually does with Workers? A bit complex
05:14:12 <tpk> tpk has left #mediaprocessing
05:14:38 <anssik> ... APIs might allow codec switching on the fly
05:15:32 <anssik> ??????: is this specific to WebRTC? Is MediaRecorder in scope of the proposal?
05:15:53 <anssik> harald: when working with PeerConnection, porting over to MediaRecorder et al. doable
05:16:08 <dom> s/??????/Youenn_Fablet_Apple/
05:16:40 <anssik> Youenn_Fablet_Apple: funny hats would probably happen pre-encoding
05:17:38 <anssik> ... WebRTC questions, between PeerConnection and codec, what happens on the network level, two params depending on network connection, how to deal with that?
05:18:12 <riju> Here are the slides for use-cases for Media processing on Web platform that I presented on ImageProcessing+WASM+ImageCapture : http://bit.ly/rijuMediaTPAC
05:18:24 <anssik> harald: PeerConnection framework would same the messages, if it needs to modify things it can
05:18:58 <anssik> ??????_Google: changing format after encoding?
05:19:19 <disooo> disooo has joined #mediaprocessing
05:19:21 <anssik> harald: we haven't defined the format yet, so can do anything
05:19:53 <Steve_Anton> s/??????_Google/Emad_Omara_Google/
05:20:53 <anssik> Jan-Ivar: a different API like VideoWorklet, would it be a better abstraction?
05:21:48 <anssik> Youenn_Fablet_Apple: Stream piped was a great model, similar to audio graph, any learning from that?
05:22:46 <anssik> paul: cloning an area of optimization, separable filters
05:22:49 <dom> RRSAgent, draft minutes
05:22:49 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html dom
05:22:53 <kiyoshi> kiyoshi has joined #mediaprocessing
05:23:08 <anssik> tidoust: where can we continue this discussion, 2 minutes left!
05:23:31 <anssik> ... Web Codecs are part of the solution, generic solution on different level, not discussed exposing GPU
05:24:06 <anssik> ... no real group now, WebRTC, Audio, Media groups available but split attention, do we want a dedicated CG for incubating this specific issue on the table?
05:24:46 <anssik> harald: we have a point problem and point solution, we have a field of problems that need a field of solutions
05:25:02 <anssik> tidoust: any suggestions on a venue?
05:25:21 <anssik> ... Media WG could be it, but not in scope, maybe M&E IG?
05:25:36 <anssik> chris: we could consider to reformulate the IG to be the place
05:26:02 <anssik> tidoust: most stakeholders in this room are in Media WG
05:26:03 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
05:26:08 <anssik> ... thank you all!
05:26:13 <dom> RRSAgent, draft minutes
05:26:13 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html dom
05:26:15 <anssik> ... thanks Anssi for scribing!!!
05:26:25 <Maru> Maru has left #mediaprocessing
05:26:35 <tidoust> RRSAgent, draft minutes
05:26:35 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/18-mediaprocessing-minutes.html tidoust
05:27:20 <NJ___> NJ___ has joined #mediaprocessing
05:27:53 <ericc> ericc has joined #mediaprocessing
05:30:55 <horiuchi> horiuchi has joined #mediaprocessing
05:31:19 <tidoust> tidoust has joined #mediaprocessing
05:31:53 <horiuchi> horiuchi has joined #mediaprocessing
05:39:12 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
05:39:52 <plamb_mozilla> plamb_mozilla has joined #mediaprocessing
05:41:13 <plamb_mozilla> plamb_mozilla has left #mediaprocessing
06:32:05 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
06:35:42 <horiuchi> horiuchi has joined #mediaprocessing
06:44:06 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
07:13:29 <horiuchi> horiuchi has joined #mediaprocessing
07:18:47 <tidoust> tidoust has joined #mediaprocessing
07:20:10 <horiuchi> horiuchi has joined #mediaprocessing
07:23:18 <dom> dom has joined #mediaprocessing
07:25:50 <Zakim> Zakim has left #mediaprocessing
07:26:44 <horiuchi_> horiuchi_ has joined #mediaprocessing
07:27:36 <ericc> ericc has joined #mediaprocessing
07:28:14 <dom> dom has left #mediaprocessing
07:31:19 <horiuchi> horiuchi has joined #mediaprocessing
07:31:52 <tidoust_> tidoust_ has joined #mediaprocessing
07:47:17 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
07:51:57 <tidoust_> tidoust_ has joined #mediaprocessing
07:59:56 <tidoust> tidoust has joined #mediaprocessing
08:23:31 <horiuchi> horiuchi has joined #mediaprocessing
08:31:24 <tidoust> tidoust has joined #mediaprocessing
08:33:22 <jib> jib has joined #mediaprocessing
08:34:44 <ericc> ericc has joined #mediaprocessing
08:37:41 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
08:38:14 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
09:33:21 <MasayaIk_> MasayaIk_ has joined #mediaprocessing
09:34:16 <horiuchi> horiuchi has joined #mediaprocessing
10:06:29 <MasayaIkeo> MasayaIkeo has joined #mediaprocessing
11:55:32 <tidoust> tidoust has joined #mediaprocessing