13:20:54 RRSAgent has joined #webmachinelearning 13:20:54 logging to https://www.w3.org/2018/10/26-webmachinelearning-irc 13:21:02 Zakim has joined #webmachinelearning 13:21:16 RRSAgent, make logs public 13:21:37 Meeting: Machine Learning for the Web CG F2F at TPAC 2018 13:21:44 Chair: Anssi 13:21:57 Agenda: https://github.com/webmachinelearning/meetings/blob/master/2018-10-26-lyon/README.md 13:23:14 Bryan has joined #webmachinelearning 13:23:58 Present+ Anssi_Kostiainen, Ningxin_Hu, Greg_Whitworth, Eric_Siow, David_Signer, Tomoyuki_Shimizu, Myles Maxfield, Bryan_Bernhart 13:24:16 ningxinhu has joined #webmachinelearning 13:24:21 RRSAgent, draft minutes v2 13:24:21 I have made the request to generate https://www.w3.org/2018/10/26-webmachinelearning-minutes.html anssik 13:24:43 the webml examples URL: https://huningxin.github.io/webml-examples/ 13:26:35 myles_ has joined #webmachinelearning 13:26:42 Greg Whitworth, Microsoft 13:26:46 scribenick: myles_ 13:26:58 RRSAgent, draft minutes v2 13:26:58 I have made the request to generate https://www.w3.org/2018/10/26-webmachinelearning-minutes.html anssik 13:27:07 Ningxin Hu, Intel 13:27:08 Myles C. Maxfield, Apple 13:27:13 takio has joined #webmachinelearning 13:27:53 present+ myles_ 13:28:00 Present+ Tatsuya_Igarashi 13:28:02 Present+ Takio_Yamaoka 13:28:24 Present+ gregwhitworth 13:28:45 anssik: it has been a long week. This is the last meeting possible at TPAC. It's exciting topic! 13:28:47 Present+ Barbaba_Hochgesang 13:28:54 anssik: let's start in a couple of minutes. 13:29:05 Present+ Yang_Gu 13:29:09 yang_gu has joined #webmachinelearning 13:29:16 RRSAgent, draft minutes v2 13:29:16 I have made the request to generate https://www.w3.org/2018/10/26-webmachinelearning-minutes.html anssik 13:29:31 Eric has joined #webmachinelearning 13:29:50 igarashi_ has joined #webmachinelearning 13:30:55 anssik: who was at the breakout session on Wednesday? 13:31:12 anssik: who was not there? 2 people 13:31:25 Topic: Welcome, Introductions 13:31:47 dsinger has joined #webmachinelearning 13:32:35 BarbaraH_ has joined #webmachinelearning 13:32:36 anssik: on Wednesday we had a breakout session. Most of you saw the demo. We saw a couple of slides about the findings based on teh Chromium implementation. To recap: the performance is pretty close to native, so we can implement the use cases on the web that require low latency. So you saw object recognition, human pose estimation, as examples of use cases that require low latency. and image classification. 13:32:55 anssik: we had discussion around the scope of the work briefly. The idea was to use more of this meeting to go into more detail. 13:33:08 anssik: ??? has prepare an API proof of concept proposal based on his implementation work 13:33:41 anssik: ??? has reviewed the existing platforms that provide native APIs for doing inference: Android, macOS, Windows, and we were going to review the API mapping table. 13:34:12 anssik: I would like to connect you guys with each other so we can continue work outside of this meeting. It's unreasonable to expect us to be at our best on Friday afternoon and make progress during these two hours. 13:34:16 anssik: Let's get started. 13:34:31 anssik: let's start with the review of the charter. 13:34:40 -> https://webmachinelearning.github.io/charter/ Charter 13:35:06 anssik: this is the charter. With a bunch of you and with the public, we reviewed over the course of the last month. 13:35:09 s/???/Ningxin/g 13:35:20 anssik: the goals section is there. We want to keep it seimple. The goal is to define a low level API for machine learning specifically for inference. 13:35:52 anssik: the constraint: Not going to define an API that doesn't work across major platforms. So we are fairly tightly scoped. We had some questions during the breakout sessiona bout that. We don't wnat to descriminate against any platforms about that. 13:36:15 anssik: Details: The Web API allows construcing neural network computational graph 13:36:25 anssik: Can compile a neural network to native form 13:36:32 anssik: and can accept input data from somewhere 13:36:43 anssik: from various sources (Array buffers, media streams, etc.) 13:37:02 anssik: We list teh platform constraints that we have. 13:37:26 anssik: Android Neural Networks API, Windows DIrectML, macOS/iOS Metal Performance Shaders and Basic Neural Network Subroutines 13:37:34 anssik: there are privacy implications. 13:37:43 anssik: we take that seriously, and we document them here. 13:38:37 anssik: in scope is inference, out of scope is training. This is because of practicalities. This is because platform's don't expose training facilities. Also we don't expose any hardware facilities. We are not interested in doing overlapping work, so we don't re-invent the wheel. 13:38:48 anssik: we coordinate with WebGPU, WebGL, and WebAssembly 13:39:04 anssik: Some people here are from those communities, that's great 13:39:18 anssik: out of scope: We don't attempt to mandate a model schema or format. 13:39:24 anssik: There are other groups that will do that. 13:39:32 wseltzer has joined #webmachinelearning 13:39:33 anssik: any questions? 13:39:35 present+ 13:39:51 anssik: Deliverable: Web Neural Network API 13:40:17 present+ 13:40:23 anssik: We also work with WebRTC for MediaStream for providing input for inference. Also audio. and devices and sensors. 13:40:34 anssik: Google proposed coordination with the immersive web working group and community group. 13:40:41 anssik: they want to use our work. e.g. object recognition 13:40:52 -> https://webmachinelearning.github.io/charter/ Charter 13:41:02 Present+ Michael_McCool 13:41:19 mmccol: Please add the Web of Things group. We are looking at virtual services. There might be a service that accepts an image and produces JSON. 13:41:36 anssik: Thanks. This charter is on GitHub, please open an issue. We need review to add a normative change. This doesn't sound normative. 13:41:44 anssik: we will integrate your proposal 13:42:15 barbarah: Is this only working groups or are you also working wiht the video interest groups? Video is importatn for machine learning. They may want collaboration 13:42:23 anssik: definitely. 13:42:23 anssik: this is informative. 13:42:36 anssik: the rest of the charter is less exciting. 13:42:48 anssik: We work on GitHub. 13:43:10 anssik: Consensus-based decisions. I'm the initial chair, but I'm happy to share the workload. Please get in touch if you want to help me chair 13:43:28 anssik: We start wtih the tight scope, but we have a plan to expand teh scope, but it requires a 30-day vote with 2/3 support. 13:43:45 anssik: If someone proposes expanding the scope, we want to make sure the community agrees with that. Any questions? 13:44:01 Michael_McCool: Are you deciding on deliverables? 13:44:03 anssik: yes. 13:44:11 Michael_McCool: It's a browser-based API? 13:44:12 anssik: yes. 13:44:41 Michael_McCool: W3C doesn't usually do server-side things. When I was thinking about edge computing, where machine learnign is applicable to edge computing. It's a reasonable thing to do there. 13:44:58 Aritamk has joined #webmachinelearning 13:44:59 dsr: If you could reach out to the node.js community, that would be good. 13:45:05 dsr: Please coordinate with the;m 13:45:06 -> https://www.w3.org/community/webmachinelearning/ CG home 13:45:09 Michael_McCool: Also test cases too 13:45:28 Michael_McCool: If we can move this in the server-side stuff, that's good 13:45:40 gregwhitworth: When you say server-side, you mean JS right? If so, then it's in-scope 13:45:48 dsr: It duplicates browser APIs 13:45:54 gregwhitworth: it doesn't actually say browser. 13:45:57 Michael_McCool: It shouldn't 13:46:39 Regarding the current POC API, it's JS, so it'spretty straightfoward in node.js. We also put the inputs and outputs. Today, Array buffers are standard, but in the futurre, we will do media streams 13:46:46 Michael_McCool: There are security concerns 13:47:05 anssik: It's good if this was implementable in node. But the browser is the primary target, and node is secondary. 13:47:38 For node, because it's different secuirty model, it can access native code, some vendors already provide a solution to expose their native API to node. 13:47:46 Michael_McCool: We're already seeing it for native APIs 13:48:08 s/For node/ningxinhu: For node/ 13:48:49 s/Regarding the current/ningxinhu: Regarding the current/ 13:48:49 Greg Whitworth, Microsoft 13:49:00 Present+ Jungkee_Song 13:49:22 helena has joined #webmachinelearning 13:49:34 present+ helena 13:50:09 Present+ Wendy_Seltzer 13:51:17 Present+ Helena_Rodriguez 13:51:39 Present+ Mark_Arita 13:51:52 Present+ Dave_Raggett 13:52:51 -> https://www.w3.org/Data/events/data-ws-2019/ W3C Graph Data Workshop 13:53:53 anssik: thank you. 13:53:58 anssik: we know who we are 13:54:01 TOPIC: Target demographic, use cases and requirements 13:54:23 anssik: In a discussion with gregwhitworth: What is the target audience. 13:54:41 anssik: We want to have a good design, so we need to understand the user. Use cases and requirements. Let's discuss it. 13:55:00 anssik: Expectation: It's a low-level API, users would be machine learning framework library authors. We want to understand their needs. 13:55:20 -> https://github.com/webmachinelearning/meetings/issues/1 WebML use cases by Tomoyuki/KDDI 13:55:21 anssik: ??? contributed a document as a startign point fo the discussion, which describes other APIs and use cases for the framework. 13:56:36 Tomoyuki: Previous cases for high level use cases. That could clarify what kind of applications can be built on top of WebML. These are application examples. first and second are strongly related to the demo. First: person detection. Recent image recognition can recognize what kind of objects are in a picture frame: human or otherwise. We can detect where the person is in the image. 13:56:46 Tomoyuki: Second: skeleton detection 13:56:55 Agenda: https://github.com/webmachinelearning/meetings/tree/master/2018-10-26-lyon 13:57:16 anssik: One observation: On the client, you can do depersonalized, without sending the data. Consider being at home with expensive objects, you dont' want to tell the world you ahve these expensive objects 13:57:19 s/???/Tomoyuki/ 13:57:34 ningxinhu: You also need per-pixel segmentation to know whic hpixels are okay and which aren't 13:57:46 anssik: this is a good example 13:58:03 ningxinhu: this is like photo uploading, but this doesn't require real time processing. 13:58:23 Tomoyuki: it requires depersonalization. This depersonalization needs to be done before uploading. 13:58:32 ningxinhu: So, you can get a preview of what will be uploaded. 13:58:41 ningxinhu: Removing hte background is useful for live streaming 13:58:54 Michael_McCool: If we want to do it for live streaming, the performance requirements become higher 13:59:02 dsr: Also blurring background 13:59:25 High level comment: These are application capabilities or features. I don't see them as use models. Is there anyindustry or type application that would utilize that 13:59:40 s/High/Barbara: High/ 13:59:47 Barbara: Would commerce applications use this? 14:00:12 anssik: It would cut across everything. It's like a JS library. There are libraries like TensorFlowjs that provide higher-level abstractions for web developers to enable implementing this use case easily. 14:00:23 ningxinhu: One is a social network. 14:00:23 gregwhitworth: ??? 14:00:33 ningxinhu: Social network website. 14:00:39 gregwhitworth: is there a specific social network? 14:00:49 anssik: it's a real problemthat people upload photos to social network without the content of others 14:01:30 gregwhitworth: I don't want to tangent too much, but I would love to know who would actually use this. Who is ready to consume this API. Which businesses are waiting fo rit. 14:02:12 aritamk: Google photos used to do image recognition, so you upload them to google drive, it does face detection across all your photos. Also, Amazon image recognition capabilities monitoring peopel that they sold to 14:02:26 Michael_McCool: Doing it on the client has advantage. 14:02:42 Which companies would this actually support their business models? 14:02:48 Barbara: Who would utilize this? 14:02:56 Barbara: Otherwise it's a science proejct 14:03:14 gregwhitworth: I'm working with our teams on this. They aren't chomping at the bit for this. I want to hear Facebook saying that they really want this. 14:03:30 Michael_McCool: If the governmetn says you can't share photos of people without their concent, suddenly everyone will want this 14:03:40 gregwhitworth: i don't want to spend my engineering resources if that isn't a reality 14:03:52 anssik: We need to have outreach to these companies so they can help influence the design 14:04:31 Michael_McCool: Instead of person detection, let's do video conferencing. Gesture detection: raising your hand in a meeting. Do you need to run multiple workloads? Do you need to have a queue, single accelerator, or multiple workloads? 14:04:57 Aritamk has joined #webmachinelearning 14:04:57 Michael_McCool. If you have a single accelerator, this becomes an issue, or you may need a queue manager. But here you need social media and video conferencing, rather than technical things. 14:05:09 anssik: This is Tomoyuki's initial contribution, which is great. 14:05:15 anssik: let's move this to GitHub. 14:05:25 anssik: maybe we can move this in the spec for "these are the problem we want to solve" 14:05:36 do we ahve any existing client side applciations that are using native? 14:05:52 Antialiasing is now shipping with ML trainined models. Hardware vendors would love to see the deployment. 14:06:06 Michael_McCool: Texture generation - running it backwards 14:06:13 gregwhitworth: who wants antialiasing? gaming? 14:06:13 yes 14:06:35 anssik: Re-emerging interest to investigate web-based gaming. There's a workshop. 14:07:07 anssik: We discussed skeleton detection. Video conferencing is an easy to understand valuable use case. Background removal, raise your hand, etc. 14:07:14 Michael_McCool: There are multiple workloads per application 14:07:24 ningxinhu: Communication with AI to do sign language recognition. 14:07:43 ningxinhu: Currently their solution is based on [inaudible] so we would liek to see if they could use this. 14:07:46 dsinger has joined #webmachinelearning 14:08:23 Skeleton detection is also useful to detect the form of surfaces. E.g. the fashion industry. Because when you have a scarf, they take a special position that help them to put ads around the fashion place 14:08:52 Michael_McCool: Image generation is a technical area, not a use case. We need to say something like "gaming" instead. 14:08:59 anssik: Maybe we can put this in the wiki and massage it. 14:09:14 anssik: low-level use cases. These are by definition more like capabilities. 14:09:54 Additional use cases options are retail/commerce, healthcare, marketing/advertising,fraud detection, online search, Natural Language Processing and audio. 14:10:27 Tomoyuki: Three examples: Sometimes neural network developer uses extension on their frameworks like TensorFlow. One exapmle: Custom layer. We sometimes face a kind of layer that is not permitted in the frameworks. So we often want how to extend such kind of layer to already permitted in existing framework. 14:10:31 anssik: comments? 14:10:32 Review the use cases on which ones have a web value. 14:11:44 ningxinhu: This is important. Faster development of Machine Learning community. New operators come year by year. The idea here is we propose some operators that are in scope, so those might be well-optimized to existing hardware or platform. That's for the existing support of operators. For new operators, we can coordinate with other web APIs like web assembly or web gpu compute shader to allow developers to implement custom layer in a 14:11:44 programmable way and connect the graph built by the WebML to WebGPU. 14:11:44 programmable way and connect the graph built by the WebML to WebGPU. 14:11:52 q+ 14:12:00 RRSAgent, draft minutes v2 14:12:00 I have made the request to generate https://www.w3.org/2018/10/26-webmachinelearning-minutes.html anssik 14:12:23 ningxinhu: Cross hardware boundary issues exist. For our implementation we saw this kind of combination in native code. E.G. Apple's Metal Performance Shader combines with Metal on teh same GPU. 14:12:36 Present+ Sangwhan_Moon 14:12:44 ningxinhu: We saw some it there. But on teh web we know we need to expose this capability as well as expose performance cliffs. 14:12:53 t_homma has joined #webmachinelearning 14:13:31 dsinger has left #webmachinelearning 14:13:33 Michael_McCool: This drives toward an architecture decision. Is there a compilation tool? Is there a Machine independent compiled model? Or do you specify it on the fly in the API? If you want performance, you probably want precompiled model. There probably should be a statement saying you want to maintina performance 14:13:42 Precompiled will be tricky to maintain performance 14:13:54 Michael_McCool: You want custom code to not blow up performance. 14:14:04 Michael_McCool: If you're going to lose performance a lot on some platforms, we need to know early 14:14:34 ningxinhu: simd.js is an example. We want to let the developer to test whether or not there is native support. 14:14:39 Bryan has joined #webmachinelearning 14:14:46 Feature detection? should we do it? 14:14:56 Michael_McCool: Compiling a language is a good strategy. 14:15:08 Which use cases and API are the target for short term versus long term based on developers needs. That will help drive the MVP with an enhancement roadmap. 14:15:14 Sean: Out of the devices, which are programmable? 14:15:20 ningxinhu: API should be hardware agnostic. 14:15:33 VPU, FPGA? 14:16:04 Michael_McCool: There are 2 ways: 1) A full compile from scratch, or use an FPGA 14:16:13 Michael_McCool: ASICs 14:16:29 IF you have discovery in a low-level API, how do you push that down to device. 14:16:56 Sean: You could throw in a capability "is this programmable or not" 14:17:12 ningxinhu: for GPU case, WebGPU has a shader language. 14:17:25 anssik: Network concatenation 14:18:09 Tomoyuki: Some recent network model of deep neural network uses module concatenation. For example, many network modules like MobileNet, ResNet, etc. Most neural network developers have insufficient time for training. We can reduce the training time by importing pre-trained models. 14:18:16 Tomoyuki: That gives us image feature extraction. 14:18:26 Tomoyuki: The following layer can be trained according to our use cases. 14:18:40 ningxinhu: is it really for training or for inference? 14:18:59 Tomoyuki: Developing process is just training but the result of the trained model. Trained modules and custom modules [inaudible] 14:19:16 Media use case potential - how do you find the right content to serve your audience quickly? The Media & Entertainment IG would have more insights. 14:19:17 ningxinhu: After training, when you want to deploy, is there a concatenation layer? Or is it already solved when training? 14:19:27 ningxinhu: Network concatenation is during training or inference? 14:19:30 Tomoyuki: Both. 14:19:53 Tomoyuki: In inference phase, we can either prepare the concacenated model or two pretrained modesl separetely. So the developercan select eithe ro fthem 14:20:01 ningxinhu: So it's possible in inference 14:20:15 Tomoyuki: We can also change the models. A large pretrained model or a small pretained model for low bitrate networks. 14:20:34 myles_: will there be built-in models? 14:21:03 anssik: Shape detection API can use built-in models. They're not in scope. But It's an itneresting idea. We haven't explored it. 14:22:12 gregwhitworth: Usually use cases revolve around 7 or 8. I've never heard of anyone asking for antialiasing. This is one of the major hurdles in Edge. We talked to Office, they are hundreds of megs big. What is the benefit of the end user. Id' like to narrow down to specific modesl and specifc use cases for those modesl. Faces, bar codes, etc. Let's have a small set of pre-canned models. 14:22:23 gregwhitworth: i don't want long load times. 14:22:23 anssik: There's no conflict there. 14:22:34 RESOLVED: We should look into pre-canned models. 14:22:52 sean: Google proposed "layered APIs" with known names. So that could work 14:23:00 sean: "std::facedetection" 14:23:11 s/sean/sangwhan/ 14:23:21 sangwhan: Who owns the models? Would loading your own model be secure? 14:23:32 s/sangwhan: Who/Who/ 14:23:37 sangwhan: W3C, presumably 14:23:54 ReinaldoFerraz has joined #webmachinelearning 14:24:23 Michael_McCool: Pretrained models allow hardware accelerators. Caching is useful. These names won't change, that helps caching. We will have cross-site problems. 14:24:32 gregwhitworth: Caching is by-origin anyways, even if they have the same name. 14:24:46 What's the type of the file? 14:24:49 sangwhan: undefined. 14:24:59 gregwhitworth: We are purposely putting that off until after this. 14:25:53 ningxinhu: Experience in POC focuses on mobile, small models. We try to cache the model. Service worker. Javascript TensorFlow.js introduces "web-friendly" model format, which is JSON-based topology description, and small-sized files. To try to fit into the cahce 14:26:09 gregwhitworth: The ones i'm worried about thigns like spell checking 14:26:57 gregwhitworth: Because of blind training, Are you referring to not do full-training, but slight pivots a little bit? Because not having access to [inaudible] makes it difficult to do it. Am I not able to say "I want the pre-canned one but I will provide 400k with different weights?" 14:27:09 ningxinhu: Like loading a delta on top of a model? 14:27:14 gregwhitworth: yes 14:27:37 ningxinhu: This is for transferrable training. You might need the on-device model to do that. 14:27:40 gregwhitworth: We want to do it. 14:28:06 sangwhan: For fine-tuning, you will need to expose gradient propagation, because that's also a propagation. 14:28:32 Michael_McCool. In some cases you might want flame points, other places you might want 8-bit, it's a tradeoff between size and quality. So each model should have multiple versions. 14:28:53 ningxinhu: Regarding these capabilities. In the POC, some native APIs support it, but not all. 14:29:25 gregwhitworth: I don't want to over-index on what native APIs do and don't support. I'd rather say "here are the use cases, and V0 is just some demos" and then native implementations can go further. 14:29:45 Michael_McCool: Some devices can only support certain quantizations. 8-bit or 32-bit. There may be a constraint on what is supported on each device. 14:29:54 sangwhan: This is a discussion bout formats, not about weights 14:30:04 Michael_McCool: It directly affects the size of the file. 14:30:14 gregwhitworth: It would be valuable to have that even if it's a V2 thing. 14:30:32 yang_gu has joined #webmachinelearning 14:31:13 Tomoyuki: Some devices can do GPU acceleration. Others cannot support GPU acceleration, only runs on CPU. Some web developer is taking care of battery consumption, so they offer image recognition only for accelerated devices. 14:31:58 Michael_McCool: There's a whole other category of what do applications that use multiple models do. For a computer, you can do CPU and GPU at the same time. You can support two use cases at the same time. There's another use case of multi-model. 14:32:08 gregwhitworth: Performance is super important 14:32:26 CPU and GPU may already be super busy. Developers need control over low-level device usage 14:32:38 Michael_McCool: Support for multiple workflows is important. 14:32:48 sangwhan: Most applications use CUDA BLAS 14:33:24 Michael_McCool: There's a CUDA vs graphics issue. We already have APIs have been extended recently to support multiple applications, when more than one device can handle the workloads. We can argue whether that is necessary or not. 14:34:03 ningxinhu: There are multiple hardware usecases. We have to select the hardware. CPU, GPU, MPU. We allow the developer to specify the preference of where they want it to run. 14:34:06 s/CUDA BLAS/CUDA, which locks - so multi-task is probably hard/ 14:34:33 ningxinhu: GPU and CPU both have pros and cons, and the site can serve different models depending on where the model will be run on. 14:34:52 anssik: Does the current scope capture the needs of the target audience? 14:35:10 anssik: Is this the tightest possible scope that we can start with? 14:35:35 Michael_McCool: Is it possible to precompile the model? 14:36:02 gregwhitworth: POC is a POC, and the answer is not yet defined. It's desirable for web developers, but you want both. 14:36:08 gregwhitworth: There will be divergent opinions. 14:36:19 t_homma has joined #webmachinelearning 14:36:35 Barbara: If you look at the use cases, which ones fit into the MVP vs an extension? Don't try to boil the ocean. MVP please. 14:37:08 Michael_McCool: We could decide this now. We could choose either way 14:37:14 gregwhitworth: It shouldn't be in the charter. 14:37:21 anssik: The group decides what is MVP out of this scope. 14:37:27 anssik: We can add more bits 14:37:36 anssik: So it sounds like we can start with this scope. 14:37:53 TOPIC: Review mapping to platform APIs 14:38:05 anssik: [inaudible] has published a mapping table. 14:38:09 -> https://github.com/intel/webml-polyfill/blob/master/docs/native_mapping.md Mapping to platform APIs 14:38:31 s/[inaudible]/Ningxin/ 14:39:27 ningxinhu: Before we go through the table, this is an overview. Convolution, depthwise convolution, concatenation. ADD and MUL, RESHAPE, SOFTMAX. We will support them by fuse into convolution or others. Of the9 operators, we have some support for models in teh POC. 14:40:46 ningxinhu: Examples: TFLite models exist. Our example grabs these models, load and parse it, and use it as a web API to construct the graph and run inference. SqueezeNet because it's small-sized. Also we tried large model like Inception V3. Except its size is bit but we can get a good speedup for the computation. For the object detection, we have SSD MobileNet. Also we have TensorFlow.js model like MobileNet and PoseNet. Also ONNX models. 14:40:59 ningxinhu: MobileNet V2 and SqueezeNet. 14:41:38 ningxinhu: Only implemented 9 operators so far, so we tried tehse models and underlying the implementation these operations on native APIs we have implementation supporting ops for MPS, BNNS on macOS, NNAPI on android, and clDDN on Linux and Window. 14:41:45 ningxinhu: clDDN is from Intel. 14:41:53 ningxinhu: We would liek DirectML POC very soon. 14:42:00 ningxinhu: that's the overview. 14:42:41 ningxinhu: It's driven from the models. There are different ecosystem's models. We start with some small models optimized for mobile, then add necessary ops to support these models. And implement them across different APIs to get performance data. 14:43:33 ningxinhu: We have data! To map the data type Float32 and Float16 is what we have looked at so far. NNAPI doesn't support Float16, everybody else supports both. 14:44:31 ningxinhu: For convolution we have how the operator can be mapped to different API. Input, filter (aka "weights"), and bias. There are some differences between the APIs, the notes are in this chart in red. 14:44:48 ningxinhu: Also for stride and fused activation, dilation rate, and output 14:45:01 ningxinhu: This is just one case for convolution. For the other 9 ops, we have the same kind of data in the table. 14:45:07 s/liek/like/ 14:45:37 anssik: We made this to satisfy this requirement that the API is implementable on top of platform APIs. We did the work so you don't have to. 14:45:44 anssik: if you spot issues, please let us know. 14:46:24 anssik: We dont' have a formal startin gpoint spec. Instead, we have POCs and sketch APIs. 14:46:28 TOPIC: Review & discuss Web Neural Network API spec proposals 14:46:36 anssik: ningxinhu has a API sketch proposal. 14:47:03 anssik: We have some ergonomic issues in this sketch proposal that we already know aboutl 14:47:28 -> https://github.com/intel/webml-polyfill/blob/master/docs/api.md WebML API proof-of-concept 14:47:30 ningxinhu: Here's the example: Construct (tensor0 + tensor1) * (tensor2 + tensor3). 0 and 2 are constants 1 and 3 are inputs. 14:48:22 RRSAgent, draft minutes v2 14:48:22 I have made the request to generate https://www.w3.org/2018/10/26-webmachinelearning-minutes.html anssik 14:48:38 ningxinhu: ::describes the chart in the doc:: 14:49:02 ningxinhu: So first you need a neural network context. It's inside navigator.ml.getNeuralNetworkContext() 14:49:30 ningxinhu: to build the model, you can see the details on the site. Promises are involved. 14:49:42 ningxinhu: Then you speicfy the tensor type 14:50:23 ningxinhu: The API is typed, so you have to annotate types along with the data 14:50:42 ningxinhu: You can set operands like the upper value in the graph to describe the shape of the tensor. Later you can upload the data to the graph. 14:50:50 ningxinhu: We use array buffer view to do the upload 14:50:58 ningxinhu: So it's just a scalar in this example. 14:51:24 ningxinhu: Tensor 1 is an input, so there's no data to upload. Then same for tensors 2 and 3 14:51:36 ningxinhu: Then you need to connect the operators together to define the graph. 14:52:07 ningxinhu: Then you add the operations, you specify the flow of computation graph. ::works through the example:: 14:53:41 myles_: Why sin't htis a programming langue? 14:54:22 tidoust has joined #webmachinelearning 14:54:38 ningxinhu: Someone has invented a languge to do it, in python. This is supposed to define a common model for hardware dispatch. In this model we just define the computation graph (DAG) Then you compile it to different hardware. API allows you to define the workload 14:54:47 Present+ Francois_Daoust 14:54:56 Michael_McCool: Another option is to use a string instead of the graph. 14:55:54 RRSAgent, draft minutes v2 14:55:54 I have made the request to generate https://www.w3.org/2018/10/26-webmachinelearning-minutes.html anssik 14:56:26 Michael_McCool: Dynamic programming allows you to glue graphs together. Having a whole langauge is an option but it has it's own problems. OpenCL uses a string for the string, and therefore views the program as data so you can send it around. The graphs can be serialized to a blob 14:56:41 Michael_McCool: The fact that it's close to TensorFlow is valuable. 14:56:47 ningxinhu: It's close to native APIs 14:57:23 Michael_McCool: If you have an object that represents the graph, you can export and import operations 14:57:48 sangwhan: Why do we want export API if we dont have a filesystem API? 14:57:54 sangwhan: each model will have a URL anyway 14:58:12 Michael_McCool: The serialized blob would be opaque. 14:58:54 Michael_McCool: You wouldn't have to define what's in the blob as long as there interoperability 14:59:38 Michael_McCool: Okay, maybe that's not true 14:59:53 ningxinhu: You can also specify a device selection preference 15:01:08 ningxinhu: Execution model, it's liek a tight loop. After you compile your code, you can upload your input data using an array buffer view. Lines like "execution.setInput(0, thingy); execution.setInput(1, thingy)" 15:01:19 ningxinhu: Your output can be an input for post-processing for WebAssembly code 15:01:40 Michael_McCool: If you wanted to support multiple devices, wouldn't the compilatino need to know which device it will be run on? 15:01:42 ningxinhu: yes 15:02:03 ningxinhu: The last step: Do the work, returns a promise 15:02:11 ningxinhu: "execution.startCompute()" 15:02:35 ningxinhu: so that's it 15:02:47 Michael_McCool: You could do multiple computations in parallel using Promise.all() 15:02:51 ningxinhu: yes 15:03:57 [not willing to bikeshed on the method names for now, but I would expect a Promise to a method called "startCompute" to resolve when the computation has started, not when it's over. In other words, I'd simply call the method "compute" if the results of the computation are available when the promise resolves] 15:04:04 ningxinhu: Here, the output is an Array Buffer view, so you have to be careful with validation. You have to map you rinput data to the output data. Ideally the promise resolution will incorporate this, but that would create a new object which is an issue. For the POC, we followed this simple design, but we're flexible. 15:04:51 1. Doesn't the current model suggest double memory allocation? 15:04:55 2. What is "fast single answer"? Is this going to be defined in the specification later on? 15:04:58 3. Choice of integer constants over string enums. Sad panda face. 15:05:01 4. Choice of procedural programming practices for model definition seems inconvenient, especially given that this is pretty much for inference only. 15:05:04 5. Output layer being re-used isn't nice. Neither does it follow idiomatic javascript practices. 15:05:07 6. Plans to define/make consistent poorly defined ops? e.g. "EMBEDDING_LOOKUP" 15:05:11 7. Constructors for cases where it makes sense? (for initializations that probably won't reject) 15:06:14 sangwhan: When you allocate float32 arrays, they get copied to the GPU, right? Double allocation! Shouldn't those be other types? 15:06:52 ningxinhu: it's not an allocation here. Here you're just describing the graph. When you compile the code's model, it will then allocate device memory. 15:07:59 ningxinhu: Embedding lookup is poorly defined. Will we define this? 15:09:12 ningxinhu: Our POC just calls into android calls. We didn't define anything. 15:09:28 s/ningxinhu: Embedding/sangwhan: Embedding/ 15:10:49 myles_: so it isn't interoperable? 15:10:56 gregwhitworth: we will define it in an interoperable way. 15:11:24 sangwhan: It's a lot of code. If we're just loading from a URL, shouldn't have to write that much code. 15:11:35 ningxinhu: We want to follow the extensible web manifesto 15:11:47 to extend on what I meant - this is a rough API shape, it's not a spec - it's a POC so they want to show what they did - but in no way is this actually defined. 15:12:56 myles_: Is this the basis for a future API, or is this just something to show that it's possible on the web? 15:13:05 anssik: just something to show that it's possible on the web