IRC log of webmachinelearning on 2023-08-24

Timestamps are in UTC.

13:57:10 [RRSAgent]
RRSAgent has joined #webmachinelearning
13:57:14 [RRSAgent]
logging to https://www.w3.org/2023/08/24-webmachinelearning-irc
13:57:14 [Zakim]
RRSAgent, make logs Public
13:57:45 [Zakim]
please title this meeting ("meeting: ..."), anssik
13:57:45 [anssik]
Meeting: WebML WG Teleconference – 24 August 2023
13:57:45 [anssik]
Chair: Anssi
13:57:45 [anssik]
Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2023-08-24-wg-agenda.md
13:57:53 [anssik]
Scribe: Anssi
13:57:53 [anssik]
scribeNick: anssik
13:57:56 [anssik]
gb, this is webmachinelearning/webnn
13:57:57 [gb]
anssik, OK.
13:57:57 [jsbell]
jsbell has joined #webmachinelearning
13:57:58 [Vivek]
Vivek has joined #webmachinelearning
13:58:02 [anssik]
Present+ Anssi_Kostiainen
13:58:13 [anssik]
Present+ Rachel_Yager
13:58:22 [anssik]
Present+ Vivek_Sekhar
13:58:27 [anssik]
Present+ Zoltan_Kis
13:58:38 [anssik]
RRSAgent, draft minutes
13:58:40 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/08/24-webmachinelearning-minutes.html anssik
13:58:53 [jsbell]
Present+ Joshua_Bell
14:00:03 [RafaelCintron]
RafaelCintron has joined #webmachinelearning
14:00:18 [anssik]
Present+ Rafael_Cintron
14:00:23 [Ningxin_Hu]
Ningxin_Hu has joined #webmachinelearning
14:00:41 [anssik]
Present+ Joshua_Lochner
14:00:54 [anssik]
Present+ Ningxin_Hu
14:01:12 [Rachel]
Rachel has joined #webmachinelearning
14:02:33 [anssik]
Regrets+ Chai_Chaoweeraprasit
14:03:02 [Joshua_Lochner]
Joshua_Lochner has joined #webmachinelearning
14:03:31 [anssik]
RRSAgent, draft minutes
14:03:32 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/08/24-webmachinelearning-minutes.html anssik
14:04:18 [anssik]
anssik: Welcome to our 24 Aug call, we have a busy and an exciting agenda!
14:04:32 [anssik]
Topic: Google Chrome team's feedback on WebNN API
14:04:56 [anssik]
anssik: I asked Vivek and Joshua to share a high-level summary of the Chrome team's feedback with the WG and document that feedback in a GH issue. Thank you Vivek and Joshua for collecting all this feedback from Google Chrome teams and sharing it with the WG, this is much appreciated.
14:05:00 [anssik]
... you'll find Google Chrome feedback in GH issue #453
14:05:01 [gb]
https://github.com/webmachinelearning/webnn/issues/453 -> Issue 453 Google Chrome Feedback on WebNN: aiming for broad device coverage and maintainability (by vsekhar)
14:05:25 [anssik]
... Vivek and Joshua, please feel free to use 10-20 minutes incl. discussion to share a high-level summary of your feedback. I expect the WG to continue discuss specific topics in the GH issue and spin new issues as appropriate.
14:05:56 [anssik]
Vivek: thank you Anssi and the WG
14:06:06 [anssik]
Vivek: Google strongly supports the work of the WebML WG
14:06:20 [anssik]
... we got together at Google to gather feedback
14:06:46 [anssik]
... feedback solicited from ML research and infrastructure teams at Google
14:07:14 [anssik]
... reach and maintainability are important lenses for us
14:07:32 [anssik]
... Chrome's key observation:
14:07:40 [anssik]
... - for new OS APIs or hardware accelerators, we must assume that most Web users don't have them
14:08:11 [anssik]
... - we have an obligation to ensure a workable experience for other users as well
14:08:36 [anssik]
... Chrome's goal:
14:08:40 [anssik]
... - achieve 80% of a device's theoretical hardware-accelerated ML runtime performance across 80% of devices on the Web, and to do so while imposing a manageable long-term support burden on browser vendors
14:08:47 [anssik]
... Ecosystem issue:
14:08:52 [anssik]
... - the ML ecosystem is still rapidly evolving, making it difficult for any API to keep up
14:09:36 [anssik]
... Proposed steps for the WG (from the issue):
14:09:39 [anssik]
... 1. Request public positions from major browser implementers
14:09:43 [anssik]
... 2. Reduce the long term support burden of WebNN by streamlining the API surface
14:09:48 [anssik]
... 3. Demonstrate WebNN performance for CPU and GPU execution across multiple OS platforms
14:09:54 [anssik]
... 4. Demonstrate WebNN performance gains utilizing OS- and hardware-specific optimizations
14:11:39 [anssik]
... Proposed steps for OS- and hardware-specific optimizations:
14:11:42 [anssik]
... 1. Select 2-5 demonstrative ML models
14:11:45 [anssik]
... 2. Run on a demonstrative set of platforms with accelerator hardware
14:11:49 [anssik]
... 3. Evaluate latency, throughput and power efficiency between lowering to CPU/GPU vs. hardware accelerators
14:11:55 [jsbell]
q+
14:11:57 [anssik]
anssik: thanks Vivek and Joshua and the Google Chrome team for this feedback!
14:12:01 [anssik]
ack jsbell
14:13:12 [anssik]
jsbell: thanks for the summary! Just wanted to share this derives from thinking what we as Google Chrome have as requirements for shipping an API, the same criteria as we have for Intent to Ship for any feature
14:13:26 [anssik]
... we stand behind what we ship for decades and these consideration are based on that expectation
14:13:58 [anssik]
q?
14:16:02 [Vivek]
q+
14:16:07 [anssik]
ack Vivek
14:16:53 [anssik]
Vivek: want to note the group has thought about the low-level and high-level ops question, appreciate that
14:17:54 [anssik]
... reducing long-term support burden, if there's consensus emerging in the broader ML space we propose to align with that on op set abstraction level and scope
14:18:45 [anssik]
q?
14:19:00 [anssik]
Ningxin_Hu: thanks for this concrete feedback, a lot of good observations
14:19:13 [anssik]
... I like that you've shared concrete guidance and recommendations
14:19:38 [anssik]
... re "Demonstrate WebNN performance for CPU and GPU execution across multiple OS platforms"
14:19:59 [anssik]
... suggestion is to implement WebNN as a polyfill on top of Wasm and WebGPU APIs
14:20:19 [anssik]
... we have a JS implementation of WebNN using TF.js kernels with Wasm and WebGPU backends
14:20:38 [anssik]
https://github.com/webmachinelearning/webnn-polyfill
14:20:55 [jsbell]
q+
14:20:55 [anssik]
Ningxin_Hu: can you elaborate this proposal?
14:20:56 [Vivek]
q+
14:20:57 [anssik]
ack jsbell
14:21:03 [Vivek]
q-
14:21:37 [anssik]
jsbell: clarifying that we are not proposing that browsers ship WebNN as a polyfill, but that the CG created polyfill would be excellent
14:22:16 [anssik]
... launch process adoption question, it is adopted by developers, we want to avoid a situation that there is no polyfill and web developers code directly to CPU or GPU backend
14:22:41 [Vivek]
q+
14:22:45 [anssik]
... if there's a quality polyfill framework authors can move to a polyfill and when browser implementations roll out we'd see immediate performance boost
14:22:59 [anssik]
... this is called ecosystem activation
14:23:01 [anssik]
ack Vivek
14:23:26 [anssik]
Vivek: a polyfill helps clarify what is needed from the platform, running workloads on a polyfill will help trace where the performance bottlenecks are
14:23:41 [anssik]
... e.g. if the polyfill is too large, we can see where the web platform can help
14:23:50 [anssik]
... some features may remain in the user space
14:24:17 [anssik]
... because they change so fast, so polyfill helps clarify those aspects and see what developers should be able to customize
14:24:18 [anssik]
q?
14:24:45 [anssik]
anssik: are we maintaining the WebNN polyfill?
14:25:03 [anssik]
Ningxin_Hu: probably not up to date with the very latest spec version
14:25:14 [anssik]
... an opportunity to improve the WebNN polyfill
14:26:26 [anssik]
... the polyfill builds on TF.js, so thanks for the TF.js team
14:26:49 [anssik]
q?
14:27:03 [Joshua_Lochner]
q+
14:27:09 [anssik]
ack Joshua_Lochner
14:28:18 [jsbell]
q+
14:28:21 [anssik]
Joshua_Lochner: wanted to ask, re caching side of things, when you save models you don't need to redownload the models, as Transformers.js author an issue close to my heart
14:28:30 [anssik]
... is this a consideration here?
14:28:40 [AramZS]
AramZS has joined #webmachinelearning
14:28:42 [anssik]
ack jsbell
14:29:20 [anssik]
jsbell: I definitely acknowledge your concern, that is part of the broader ecosystem adoption consideration
14:29:35 [anssik]
... we have folks in the Chrome team for improvements to storage
14:29:48 [anssik]
... will be discussed outside this call, in another issue or a future call
14:30:05 [anssik]
RRSAgent, draft minutes
14:30:06 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/08/24-webmachinelearning-minutes.html anssik
14:30:12 [anssik]
q?
14:30:26 [anssik]
Topic: WebNN v2: review proposed new ops and data types
14:30:47 [anssik]
Present+ Dwayne_Robinson
14:31:13 [anssik]
anssik: I'd like us to review and discuss the proposed new ops and data types informed by v2 model targets and recent prototyping efforts.
14:31:25 [anssik]
... Dwayne posted a well formulated list of proposals into a GH issue #375 -- thank you!
14:31:28 [gb]
https://github.com/webmachinelearning/webnn/issues/375 -> Issue 375 Support for transformers (by dontcallmedom) [v2] [operation set]
14:31:30 [anssik]
... Details in https://github.com/webmachinelearning/webnn/issues/375#issuecomment-1674224992
14:31:40 [anssik]
... let me first summarize what was proposed and then let Dwayne fill me in
14:31:48 [anssik]
... Proposed models:
14:31:58 [anssik]
... - Text-to-image: Stable Diffusion unet/VAE/text encoder
14:32:09 [anssik]
... - Image segmentation: Segment Everything decoder
14:32:20 [anssik]
... - Speech-to-text: Whisper Tiny
14:32:33 [anssik]
anssik: We don't have text-to-text models proposed, so I'd like the WG to discuss if some would be applicable, examples:
14:32:53 [anssik]
... Text-to-text: Summarization, translation, code completion demonstrated by Transformers.js?
14:32:57 [Joshua_Lochner]
q+
14:33:00 [dwayner]
dwayner has joined #webmachinelearning
14:33:02 [anssik]
ack Joshua_Lochner
14:33:27 [anssik]
Joshua_Lochner: text-to-text, I'll add some additional material
14:34:06 [anssik]
Joshua shared:
14:34:06 [anssik]
Text-to-text:
14:34:06 [anssik]
- Summarization: https://xenova.github.io/transformers.js/?demo=summarization
14:34:06 [anssik]
- Code completion: https://huggingface.co/spaces/Xenova/ai-code-playground
14:34:06 [anssik]
- Translation: https://huggingface.co/spaces/Xenova/react-translator
14:34:23 [anssik]
Joshua_Lochner: application-level tasks that are helpful in web apps
14:34:37 [anssik]
... code completion would be helpful e.g. in GH or codespaces
14:36:28 [anssik]
... this playground uses 300M params, no GPU, Wasm backend, reasonable performance already as is
14:36:32 [anssik]
... privacy focused browser extension could make use of this, for example
14:36:32 [anssik]
... star coder in collaboration with HuggingFace
14:36:38 [anssik]
... I think translation is a great idea, but the concern is the model is huge, 600M parameter model, 1.3GB size
14:37:01 [Joshua_Lochner]
will do!
14:37:10 [anssik]
q?
14:37:23 [anssik]
... Proposed new ops:
14:37:37 [anssik]
- Logical elementwise comparison/selection operations: equal, greater, lesser, logicalNot, elementwiseIf/ternary, greaterOrEqual/lesserOrEqual
14:37:38 [anssik]
- More elementwise unary operations: identity, sqrt, erf (Gauss err func), reciprocal
14:37:42 [anssik]
- Reshaping operations: squeeze, unsqueeze, flattenTo2d
14:37:46 [anssik]
- Data rearrangement operations: expand, gather
14:37:52 [anssik]
- Normalization operations: meanVarianceNormalization
14:37:56 [anssik]
- Index seeking operations: argMin/argMax
14:38:00 [anssik]
- Misc: cast, fillSequence, triangularMatrix, shape
14:38:04 [anssik]
- Others?
14:38:08 [anssik]
... Proposed new data types:
14:38:13 [anssik]
... - int64
14:38:17 [anssik]
... - uint64
14:38:27 [anssik]
... Relevant background material:
14:38:27 [anssik]
-> Transformers.js presentation by Joshua Lochner https://lists.w3.org/Archives/Public/www-archive/2023Jun/att-0000/Transformers_js.pdf
14:38:27 [anssik]
-> Transformer models presentation by Dwayne Robinson https://lists.w3.org/Archives/Public/www-archive/2023Jun/att-0005/2023-06-29_WebNN_and_Transformers_Progress_W3C.pdf
14:38:43 [anssik]
anssik: Dwayne, thanks for this proposal! Please feel free to share with the WG your questions and areas of focus.
14:38:50 [anssik]
RRSAgent, draft minutes
14:39:21 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/08/24-webmachinelearning-minutes.html anssik
14:39:59 [anssik]
Dwayne: I'd appreciate feedback from anyone else who has goals for models that are missing ops, or feedback on whether we should not have some of these ops because they can be satisfied by low-level ops
14:39:59 [anssik]
... or feedback on naming
14:39:59 [anssik]
... these ops enable the models we focus on, but happy to expand to other valuable models, e.g. the text-to-text models Joshua proposed
14:40:25 [Vivek]
q+
14:40:33 [anssik]
q?
14:41:33 [anssik]
ack Vivek
14:42:04 [anssik]
Vivek: I wanted to understand the motivations for data types, in our work with WebGPU we have been using less data types and plumbing it
14:42:04 [anssik]
... floating point usually used in training use cases
14:42:24 [anssik]
dwayner: larger data types go past 4GB barrier, used by e.g. ONNX
14:43:20 [Rachel]
q+
14:43:26 [anssik]
ack Rachel
14:45:00 [anssik]
Rachel: Joshua_Lochner shared translation task, is it using WebNN API?
14:45:12 [anssik]
Joshua_Lochner: uses ONNX Runtime Wasm backend currently
14:45:54 [anssik]
... I convert pretrained models to ONNX and use ORT backend to do inference, tokenization etc. other steps done in JS, everything is running in the browser
14:46:28 [Joshua_Lochner]
Source code! https://github.com/xenova/transformers.js
14:47:21 [anssik]
anssik: you can validate it is on-device inference by disconnecting from internet, it still works
14:47:49 [Joshua_Lochner]
Yes it uses this model: https://huggingface.co/Xenova/nllb-200-distilled-600M
14:48:30 [Joshua_Lochner]
which itself is an ONNX export of https://huggingface.co/facebook/nllb-200-distilled-600M
14:48:52 [Joshua_Lochner]
yes exactly, the performance is limited by the model itself (and this model is quite old!)
14:48:57 [anssik]
RRSAgent, draft minutes
14:48:58 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/08/24-webmachinelearning-minutes.html anssik
14:49:04 [anssik]
Topic: WebIDL and Infra standard conventions
14:49:12 [anssik]
anssik: first, thank you Zoltan for keeping this big PR updated with comments and everyone for your review, this has been a great team effort across orgs!
14:49:16 [anssik]
... these changes align the entire specification with modern specification conventions and add stylistic improvements on top that make navigating this specification more delightful experience.
14:49:25 [anssik]
... today I'd like us to make a decision whether we're ready to merge the zk-conventions-integration integration branch to main.
14:49:30 [anssik]
... the big PR is #446
14:49:30 [gb]
https://github.com/webmachinelearning/webnn/issues/446 -> Pull Request 446 Add missing algorithms, add stylistic improvements, update with spec conventions (by zolkis)
14:49:52 [anssik]
... first I'd ask Zoltan to summarize the big PR #446 latest status and then Joshua Bell would like to spend a couple minutes discussing some of the motivations behind modern spec style, and the "processing model" (i.e. how JS types pass through Web IDL to become Infra types) etc. A part of Joshua's feedback is captured in issue #450
14:49:54 [gb]
https://github.com/webmachinelearning/webnn/issues/450 -> Issue 450 Use web spec best practices (by zolkis) [conventions-integration]
14:49:57 [anssik]
... thanks to Joshua's contributions we now have a Bikeshed build with no warnings! :)
14:50:01 [anssik]
RRSAgent, draft minutes
14:50:02 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/08/24-webmachinelearning-minutes.html anssik
14:50:18 [anssik]
zkis: 150 commits in that PR, squashed from many commits
14:50:37 [anssik]
... adding algorithms that were missing, following modern specification best practices
14:50:54 [anssik]
... jsbell helped a lot here, thank you
14:50:58 [anssik]
... a lot of work over the past 2 week
14:51:25 [anssik]
... other changes waiting for this to land, happy to report we did more than what we promised to do
14:51:45 [anssik]
... thanks to the extended team for reviews, I included names in the ack section of the spec
14:51:55 [anssik]
... next step is for editors to approve the big PR and merge it
14:52:18 [anssik]
... I can do quick fixes, but planning to start holiday next week
14:53:20 [anssik]
anssik: can you work with Chai to merge this PR?
14:53:32 [anssik]
... any concerns from anyone for merging this PR to main?
14:54:19 [anssik]
Ningxin_Hu: LGTM, some remaining open issues we can keep open in GH
14:54:50 [anssik]
... we do a final check and let Zoltan know if last-minute changes are needed
14:55:23 [anssik]
jsbell: I want to acknowledge I joined the process very late, appreciate your support for my contributions
14:56:50 [anssik]
anssik: I'm hearing we are ready to merge after a final check by Ningxin, Ningxin to work with Chai to get his GH approval and then merge
14:57:16 [anssik]
[no concerns with the proposed plan]
14:57:31 [anssik]
anssik: we'll proceed with the merge as noted
14:57:33 [anssik]
anssik: thank you everyone!
14:58:12 [anssik]
zkis: I will handle issues, mostly will be away
14:58:16 [anssik]
Ningxin_Hu: we can do this tomorrow latest
14:59:00 [anssik]
RRSAgent, draft minutes
14:59:01 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/08/24-webmachinelearning-minutes.html anssik
17:00:19 [Zakim]
Zakim has left #webmachinelearning