IRC log of webmachinelearning on 2023-09-21

Timestamps are in UTC.

13:54:36 [RRSAgent]
RRSAgent has joined #webmachinelearning
13:54:40 [RRSAgent]
logging to https://www.w3.org/2023/09/21-webmachinelearning-irc
13:54:40 [Zakim]
RRSAgent, make logs Public
13:55:11 [anssik]
Meeting: WebML WG Teleconference – 21 September 2023
13:55:11 [Zakim]
please title this meeting ("meeting: ..."), anssik
13:55:13 [anssik]
Chair: Anssi
13:55:13 [anssik]
Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2023-09-21-wg-agenda.md
13:55:13 [anssik]
Scribe: Anssi
13:55:13 [anssik]
scribeNick: anssik
13:55:13 [anssik]
gb, this is webmachinelearning/webnn
13:55:13 [gb]
anssik, OK.
13:55:13 [anssik]
Present+ Anssi_Kostiainen
13:55:23 [anssik]
RRSAgent, draft minutes
13:55:24 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/09/21-webmachinelearning-minutes.html anssik
13:56:59 [etienne]
etienne has joined #webmachinelearning
13:58:42 [anssik]
Present+ Etienne_Noel
14:00:31 [jsbell]
jsbell has joined #webmachinelearning
14:00:39 [anssik]
Regrets+ Dominique_Hazael-Massieux
14:00:45 [anssik]
Present+ Joshua_Bell
14:00:50 [ningxin_hu]
ningxin_hu has joined #webmachinelearning
14:00:53 [anssik]
Present+ Joshua_Lochner
14:01:12 [anssik]
Present+ Ningxin_Hu
14:03:10 [Joshua_Lochner]
Joshua_Lochner has joined #webmachinelearning
14:03:13 [anssik]
anssik: please welcome Etienne Noël from Google to the WG! SW Eng Manager at Google Chrome, has worked on Project Fugu among other things, also experience with Storage APIs, caching all the things, Pacific timezone
14:03:31 [anssik]
Present+ Chai_Chaoweeraprasit
14:03:40 [anssik]
Present+ Dwayne_Robinson
14:04:05 [anssik]
Present+ Rafael_Cintron
14:04:13 [anssik]
Topic: Update from TPAC 2023
14:04:42 [RafaelCintron]
RafaelCintron has joined #webmachinelearning
14:05:02 [anssik]
anssik: As discussed and agreed, the W3C's Web Machine Learning WG did not formally meet at this year's TPAC. However, the WG's work received a lot of interest during the hallway track and in other meetings.
14:05:10 [anssik]
... AI/ML topics were discussed in the Advisory Committee meeting and in multiple breakout sessions:
14:05:22 [anssik]
-> AI @ W3C (AC meeting) https://www.w3.org/2023/Talks/TPAC/ac-activities-update/
14:05:22 [chai]
chai has joined #webmachinelearning
14:05:48 [anssik]
-> Ethical AI (breakout) https://www.w3.org/2023/09/TPAC/breakouts.html#b-9f1667d4-a23b-4893-80aa-06d5ecca8434
14:05:52 [anssik]
-> Generative AI (breakout) https://www.w3.org/2023/09/TPAC/breakouts.html#b-53380d18-eb3f-47f2-a6ac-435ce6294e0c
14:06:08 [anssik]
... in the AC meeting, W3C's Dom gave an update on "AI @ W3C".
14:06:42 [anssik]
... WebNN API and Ethical Principles work noted explicitly as key efforts in this space, also our latest work on transformer-based generative models and the broader impact of generative AI was discussed
14:06:53 [anssik]
... generative AI discussion was centered around impacts:
14:07:03 [anssik]
... - transforming content creation
14:07:09 [anssik]
... - interaction with content and services
14:07:17 [anssik]
... - new centralized risks
14:07:26 [anssik]
... - web-scraped content re-use (for training)
14:07:37 [anssik]
... questions for W3C roadmap that came up:
14:07:51 [anssik]
... - marking content as AI generated
14:08:00 [anssik]
... - marking content usable for training
14:08:14 [anssik]
... - exposing browser-provided ML models through Web APIs
14:08:35 [RachelY]
present+
14:08:42 [anssik]
... - large asset download/caching/sharing from browsers
14:08:56 [anssik]
... - delegating partial model execution to the network
14:09:11 [anssik]
... operational questions for W3C:
14:09:19 [anssik]
... generative AI role in:
14:09:22 [anssik]
... - spec writing
14:09:26 [anssik]
... - test writing
14:09:30 [anssik]
... - docs writing
14:10:01 [anssik]
anssik: in both the breakout sessions on Ethical AI and Generative AI the discussion was pretty much around ethical issues, and not so much on technical solutions
14:10:33 [anssik]
... in a WG chair capacity I invited interested folks who were interested to join the WG and help contribute to the Ethical Principles guidelines
14:10:44 [anssik]
... I shared that the WG is currently focused on delivering on its WebNN API v2 technical goals and we'd welcome new participants to help drive the ethical work forward.
14:11:17 [anssik]
... that's my summary from TPAC relevant to this WG, questions?
14:11:59 [RachelY]
q+
14:12:11 [anssik]
ack RachelY
14:12:27 [anssik]
RachelY: it was great to see Anssi IRL
14:13:01 [anssik]
... we had a breakout session on computational intelligence too, Joshua Lochner joined and university researchers
14:13:33 [anssik]
-> Unleashing the Power of Computational Intelligence on the Web https://www.w3.org/events/meetings/ab0de26f-c84d-40b6-80a7-79f3020da3aa/
14:14:57 [anssik]
... I'd like to ask this WG's participants to support the Community Group we're starting for Computational Intelligence
14:16:12 [anssik]
q?
14:16:25 [anssik]
Topic: WebNN v2: Proposed model targets
14:16:39 [anssik]
anssik: Over the last few months the WG has identified the following as its proposed v2 model targets:
14:16:44 [anssik]
... - Text-to-image: Stable Diffusion unet/VAE/text encoder
14:16:55 [anssik]
... - Image segmentation: Segment Everything decoder
14:16:59 [anssik]
... - Speech-to-text: Whisper Tiny
14:17:09 [anssik]
... - Text-to-text generation (encoder-decoder): t5 and m2m100
14:17:16 [anssik]
... - Text-generation (decoder-only): llama
14:17:28 [chai]
q+
14:17:43 [anssik]
... the participants have proposed these targets that are backed with implementation experience, from the WebNN Chromium DirectML prototype and Transformers.js
14:17:46 [anssik]
ack chai
14:18:17 [anssik]
chai: in the topic of transformer support, want to solicit feedback from the WG, I think this will be iterations
14:18:31 [anssik]
... we can start with the models now proposed, SD, Segment Everything
14:18:41 [anssik]
... we want to acknowledge adding this work will be incremental
14:18:44 [anssik]
q?
14:19:40 [anssik]
... I think SD and Segment Everything are good start
14:20:05 [anssik]
... adding these to the spec will help the browsers improve other browser infrastructure such as the caching system
14:20:19 [anssik]
... SD 1.5 is a good starting point, not too big
14:21:07 [anssik]
... there is also interest in running SD on NPU, NPU support is separate from transformers support, but they're actually almost intertwined
14:21:22 [anssik]
... good opportunity to tack NPU support at the same time start small
14:21:49 [anssik]
q?
14:21:55 [ningxin_hu]
q+
14:21:58 [anssik]
ack ningxin_hu
14:22:36 [anssik]
ningxin_hu: initial targets are good, with my collegue Wanming we started op breakdown work, want to provide that as input to the WG in the coming days
14:22:54 [anssik]
... in this analysis we observed almost all these models in ONNX leverage dynamic shaping for input size
14:23:05 [anssik]
... today WebNN supports only static size tensors
14:23:28 [anssik]
... we looked at how WebNN implementation can handle this, leave dynamic shape to framework
14:23:43 [anssik]
... let WebNN deal with static shapes, doing this investigation in parallel
14:23:58 [anssik]
... we'll also provide that input, Jiawei also contributed to this work
14:24:19 [Joshua_Lochner]
@Rachel @Anssi Link to my TPAC presentation (I reused the template from my previous talk, and updated the content to include the other HF JavaScript libraries): https://docs.google.com/presentation/d/1hx-Y1HFw_F88FJiqZnJ3FYlRnj7HKKwuYkJIRL0TcU0/edit?usp=sharing
14:24:19 [gb]
https://github.com/Rachel -> @Rachel
14:24:19 [gb]
… https://github.com/Anssi -> @Anssi
14:24:34 [anssik]
q?
14:25:01 [jsbell]
No concerns, just excited!
14:25:12 [RachelY]
I submitted a proposal for a business group with several university research groups and companies joining as new members. You are invited to support the creation of this group: http://www.w3.org/community/groups/proposed#computationalibg I would appreciate your input on how this community group can contribute to new use cases and learn from the good work of the Web ML WG.
14:25:17 [anssik]
... if there are no concerns with this set of model targets I'd ask the WG to initiate work on an op breakdown to better understand what is common across these architectures to inform WebNN API v2 priorities
14:25:27 [anssik]
... this work is similar to what we did for the so-called first-wave models a few years back when "v1" API effort was kicked off
14:25:32 [anssik]
-> The first-wave models op breakdown https://github.com/webmachinelearning/webnn/blob/main/op_compatibility/first_wave_models.md
14:25:38 [anssik]
... at that time we looked at the following models:
14:25:45 [anssik]
... - Image classification: SqueezeNet, MobileNet, ResNet
14:25:51 [anssik]
... - Object detection: TinyYOLO
14:25:55 [anssik]
... - Noise suppression: RNNoise, NSNet
14:26:20 [anssik]
... we don't need to copy this approach exactly, the important part is to conduct this work in public to allow the public to review and participate
14:26:24 [anssik]
... issue #375 is the place for discussions, suggestions and feedback
14:26:25 [gb]
https://github.com/webmachinelearning/webnn/issues/375 -> Issue 375 Support for transformers (by dontcallmedom) [v2] [operation set]
14:26:41 [anssik]
... from that issue you'll also find Dwayne's initial proposal for v2 ops for the proposed text-to-image, image segmentation and speech-to-text models
14:26:57 [anssik]
... comments? it seems we're ready to move forward with this plan?
14:27:13 [anssik]
RRSAgent, draft minutes
14:27:14 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/09/21-webmachinelearning-minutes.html anssik
14:27:21 [anssik]
q?
14:27:33 [anssik]
Topic: Storage APIs for caching large models
14:28:04 [anssik]
anssik: I'd like to have a discussion on issues related to caching large models in the browser, current Storage APIs, specific issues and workarounds for those APIs.
14:28:14 [anssik]
... first, I'd like to set the expectation and then let Joshua share his insights on Storage APIs in general
14:28:22 [anssik]
... to frame this discussion:
14:28:44 [anssik]
... I'm not proposing we make any changes to the WebNN API to support caching, I'd like us to discuss issues related to current Storage APIs that could be submitted as feedback to the respective Storage API specs
14:28:59 [anssik]
... Storage APIs (plural) in browsers mean many things:
14:29:22 [anssik]
... for browsers we have: IndexedDB, localStorage, sessionStorage, Service Worker Cache API, applicationCache (deprecated, superseded by Service Worker Cache API)
14:29:39 [jsbell]
q+
14:29:40 [anssik]
... there's also the Storage Standard that defines APIs for persistent storage and quota estimates across all these Storage APIs.
14:29:47 [anssik]
-> IndexedDB https://www.w3.org/TR/IndexedDB/
14:29:50 [anssik]
-> localStorage https://html.spec.whatwg.org/multipage/webstorage.html#the-localstorage-attribute
14:29:53 [anssik]
-> sessionStorage https://html.spec.whatwg.org/multipage/webstorage.html#dom-sessionstorage
14:29:57 [anssik]
-> Service Worker Cache API https://w3c.github.io/ServiceWorker/#cache-interface
14:30:02 [anssik]
-> applicationCache (deprecated, background info) https://web.dev/appcache-removal/
14:30:05 [anssik]
-> Storage Standard https://storage.spec.whatwg.org/
14:30:27 [anssik]
... I'd like to Joshua to give an intro to this topic and then we can discuss the pain points with current APIs for caching large models.
14:30:48 [anssik]
jsbell: some background info then handover to Etienne
14:31:11 [anssik]
... first, we need to distinguish imperative from network caching
14:31:28 [anssik]
... latter is not explicitly controlled, by hinted by HTTP Headers
14:31:41 [anssik]
... network caching in confusing to web developers
14:31:57 [anssik]
... x-site resources could be cached and it make loading pages faster
14:32:18 [anssik]
... we've learned that is a privacy concern, so using separate network caches nowadays
14:32:50 [anssik]
... today two top-level site a.com and b.net, even if the same resource is requested from CDN they receive separate resources
14:33:32 [anssik]
... there has been research on whether we can maintain privacy properties with transparent caches
14:34:19 [anssik]
... this WG wants to cache GBs, more pressing need to transparent caching or something that the browser caches and can be shared
14:34:38 [anssik]
... this leads to sites using imperative APIs to ensure data is not downloaded more than once
14:34:56 [anssik]
... a few new Storage APIs added include: File System API
14:35:11 [anssik]
... there's also a similarly names older API, not to be confuesd
14:35:45 [anssik]
... this new File System API aka Origin-Private File System (OPFS) ships in all browsers
14:36:00 [anssik]
... POSIX file semantics, accessible synchronously from workers
14:36:14 [anssik]
... works with Wasm, Emscipten native support
14:36:31 [anssik]
... AppCache is gone as Anssi mentioned, we can forget it
14:36:58 [anssik]
... SW Cache API is a replacement for AppCache, resource on any site pulled down, they're immutable resources, Cache API may be a good fit for those
14:37:17 [anssik]
... API fully accessible from Worker and Window contexts, please use it also outside ServiceWorker context
14:37:32 [anssik]
... IndexedDB has complexity on top of KVS
14:37:35 [anssik]
q?
14:37:40 [anssik]
ack jsbell
14:37:59 [anssik]
etienne: I've researched memory impact of downloading large file
14:38:21 [anssik]
... tested Cache API, OPFS
14:38:59 [Joshua_Lochner]
q+
14:39:14 [anssik]
... downloading without hitting memory important, wanted to hear issues and what can we do to improve the existing APIs, do we need to inform developers better, do these APIs not meet the reqs of web developers?
14:39:20 [anssik]
ack Joshua_Lochner
14:39:59 [anssik]
Joshua_Lochner: great to meet you Etienne! I've created Transformers.js and for the most part it's been way simpler that people would expect to cache, I have >1GB model cached
14:40:16 [jsbell]
q+
14:40:48 [anssik]
... released today is in-browser close of HF Chat UI that is 800M param model, using SW Cache API, Transformers.js only uses the Cache API
14:41:22 [anssik]
... early creation of Transformers.js requested HF Hub and stored model downloaded with Cache API on the client
14:41:55 [anssik]
... top-level requirements, share store, manage multigig models
14:42:13 [anssik]
... key issue for me is x-site sharing of models
14:42:34 [anssik]
... now have to download model A on sites 1, 2 and 3 and share across the sites.
14:42:50 [anssik]
... workaround is to use a browser extension
14:43:57 [anssik]
... I created a sample browser extension and had issues with IndexedDB, when set active tab permission in manifest.json, running IndexedDB in the context of the web page had issues
14:44:26 [anssik]
... I serialized model data with base64 to work around this which of course increased the model size to work around the sharing limitation
14:44:50 [anssik]
... File System API is also another alternative for sharing across domain, the problem is it requires a buy-in from the user
14:44:53 [chai]
q+
14:45:07 [anssik]
... I want to make this completely invisible to the user where the model is stored and how it is stored
14:45:24 [anssik]
... needing to ask the user where the model is shared in his system is a problem
14:45:25 [anssik]
q?
14:45:41 [anssik]
ack jsbell
14:46:11 [anssik]
jsbell: thanks for the good news using the Cache API just worked with large models
14:46:26 [anssik]
... web developers don't understand always Cache API has improved over time so much
14:46:45 [anssik]
... back in the day only localStorage and sessionStorage with 5MB was available to web developers
14:47:00 [anssik]
... Chrome has improved quota and other browsers have followed
14:47:46 [anssik]
... limited storage wrt native apps was a big gap, in Chrome we piloted an upper limit of sites able to use up to 80% of the available disk space, we clean up that pool
14:47:56 [anssik]
... sites may compete for storage but that is not common
14:48:19 [anssik]
... Chrome, Safari, Firefox, Edge all can store gigabyte models today
14:48:51 [anssik]
... querying quota is also supported, Safari 17 supports the Storage API and its quota features
14:49:24 [anssik]
... JoshuaL talked about browser extensions to mediate between sites, going beyond web proper, but we want to learn how web developers work around these issues
14:49:34 [anssik]
... good way to prototype these things
14:49:50 [anssik]
... Anssi mentioned at TPAC, what if browser shipped common models
14:50:20 [anssik]
... interesting idea, wonderful exploration
14:51:18 [anssik]
... re IndexedDB issues, each top-level site sees its own storage, extension uses the extension's domains and proxies the data from the extension to the individual pages
14:51:38 [anssik]
... using base64'd strings to pass data shouldn't be needed
14:52:12 [anssik]
... new File System API has access to local files user can interact with, allows image editor on the web
14:52:36 [anssik]
... internals of this API and file handles have access to this OPFS, similar to IndexedDB, hidden from the user
14:53:06 [anssik]
... the privacy FS is hidden from the user, partitioned by top-level site, does not help with x-site sharing of cached models
14:53:07 [anssik]
q?
14:53:13 [anssik]
ack chai
14:53:15 [Joshua_Lochner]
Thanks so much Josh B!
14:53:55 [etienne]
Joshua L. I definitely want to follow up with you to understand better why serializing was necessary, we can discuss offline
14:54:11 [Joshua_Lochner]
Sounds good!
14:54:11 [anssik]
chai: this reminds me of the earlier days of web browser that video file is too large to be cached, reminds me of the evolution of media, over time problems are mitigated by ecosystem, web sites and browsers, e.g. video streaming fixes the video download problem
14:54:38 [anssik]
... LLM download issue may solve itself in creative ways in the future
14:54:39 [anssik]
q?
14:54:50 [anssik]
RRSAgent, draft minutes
14:54:51 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/09/21-webmachinelearning-minutes.html anssik
14:55:17 [anssik]
q?
14:55:23 [RafaelCintron]
q+
14:55:43 [anssik]
q?
14:55:51 [anssik]
ack RafaelCintron
14:56:23 [anssik]
RafaelCintron: JoshuaB, what about the domain restrictions? Can JL do something to make users live easier?
14:56:40 [anssik]
jsbell: privacy protection is the most important
14:56:57 [anssik]
q?
14:57:29 [anssik]
Topic: Intel Innovation update
14:58:39 [anssik]
chai: Intel Innovation 23 was this week, I was invited to give a talk on WebNN, I'll share the presentation
14:58:42 [anssik]
... a few important announcements, Gaudi datacenter, also AI chip in Intel MTL platform aka Intel Core Ultra
14:59:48 [anssik]
... I sense people want to see models running on NPU now, when we talked about WebNN, we have three processor types, Wasm for CPU SIMD, WebGPU for GPU, WebNN as the standard solution to accelerate on GPU and NPU, this was well received
15:00:00 [AramZS]
AramZS has joined #webmachinelearning
15:00:24 [anssik]
... implementation work in Chromium on DirectML was also discussed
15:01:07 [anssik]
... the web is coming up, AI on the web can be accelerated similarly to what happens outside the web
15:01:18 [anssik]
anssik: thanks for the update Chai!
15:01:57 [chai]
The link to the deck is here: https://static.rainfocus.com/intel/innv2023/sess/1689805601141001Xyjy/supmat/DevelopingWebBasedAIApps_Final_1694887505835001jjpL.pdf
15:03:13 [anssik]
RRSAgent, draft minutes
15:03:14 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/09/21-webmachinelearning-minutes.html anssik
15:03:48 [anssik]
q?
15:52:39 [anssik]
s/Intel Innovation update/Update from Intel Innovation
15:52:41 [anssik]
RRSAgent, draft minutes
15:52:42 [RRSAgent]
I have made the request to generate https://www.w3.org/2023/09/21-webmachinelearning-minutes.html anssik
16:24:14 [AramZS]
AramZS has joined #webmachinelearning
17:24:12 [Zakim]
Zakim has left #webmachinelearning