14:50:58 RRSAgent has joined #webmachinelearning 14:51:02 logging to https://www.w3.org/2024/01/25-webmachinelearning-irc 14:51:02 RRSAgent, make logs Public 14:51:03 please title this meeting ("meeting: ..."), anssik 14:51:06 Meeting: WebML WG Teleconference – 25 January 2023 14:51:07 Chair: Anssi 14:51:12 Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2024-01-25-wg-agenda.md 14:51:15 Scribe: Anssi 14:51:19 scribeNick: anssik 14:51:31 gb, this is webmachinelearning/webnn 14:51:32 anssik, OK. 14:51:40 Present+ Anssi_Kostiainen 14:57:31 jsbell has joined #webmachinelearning 14:57:36 Present+ Joshua_Bell 14:58:25 RRSAgent, draft minutes 14:58:27 I have made the request to generate https://www.w3.org/2024/01/25-webmachinelearning-minutes.html anssik 14:58:51 Present+ Zoltan_Kis 14:59:49 Present+ Rafael_Cintron 15:00:01 RafaelCintron has joined #webmachinelearning 15:00:30 Ningxin_Hu has joined #webmachinelearning 15:01:16 Present+ Austin_Sullivan 15:01:26 Present+ Ningxin_Hu 15:01:34 Present+ Bryan_Bernhart 15:01:54 Present+ Joshua_Lochner 15:02:03 Present+ Phillis_Tang 15:02:09 Present+ Chai_Chaoweeraprasit 15:02:18 RRSAgent, draft minutes 15:02:19 I have made the request to generate https://www.w3.org/2024/01/25-webmachinelearning-minutes.html anssik 15:02:50 Topic: Delta wide review and a new Candidate Recommendation 15:03:11 chai has joined #webmachinelearning 15:03:13 anssik: I want to review the proposed plan for delta wide review and a new CR Snapshot expected in Q1'24. 15:03:20 ... let me recap the key concepts: 15:03:40 zkis has joined #webmachinelearning 15:03:48 dwayner has joined #webmachinelearning 15:04:20 ... - wide review: objective is to ensure all web stakeholders are able to perform reviews of the spec and provide comments, "delta" prefix means we seek feedback and comments on changes since last Q1'23 CR Snapstop publication 15:04:37 asully has joined #webmachinelearning 15:05:15 ... - CR Snapshot: if substantive changes are made to a CR other than to remove features, the WG should publish a new CR Snapshot. This publication has more weight than CR Draft in that it signals it has gone through closer scrutiny 15:05:34 Subtopic: Delta wide review 15:06:04 anssik: I'm happy to handle on behalf of the WG all the wide review coordination and CR publication tasks, but I will seek the WG's review to ensure we're all aligned 15:06:24 ... my proposed plan is for the WG is to build upon the work we've done for the initial CR published March 2023 15:06:30 ... so we're focus on changes since our initial CR in Q1'23 15:06:49 ... I'll discuss a few important things we need to agree on to qualify for a new CR Snapshot publication 15:07:08 Subtopic: Implementation experience 15:07:26 anssik: I want the WG to highlight in the new CR Snapshot the subtantive progress made in implementation experience since initial CR. 15:07:49 ... I propose we use the WebNN implementation status page page as an evidence (shout out to Belem & co for keeping this important resource up to date!) 15:08:02 -> Implementation Status of WebNN Operations https://webmachinelearning.github.io/webnn-status/ 15:08:37 anssik: given WebNN API sits at the middle in the "Web ML stack", we're tracking both web engine and browser implementations as well as frameworks that are the prime consumers of the WebNN API 15:08:42 Phillis_Tang has joined #webmachinelearning 15:08:53 ... for Chromium-based browsers Chrome and Edge, we have XNNPACK/CPU backend and DirectML/GPU backend 15:09:06 ... for ChromeOS, we have MLService/CPU backend 15:09:29 ... as of today, we're 42%, 90%, 17% code complete respectively, with DirectML backend most advanced 15:09:53 ... in addition to browser implementations, we're implementing JS ML framework integrations to WebNN API 15:09:58 ... currently focusing on TensorFlow Lite for TF.js External Delegate and ONNX Runtime Web Execution Provider 15:10:15 ... (consider these as the glue libraries between the framework and the WebNN API) 15:10:36 ... currently, TF integration in being worked on in a fork, ONNX is in upstream and is 94% code complete 15:10:50 anssik: the Implementation Status page also links to w-p-t dashboards for details 15:11:33 q+ 15:11:37 ack RafaelCintron 15:11:42 Present+ Dwayne_Robinson 15:12:02 Joshua_Lochner has joined #webmachinelearning 15:12:23 RafaelCintron: for Apple platforms there's a Chromium PR that adds basic support for WebNN, basic infrastructure, translating all ops, WIP 15:12:59 q? 15:13:29 Subtopic: Test coverage 15:14:14 anssik: We are also expected to demonstrate how we ensure implementations are and will remain interoperable with new implementations we don't yet know about 15:14:21 ... the cross-browser and cross-platform web-platform-tests is our tool for that 15:14:28 -> Web Platform Tests dashboard for WebNN https://wpt.fyi/results/webnn 15:14:46 ... currently we have in total 3750 subtests with a pass rate of roughly 40% 15:14:56 brb 15:14:59 CoreML Initial backend standup https://chromium-review.googlesource.com/c/chromium/src/+/5075312 15:14:59 q+ 15:15:05 ack Ningxin_Hu 15:15:24 Ningxin_Hu: 40% represents XNNPACK CPU backend tests, it does not test GPU yet 15:15:50 ... XNNPACK op coverage is tested, matches 41% op coverage for XNNPACK 15:16:11 b 15:16:41 ... my expectation is we continue to evolve the w-p-t alongside the spec and our pass rate will increase, we will add more test cases for existing ops too e.g. when we unearth new edge cases (thanks Bruce & co for you work on w-p-t!) 15:17:04 Subtopic: Significant new & removed features, conventions, use cases 15:17:15 anssik: We should also note significant new & removed features, conventions update, new use cases since our previous CR Snapshot in Q1'23 15:17:32 ... this information usually goes to the Status section of the spec, I'll prepare a PR for the WG to review 15:17:43 ... for new features, I propose to highlight the new ops and data types "int64" and "uint64" added to support well-known transformers landed in #478 and discussed in #375 15:17:43 https://github.com/webmachinelearning/webnn/issues/478 -> CLOSED Pull Request 478 Add support for operations needed for well-known transformers e.g. Segment Anything, Stable Diffusion, etc. (by wchao1115) 15:17:43 https://github.com/webmachinelearning/webnn/issues/375 -> Issue 375 Support for transformers (by dontcallmedom) [v2] [operation set] 15:18:07 ... I also want to note any ops removed based on implementation experience to streamline the API 15:18:28 ... I have pushed a PR for updated use cases #507, thanks for your review JoshuaB and Zoltan, I will merge this soon 15:18:28 https://github.com/webmachinelearning/webnn/issues/507 -> Pull Request 507 Revise use cases with transformers (by anssiko) 15:19:08 ... Zoltan and JoshuaB have improve the spec conventions significantly, I want to highlight this work, it greatly enhances interop by removing ambiguity and makes future implementers work easier 15:19:45 ... to summarize, I'll prepare a PR for the WG to review the changes to the spec to prepare it for CR 15:19:57 q? 15:20:10 q+ 15:20:13 ack Ningxin_Hu 15:20:39 Ningxin_Hu: I'd like to give a heads up about the sync API and implementation experience from ONNX RT EP 15:21:19 https://bugs.chromium.org/p/chromium/issues/detail?id=1488162 15:21:22 ... we did a performance comparison for sync vs async via asyncify 15:21:37 ... ask is to check the state of the sync API 15:22:22 https://github.com/microsoft/onnxruntime/pull/19145 15:22:22 ... we are informed JSPI, promise integration is coming, comparing with asyncify model inference perf is close to sync esp with WebGPU 15:22:48 Ningxin_Hu: ONNX RT is using async API rather than sync API as before, proposal for the WG to consider dropping sync API support 15:23:14 ... this is to remove a feature 15:24:22 anssik: thanks for this, we have an option to mark the sync API as "at risk" if we believe it might take longer to deprecate 15:24:44 q? 15:25:38 Ningxin_Hu: proposed for removal are computeSync(), buildSync() and createContextSync() specifically 15:26:14 q+ 15:26:24 jsbell has joined #webmachinelearning 15:26:28 q? 15:26:59 anssik: is this spec feature removal an intrusive change? 15:27:32 Ningxin_Hu: I need to look carefully, but I think we can just drop the sync execution paths are remove these three 3 *Sync() methods 15:27:48 ack chai 15:28:12 I'll open an issue 15:28:16 chai: I think it'd be helpful to capture this in an issue and PR for the record, but sounds reasonable to me 15:28:17 q? 15:28:40 Subtopic: Refreshing the current Status section of the spec 15:28:46 -> WebNN: Status of this document (SOTD) https://www.w3.org/TR/webnn/#sotd 15:28:52 anssik: SOTD is for busy people to get a high-level view where we're at 15:29:00 ... I want to seek the WG's feedback on how to update this section, currently it reads: 15:29:04 "Further implementation experience and user feedback is being gathered for the MLCommandEncoder interface that proposes to enable more efficient WebGPU integration. A proposal to simplify MLContext creation is being discussed. This document is maintained and updated at any time. Some parts of this document are work in progress and further improvements are expected to be reflected in revised Candidate Recommendation Drafts and 15:29:04 Snaphots." 15:29:14 anssik: my questions: 15:29:22 ... - how do we want to phrase the MLCommandEncoder status? 15:30:06 chai: I think it depends what we think is the milestone for the next CR 15:30:23 ... we want to highlight delta, do we think WebGPU interop is a milestone we want to cross? 15:31:04 anssik: I think we should go to CR in Q1 and not block on WebGPU interop 15:31:52 chai: MLCommandEncoder is there and has been there since last CR 15:32:14 ... - what is the current status re "simplify MLContext creation" #322? 15:32:15 https://github.com/webmachinelearning/webnn/issues/322 -> Pull Request 322 Simplify MLContext creation (by wchao1115) 15:32:56 chai: I think at this point we should remove this sentence 15:32:57 q? 15:33:31 anssik: we're drop Simplify MLContext creation from the status, not highlight it 15:33:52 ... - MLBuffer, should we note this important work in status given it provides an abstraction for CPU, GPU, NPU to interop more efficiently? 15:34:06 q? 15:34:50 ... - any other new features, removals of important WIP topics to highlight to busy people in this section, chime in on the upcoming PR 15:34:54 RRSAgent, draft minutes 15:34:55 I have made the request to generate https://www.w3.org/2024/01/25-webmachinelearning-minutes.html anssik 15:35:17 jsbell has joined #webmachinelearning 15:35:30 Topic: Issue prioritization 15:35:39 q+ 15:35:50 anssik: As we're approaching another spec milestone aka CR Snapshot I want to discuss with you: 15:36:05 ... - the most urgent and important issues for the group, or buckets of issues 15:36:27 ... - practical steps we can take to make those issues more visible and actionable to the broader group 15:36:49 ... I'd characterize the WG's current work mode as implementation-driven. That is a great work mode. 15:37:16 ... OTOH that means that while the core group has a shared understanding of where our priorities are, the broader group would benefit from hints and guidance where they should focus their attention and contributions 15:37:49 ... I've used these calls to "check the pulse" on issues to build shared understanding, but not everyone can join these calls, and our meeting cadence cannot fit all open issues 15:38:20 ... I should acknowledge spec contributions come in many shapes and forms, a typical implementation-driven contribution is a normatively defined feature, this is what I label as "new features" in our agendas 15:38:28 ... in addition to "new features" there's a wide range of other contributions: 15:38:39 ... - identifying and reporting issues, also help spot stale issues or suggest issues to be closed if addressed 15:38:57 ... - patches to keep the spec in a cohesive state e.g. refresh informative parts whenever there's a normative change 15:39:23 ... e.g. code examples, use cases, explainer, programming model updates, privacy & security, ethical considerations, notes to implementers (notes are those green boxes) 15:39:41 ... - patches that improve normative definitions, fix bugs, align with conventions 15:39:46 ... on the agenda these are known as "enhancements" 15:40:00 ... contributions outside "new features" are equally or not more important and are great opportunities for new contributors, no contribution is too small 15:40:26 ... I'd like to open the floor for discussion and brainstoring on concrete things we could do as a group to help new contributors join with concrete contributions 15:40:39 ... maybe we can create a group of volunteers to triage our issues, we can use the GH facilities at our disposals better (labels etc.) 15:40:42 q? 15:40:58 ack jsbell 15:41:31 jsbell: thanks Zoltan, Ningxin, Anssi for reviews of my PRs 15:41:44 ... as a new contributor it's been hard to understand where to focus my contributions 15:42:17 ... started with clean up first, then going over the 100+ issue we have open and labels would help 15:42:44 ... a bulk of the issues are about a specific op and not substantial architectural issues 15:43:00 ... knowing the status would help, e.g. "needs a PR" label 15:43:24 ... "ops that cross different backends" would be great to know 15:44:11 ... I know there are some meta issues, prefer smaller issues over big meta issues 15:44:48 ... big directional issues, e.g. sync/async, op sets StableHLO, gigantic issues may be hidden, these should be surfaced clearly with a label 15:45:05 ... also areas where the spec may be incomplete even if no interop issues 15:45:32 ... e.g. details of some ops refer to references behind a paywall, this is an issue, we should unearth these and label them 15:45:33 q? 15:46:35 q? 15:47:15 q? 15:47:22 +1 on opset query as important issue we should tackle 15:47:31 Topic: New features 15:47:36 Subtopic: MLBuffer 15:47:41 jsbell has joined #webmachinelearning 15:47:54 anssik: I want to use this call for a synchronous discussion on the proposal for a backend-agnostic storage type for WebNN operation (aka MLBuffer) informed by implementation experience. 15:48:02 anssik: issue #482 15:48:02 https://github.com/webmachinelearning/webnn/issues/482 -> Issue 482 Support for device-based tensor storage objects (by bbernhar) 15:48:09 -> Chromium implementation: MLBuffer https://chromium-review.googlesource.com/c/chromium/src/+/5173676 15:48:23 anssik: thank you for very professional and in-depth discussion on this MLBuffer issue 15:48:27 ... this is a very complex issue and baking it takes time 15:48:41 ... perhaps Bryan can take the lead in sharing the latest on this feature and to have a discussion on open questions with Rafael chiming in 15:48:47 q? 15:49:04 Bryan: summary is decisions to be made :) 15:49:25 ... how to proceed with buffer transfers? 15:49:51 ... MLBuffer is GPU mapped now to simplify it 15:50:32 ... how does interop work, concrete story upfront? 15:50:42 q? 15:51:08 q+ 15:51:17 ack RafaelCintron 15:51:39 RafaelCintron: thank everyone who participated in this issue, great feedback Austin and Reilly, others 15:52:43 ... WebGPU today has mapAsync, implementation experience 15:53:28 ... WebGPU interop is important, several scenarios for camera input, anything that resembles real-time, WebGPU interop required for those use cases 15:53:45 ... we should stage the work so we don't have a giant PR 15:53:52 q+ 15:54:15 ... we should ship something we want to change later, it is OK to have the implementation develop gradually 15:54:16 q? 15:54:19 ack chai 15:54:53 chai: I agree with Rafael that the best way to qualify a complex proposal is to back it with an implementation and simple samples that show WebGPU interop 15:55:26 ... I think this is a long discussion and people brought out good perspectives, but running code is a compelling way to demonstrate this can be worked out, I would focus on WebGPU interop 15:55:33 q+ 15:55:38 ... conceptually it could be anything but WebGPU is on top of people's minds 15:55:48 q? 15:55:48 ack asully 15:55:54 asully: I reviewed the PR, thanks for the great work! 15:56:14 ... I'm excited about the proposal, I agree WebGPU interop is top of mind, MLBuffer to work with WebGPU 15:56:38 ... assumption that MLBuffer is in GPU may be a mistake, needs to be device-agnostic 15:56:56 ... the buffer could live elsewhere 15:57:10 q+ 15:57:23 ack Ningxin_Hu 15:58:36 Ningxin_Hu: thanks for the great discussion, agree WebGPU MLBuffer interop is important, want to highlight non-interop part is a demanded feature, chained inference scenario for LLMs 15:58:36 ... ONNX has I/O Binding with WebGPU backend to keep the output from the previous inference and use it as input for the next inference 15:58:44 ... if we only consider non-interop parts that is super helpful in itself 15:58:54 ... for the chained inference scenario 16:00:08 ... API change for the WebNN spec can fullfill this use case, agree with Austin we need to consider WebGPU interop but also want to note this WebNN only use case is important 16:00:08 q+ 16:00:08 ... Rafael's proposal for staging work sounds good 16:00:08 q? 16:00:08 ack asully 16:00:21 asully: parallel to the discussion on MLBuffer, WebNN timelines and WebGPU timelines need some addition work 16:00:40 q? 16:01:42 q? 16:01:49 Bryan: I can put down the opens and everyone's positions on them 16:03:18 anssik: we test if "everyone can live with a proposal" to test if we can more forward, as a test for consensus 16:03:33 RRSAgent, draft minutes 16:03:34 I have made the request to generate https://www.w3.org/2024/01/25-webmachinelearning-minutes.html anssik 16:05:35 s/page page/page 16:07:25 s/Ningxin_Hu: my expectation/anssik: my expectation 16:07:28 RRSAgent, draft minutes 16:07:30 I have made the request to generate https://www.w3.org/2024/01/25-webmachinelearning-minutes.html anssik 16:12:55 s/have improve/have improved 16:14:33 s/are remove/and remove 16:16:20 s/block on WebGPU interop/block on WebGPU interop and keep the current status text 16:18:22 s/or not/if not 16:22:03 s/more forward/move forward 16:22:07 RRSAgent, draft minutes 16:22:08 I have made the request to generate https://www.w3.org/2024/01/25-webmachinelearning-minutes.html anssik 18:14:23 Zakim has left #webmachinelearning