14:00:11 RRSAgent has joined #webmachinelearning 14:00:15 logging to https://www.w3.org/2023/05/11-webmachinelearning-irc 14:00:15 RRSAgent, make logs Public 14:00:16 please title this meeting ("meeting: ..."), anssik 14:00:17 Meeting: WebML WG Teleconference – 11 May 2023 14:00:21 Chair: Anssi 14:00:26 Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2023-05-11-wg-agenda.md 14:00:33 Scribe: Anssi 14:00:37 scribeNick: anssik 14:00:42 ningxin_hu has joined #webmachinelearning 14:00:53 ghurlbot, this is webmachinelearning/webnn 14:00:53 anssik, OK. 14:00:58 Present+ Anssi_Kostiainen 14:01:05 Present+ Zoltan_Kis 14:01:13 Present+ Ningxin_Hu 14:01:23 Present+ Rafael_Cintron 14:01:40 RafaelCintron has joined #webmachinelearning 14:01:43 Present+ Wanming_Lin 14:02:38 RRSAgent, draft minutes 14:02:39 I have made the request to generate https://www.w3.org/2023/05/11-webmachinelearning-minutes.html anssik 14:03:00 Topic: WebNN - WebIDL and Infra standard conventions 14:03:30 wm has joined #webmachinelearning 14:04:03 Subtopic: The constant() method steps 14:04:08 anssik: meta issue #210 and PR #365 14:04:15 https://github.com/webmachinelearning/webnn/issues/210 -> Issue 210 Use modern WebIDL and Infra standard conventions (anssiko) enhancement, Editorial 14:04:15 https://github.com/webmachinelearning/webnn/issues/365 -> Pull Request 365 Add the constant() method steps. (zolkis) 14:04:21 ... this PR now has the required approvals, ready to merge 14:04:31 ... thanks Zoltan to interating on this PR carefully and Chai and Ningxin for review and comments! 14:05:18 Zoltan: it looks like mergeable, first PR that is a dependency for the rest 14:06:32 ningxin_hu: thanks Zoltan, do we want to squash the commits? 14:06:54 anssik: editors to decide whether squash the commit on merge 14:07:10 Subtopic: Sync and async algorithms 14:08:01 anssik: issue #316 and PR #329 14:08:01 https://github.com/webmachinelearning/webnn/issues/329 -> Pull Request 329 Rework the sync async algorithms based on #323 (zolkis) 14:08:02 https://github.com/webmachinelearning/webnn/issues/316 -> Issue 316 Review sync vs async compute differences (zolkis) Editorial 14:08:08 ... I proposed in the PR to remove the note the "During asynchronous execution ..." note 14:08:14 ... because RFC 2119 terminology is not to be used in an informative note. 14:08:21 ... Zoltan updated the PR accordingly 14:08:33 ... it looks like that was the remaining open conversation on this PR? 14:08:55 Zoltan: yes, that was the only open one 14:09:21 anssik: ready to merge after Chai's final review 14:09:32 Subtopic: The builder.input() method steps 14:09:36 anssik: meta issue #210 and PR #364 14:09:36 https://github.com/webmachinelearning/webnn/issues/364 -> Pull Request 364 Add the builder.input() method steps (zolkis) 14:09:47 ... it looks like this PR has the required approvals, ready to merge 14:09:53 ... thanks again Zoltan for this PR and Chai and Ningxin for your review 14:10:01 Zoltan: this needs rebase before merge 14:10:19 ... I will do the rebase 14:10:42 ... editors are free to merge after that 14:11:00 Subtopic: Review the rest of the standard conventions PRs 14:11:07 anssik: my summary of the status of the remaining standard conventions PRs 14:11:13 ... #337 MLOperand and MLActivation internal slots - 1 change requested by Chai 14:11:14 https://github.com/webmachinelearning/webnn/issues/337 -> Pull Request 337 Add internal slots to MLOperand and MLActivation (zolkis) 14:11:18 ... #348 clamp() algorithm - 1 change requested by Chai & addressed by Zoltan 14:11:18 https://github.com/webmachinelearning/webnn/issues/348 -> Pull Request 348 Add the clamp() algorithm (zolkis) 14:11:22 ... #364 builder.input() method steps - 2 approvals, ready to merge 14:11:26 ... #366 concat() algorithm - 1 change requested by Chai & addressed by Zoltan 14:11:26 https://github.com/webmachinelearning/webnn/issues/366 -> Pull Request 366 Add the concat algorithm (zolkis) 14:11:29 ... #339 batchnorm() algorithm - re-review requested from Chai and Ningxin 14:11:30 https://github.com/webmachinelearning/webnn/issues/339 -> Pull Request 339 Fix #334: Improve the batch norm algorithm (zolkis) 14:12:21 Zoltan: #337 is being taken apart by other PRs because this is the dependency PR, so I'll leave this as the last 14:12:54 ... the rest can be merged after review 14:13:39 Zoltan: after merging these open PRs we do the stylistic PR, these introduce building blocks for the stylistic PR 14:14:44 ... as soon reviews are cleared I make the stylistic change PR and we can merge Chai's outstanding PR #322 14:14:45 https://github.com/webmachinelearning/webnn/issues/322 -> Pull Request 322 Simplify MLContext creation (wchao1115) 14:15:06 Topic: WebNN - enhancements, editorials, questions 14:15:28 Subtopic: float16 support 14:15:40 anssik: I wanted to discuss prototype findings to inform v2 feature work 14:15:52 ... I invited Wanming to join us for this call, working with Zesong he added float16 support to ONNX Runtime Web 14:15:56 -> PR for ONNXRuntime float16 support https://github.com/Honry/onnxruntime/pull/1 14:16:01 anssik: we discussed earlier that ECMA TC39 has a Float16Array proposal, see issue #373 14:16:01 https://github.com/webmachinelearning/webnn/issues/373 -> Issue 373 heads up re: proposal to add Float16Array to JavaScript (bakkot) v2 14:16:05 -> https://tc39.es/proposal-float16array/ Float16Array proposal (ECMA Stage 2 ~proposal) 14:16:28 ... we don't yet have Float16Array in the official JS standard so one workaround is to pass raw bits via Uint16Array typed array that is an ECMA standard 14:16:32 -> https://tc39.es/ecma262/multipage/indexed-collections.html#table-49 Uint16Array (ECMA Stage 4 ~ formal standard) 14:16:47 ... Wanming, thanks for your work on this, I see the PR has been merged 14:17:06 ... did you yet get a chance to try float16 support with some of the models that benefit from float16? Any other insights from your implementation work? 14:18:14 Wanming: this implements translation of float16 to Uint16Array 14:19:37 8% better than float32 model 14:20:49 ningxin_hu: thanks Wanming, this PR is not merged in an official ONNX Runtime repo yet, it is in Wanming's fork as a staging repo for experiment 14:21:39 ... open issue in ONNX Runtime about float16 support, various options discussed, this workaround is one of them 14:21:59 ... Wanming demonstrated this workaround works 14:22:04 ... even if in a downstream fork 14:22:57 ... for WebNN spec we have a mapping table where we note this is not yet natively supported but there's a workaround, this work demonstrates this is doable with a workaround 14:23:16 ... we can share this prototype with the audience 14:23:36 ... performance improvement is good, memory footprint is reduced in ~half 14:23:49 ... Btw. I submitted a small PR #386 to add a provisional reference to the upcoming TC39 Float16Array type into MLOperandType and ArrayBufferView compatibility table 14:23:50 https://github.com/webmachinelearning/webnn/issues/386 -> Pull Request 386 Add float16 to MLOperandType and ArrayBufferView compatibility table (anssiko) 14:24:54 anssik: thanks Wanming for this work! 14:26:04 ningxin_hu: related announcement ONNX Runtime Web support, WebNN Execution Provider was merged in an official repo this week, float16 is a downstream experiment 14:26:31 https://github.com/microsoft/onnxruntime/pull/15698 14:27:02 Subtopic: Subclass MLGraph based on the context that creates it 14:27:07 anssik: issue #344 14:27:07 https://github.com/webmachinelearning/webnn/issues/344 -> Issue 344 Subclass MLGraph based on the context that creates it (huningxin) question 14:27:16 ... we deferred this from our previous call, I was asking whether this mainly an API ergonomics improvement? 14:27:29 ... does the WG feel like pursuing this proposal further? 14:28:04 ningxin_hu: no update at this time 14:28:18 Subtopic: Error handling of MLNamedArrayBufferViews transfer algorithm 14:28:22 anssik: issue #351 14:28:22 https://github.com/webmachinelearning/webnn/issues/351 -> Issue 351 Need to define error handling of MLNamedArrayBufferViews transfer algorithm (huningxin) 14:28:26 ... Ningxin has a very good description of this in the issue comment, not repeating it here 14:28:44 ... the proposal is for WebNN spec to define how to handle this exception and what's the impact to the MLNamedArrayBufferViews 14:28:51 ... this issue was identified as part of implementation review by Jiawei 14:29:31 ningxin_hu: we addressed the previous related issue and introduced this one 14:29:51 ... main and worker thread accessing ArrayBufferView with input and output, transfer in a loop 14:30:01 ... possible that an error happens when transferring a buffer in the middle 14:30:34 ... in a detached state, this is what the Chromium impl does today, Jiawei feel this is not ideal so we brought this to the WG for input 14:30:44 ... what is the ideal way to handle this error? 14:31:00 ... should the implementation do something with the already detached ArrayBufferViews? 14:31:44 anssik: is this a regression or clarification? 14:32:07 ningxin_hu: before the previous PR there was no transfer defined, no such issue existed before 14:32:23 q+ 14:32:29 https://github.com/webmachinelearning/webnn/issues/318 14:32:34 ... this was introduced in PR #318 14:32:35 https://github.com/webmachinelearning/webnn/issues/318 -> Issue 318 [closed] The input and output resources race condition issue of asynchronous execution (huningxin) 14:33:14 anssik: this was a TODO in the implementation? 14:33:48 ningxin_hu: I will need to check that 14:34:47 TODO in the implementation: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/ml/webnn/ml_graph_xnnpack.cc;l=440 14:35:24 q? 14:35:26 ack RafaelCintron 14:36:04 RafaelCintron: is the problem that when you detach an ArrayBuffer on the foreground thread it says sure, then in the worker when we open it we see problems and we don't know how to surface the problem to the main thread? 14:36:21 ningxin_hu: not exactly, there's a loop that iterates one ArrayBuffer at a time 14:36:52 ... the issue happens when we iterate in the middle of the loop and find the ArrayBuffer is not detachable, Wasm heap memory or mapped to WebGPU buffer that cannot be detached 14:37:00 ... this loop breaks out, report an error 14:37:26 ... there are several early ABs detached and cannot be used in the main thread and no recovery mechanism to help the current implementation 14:37:46 RafaelCintron: we can loop through everything before detaching? to make sure everything can be detached? 14:38:12 ... put a pointer to say "this cannot be detached" and only when we know everything can be detached we detach everything 14:38:45 ningxin_hu: validation step before is a good idea, that would be an atomic operation, thanks for mentioning that approach, we can put that in spec language 14:39:14 ... we want to hear the input from implementers on this issue what makes for the best design 14:39:37 ... two loops are atomic operations, want to confirm if that is possible and doable, probably we can address this issue that way 14:40:03 RafaelCintron: probably that is good, could also check with Dom for feedback, see if there are other APIs that have a similar problem 14:40:22 ningxin_hu: I will summarize what RafaelCintron suggested and put that in the issue 14:40:53 s/Dom/Domenic 14:41:23 q? 14:41:45 Subtopic: Support depth_multiplier > 1 for a depthwise conv2d op 14:41:52 anssik: issue #353 14:41:53 https://github.com/webmachinelearning/webnn/issues/353 -> Issue 353 Support `depth_multiplier > 1` for a depthwise conv2d op (Honry) 14:41:56 ... an issue from Wanming 14:42:06 ... suggests a fix to a note in conv2d op 14:42:10 -> https://www.w3.org/TR/webnn/#api-mlgraphbuilder-conv2d The conv2d() method 14:42:15 ... note now reads "A depthwise conv2d operation is a variant of grouped convolution, used in models like the MobileNet, where the options.groups = input_channels = output_channels" 14:42:36 ... suggested fix: "options.groups = input_channels = output_channels / depth_multiplier" 14:43:02 q? 14:44:12 Wanming: we found this issue when implementing TFLite WebNN delegate 14:44:32 q+ 14:44:34 q? 14:44:39 ack ningxin_hu 14:45:06 ningxin_hu: this is probably a framework compatibility issue, Wanming works on TFLite delegate 14:46:17 ... for WebNN we try to cover depthwise conv2d op, we can emulate other variants 14:47:12 ... another aspect, in implementation of XNNPACK there's depthwise conv kernel 14:48:27 ... we could have a note in the spec, this issue is motivated by frameworks 14:49:00 ningxin_hu: there's a real model that requires depth_multiplier, my open is we need a survey on implementability side, how to implement this with native ML APIs 14:49:53 Wanming: I can do this investigation 14:49:58 anssik: Thanks! 14:50:08 q? 14:50:17 Subtopic: Clarify interpolation algorithm for resample2d 14:50:27 anssik: issue #358 and related issue #270 14:50:27 https://github.com/webmachinelearning/webnn/issues/358 -> Issue 358 Please clarify interpolation algorithm for resample2d (BruceDai) enhancement 14:50:27 https://github.com/webmachinelearning/webnn/issues/270 -> Issue 270 Support coordinate transformation modes for Resample2d (Honry) 14:51:08 https://www.w3.org/TR/webnn/#enumdef-mlinterpolationmode 14:51:28 anssik: resample2d() resamples the tensor values from the source to the destination spatial dimensions according to the scaling factors 14:51:42 ... resample2d() supports two interpolation algorthms to fill the output tensor values: 14:51:48 enum MLInterpolationMode { 14:51:48 "nearest-neighbor", 14:51:48 "linear" 14:51:48 }; 14:52:00 anssik: the issue asks for clarifications to these modes on how to properly implement them 14:52:15 ... it appears there are various interpretations of these algos depending on the domain 14:52:20 ... Dwayne gives an example for nearest neighbor sampling 14:52:27 ... the banking industry uses "round halves up" 14:52:38 ... in graphics "round to nearest with X.5 halves toward negative infinity" 14:52:47 ... thoughts? 14:54:13 q? 14:54:55 [ no comments at this time ] 14:55:03 anssik: input on GH welcome 14:55:22 Topic: Support for transformers 14:56:16 anssik: Continue discuss transformers and related requirements and gaps. Test our improved contribution guidelines with these new ops: explore key use cases, sample models, cross-framework support, cross-platform implementability. 14:56:26 ... Status: The WG decided to start explore support for transformers in WebNN. 14:56:42 ... Next step: Identify use cases. Contributions welcome via issue #375. 14:56:43 https://github.com/webmachinelearning/webnn/issues/375 -> Issue 375 Mention transformer in use cases (dontcallmedom) v2 14:57:56 q+ 14:58:05 anssik: for example, is there interest to look at the gaps in Stable Diffusion? 14:58:16 ack RafaelCintron 14:58:40 RafaelCintron: I have no objection to explore this space of transformers, they are upcoming cool thing in the industry 15:00:15 q? 15:00:46 ningxin_hu: Stable Diffusion would be good one 15:01:17 ... Transformers.js is another good one it supports many HuggingFace models 15:01:17 ... Segments anything and segmentation usages we can also check out 15:02:31 q? 15:02:32 RRSAgent, draft minutes 15:02:33 I have made the request to generate https://www.w3.org/2023/05/11-webmachinelearning-minutes.html anssik