14:49:44 <RRSAgent> RRSAgent has joined #webmachinelearning
14:49:44 <RRSAgent> logging to https://www.w3.org/2022/02/24-webmachinelearning-irc
14:49:47 <Zakim> RRSAgent, make logs Public
14:49:48 <Zakim> please title this meeting ("meeting: ..."), anssik
14:49:54 <anssik> Meeting: WebML WG Teleconference – 24 February 2022
14:49:59 <anssik> Chair: Anssi
14:50:06 <anssik> Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2022-02-24-wg-agenda.md
14:50:12 <anssik> Scribe: Anssi
14:50:17 <anssik> scribeNick: anssik
14:51:32 <anssik> Present+ Anssi_Kostiainen
14:51:38 <anssik> RRSAgent, draft minutes
14:51:38 <RRSAgent> I have made the request to generate https://www.w3.org/2022/02/24-webmachinelearning-minutes.html anssik
14:59:47 <ningxin_hu> ningxin_hu has joined #webmachinelearning
15:00:49 <dom> Present+
15:00:51 <dom> Present+ ningxin_hu
15:01:36 <anssik> Present+ Ganesan_Ramalingam
15:01:52 <anssik> Present+ James_Fletcher
15:01:59 <anssik> Present+ Chai_Chaoweeraprasit
15:02:10 <chai> chai has joined #webmachinelearning
15:02:44 <anssik> Present+ Jonathan_Bingham
15:02:56 <dom> scribe+
15:02:58 <anssik> Topic: Announcements
15:03:10 <Jonathan> Jonathan has joined #webmachinelearning
15:03:12 <anssik> scribe+ Jonathan
15:03:28 <anssik> Present+ Rafael_Cintron
15:03:32 <anssik> Subtopic: Survey about TPAC 2022
15:03:33 <Jonathan> i'm getting coffee, so i'll let Dom/Anssi start with the scribing
15:03:45 <bbcjames> bbcjames has joined #webmachinelearning
15:04:01 <dom> anssi: tpac 2022 plans - a few weeks a survey was sent to the chairs to figure out what we would like to do for the next TPAC
15:04:12 <dom> ... tentatively scheduled Sep 12-16 in Vancouver, as a hybrid meetnig
15:04:15 <dom> s/nig/ing/
15:04:35 <dom> ... lots of uncertainty on whether that meeting can be held in person vs a fully virtual meeting as the previous 2 years
15:05:11 <dom> ... If you feel like sharing (possibly privately) about your ability to be in Vancouver or lack thereof, this would help me to inform W3C TPAC organizers
15:05:18 <RafaelCintron> RafaelCintron has joined #webmachinelearning
15:05:19 <dom> ... ideally before our next meeting
15:05:54 <dom> ... my observations on this: this WG hasn't met in person yet, but we've been executing above my expectations given the environment - thank you everyone
15:06:02 <dom> ... we had a short CG F2F back in 2019
15:06:20 <dom> ... that's our only physical meeting experience
15:06:41 <dom> ... TPAC is useful for x-group interactions and making informal connections, which can help resolve challenging issues
15:07:04 <rama> rama has joined #webmachinelearning
15:07:52 <anssik> dom: none of us can predict the future, any information on personal willingness, company policies is welcome
15:08:27 <anssik> Subtopic: Update on Ethical Considerations
15:08:35 <dom> anssi: an update from James on ethical considerations
15:09:01 <dom> James: a reminder on the process I proposed in December for our work on ethical considerations:
15:09:20 <dom> ... first a literature review, consultation with experts and stakeholders, leading to a set of guiding principles
15:09:48 <dom> ... that could be used as a draft note, and then a set of discussions where these principles would be reviewed & brainstormed on
15:10:05 <anssik> Present+ Raviraj_Pinnamaraju
15:10:07 <dom> ... to identify high level risks & mitigations
15:10:15 <anssik> RRSAgent, draft minutes
15:10:15 <RRSAgent> I have made the request to generate https://www.w3.org/2022/02/24-webmachinelearning-minutes.html anssik
15:10:21 <dom> ... integrate the outcomes of the sessions into a note that would be presented to the group
15:10:47 <dom> ... literature review is ongoing, we'll have a draft of guiding principles available next week on github
15:11:02 <dom> ... the review & brainstorm sessions would be end of March or early April
15:11:28 <dom> ... we could have the updated draft WG note around April 21st
15:12:03 <dom> anssi: I'll work with you & Dom in circulating the invitation to these sessions
15:12:12 <dom> ... eager as others to see drafts!
15:12:14 <anssik> q?
15:12:36 <dom> ... Thanks James! more on this soon
15:12:42 <anssik> Topic: Security review response
15:13:17 <anssik> -> Update Security Considerations per review feedback https://github.com/webmachinelearning/webnn/pull/251
15:13:27 <dom> anssi: thanks to everyone who provided input & feedback
15:13:38 <anssik> -> PR Preview: Guidelines for new operations https://pr-preview.s3.amazonaws.com/webmachinelearning/webnn/pull/251.html#security-new-ops
15:13:43 <dom> ... I've added a commit to this pull request with guidelines for new operations
15:13:57 <dom> ... incorporating also input from the ONNX TPAC session
15:14:36 <anssik> -> General Security Questions https://github.com/webmachinelearning/webnn/issues/241
15:14:44 <anssik> Subtopic: Notion that the API introduces a new scripting language
15:14:57 <anssik> anssik: WG asked: "The group did not agree with the notion that the API introduces a new scripting language and wants to understand what aspects of the API are of concern?"
15:15:12 <anssik> ... Alex responded:
15:15:17 <anssik> ... "My security concern is that the network is compiled into a program that is entirely under an attackers control. While not as powerful as javascript it is likely powerful enough to make some exploits easier as a result. It argues for careful implementation, and avoiding introducing too much control of control-flow at the API side. Hopefully these concerns can be considered as new operations are introduced."
15:15:45 <anssik> anssik: can we formulate this into an informative note for security considerations?
15:17:10 <anssik> dom: briefly looked at this, not still fully clear what would typically mitigate this problem?
15:17:29 <anssik> ... conceptually understand where the reviewer comes from with expressiveness of the API and concerns around that
15:17:31 <anssik> q?
15:17:55 <anssik> ... the reviewer seems to want some additional information to think about the security properties of the API
15:18:56 <dom> chai: we takes security very seriously in my team working on Windows; when we go through security review, we start by identifying the threats that the system may be exposed to
15:19:14 <dom> ... that includes understanding what data comes in & out the system, and whether it should be treated as trusted or untrusted
15:19:53 <dom> ... e.g. a PDF displayed in a browser is considered untrusted because the attacker may be in control of the content to exploit vulnerabilities
15:20:19 <dom> ... in the context of WebNN API, a legitimate question is whether any data that can be produced by someone in your browser through WebNN can be considered untrusted
15:21:08 <anssik> Present+ Rachel_Yager
15:21:33 <dom> ... another question is whether a format that would feed into WebNN is vulnerable to attacks - e.g. does it give ability to change boundary checks that can lead e.g. to buffer overruns
15:21:50 <dom> ... In practice, the only time you attack the system is when you make it _execute_ something
15:22:10 <dom> ... WebNN, as I clarified in #243, is not about executing the data - it's about defining the data
15:22:36 <dom> ... the graph builder interface doesn't execute anything, it constructs a piece of data
15:22:51 <dom> ... the attack is limited to when you bind the data to the graph before your execute it in the compute method
15:23:37 <dom> ... assuming an untrusted ML model, the question is how to harden the compute method
15:24:32 <dom> ... the key is ensuring that compute honors the boundary of data
15:24:58 <dom> ... where compute needs to fail if these boundaries aren't respected
15:25:23 <dom> ... That's how I look at security in general, not sure how it maps to the feedback we got
15:25:33 <dom> anssi: this could be part of our security considerations section
15:25:33 <dom> q+
15:26:28 <dom> ... part of our goal is to ensure anyone evaluating the spec can determine the answer to that kind of questions we're getting
15:27:05 <anssik> -> General Security Questions https://github.com/webmachinelearning/webnn/issues/241
15:27:36 <anssik> q?
15:28:37 <anssik> ack dom
15:29:20 <anssik> dom: building on what was said, two things we want to do: 1) provide all the information we can to implementers so they avoid security bugs 2) surface information that help security reviewers make assessment on the risks
15:30:49 <anssik> ... we should explain how the API is sufficiently hardened, principles behind the design should be explicit in the spec so reviewers and implementers can feel confident
15:31:11 <anssik> ... Anssi's proposal to add Chai's model in the Security Considerations would be great
15:32:29 <anssik> -> Update Security Considerations per review feedback PR #251 https://github.com/webmachinelearning/webnn/pull/251
15:32:54 <anssik> q?
15:32:57 <dom> anssi: this PR is where we hope to address these issues - providing feedback on it would be very useful
15:33:05 <anssik> Subtopic: Operation level metadata
15:33:10 <anssik> anssik: WG asked: "The group was not sure what "operation level metadata" exactly means in this context? (Would addressing #243 satisfy all "operation level metadata" requirements?)"
15:33:26 <anssik> ... Alex responded: "Yes, that would be a helpful step."
15:33:34 <anssik> -> Op metadata that helps avoid implementation mistakes https://github.com/webmachinelearning/webnn/issues/243
15:33:49 <anssik> anssik: I notice Chai clarified in #243 the MLGraphBuilder API isn't a data execution API, but rather a data definition API. Thanks for that input.
15:33:59 <dom> q+
15:34:02 <anssik> ack dom
15:34:40 <anssik> dom: in terms of data definition vs. model execution -- what errors implementers must identify and fail on for each?
15:34:47 <chai> q+
15:35:09 <anssik> ... I think making this distinction is important, but the question is when we compile or compute when we need to fail, can we be explicit when?
15:35:10 <anssik> ack chai
15:36:00 <dom> chai: adding the buffer-metadata to the description of operations feels redundant
15:36:29 <dom> ... eventually, you'll compare the size of the data you receive with the one you're expecting to receive, you fail
15:36:43 <dom> ... graph builder defines the size of data you expect to receive
15:37:03 <dom> ... the size of each buffer within each location within the graph is already defined
15:37:36 <dom> ... the problem of lack of metadata; the implementation will have to walk the graph to calculate the shape of the data
15:37:48 <dom> q+
15:38:01 <dom> chai: the important part is when you call compute
15:38:05 <anssik> q?
15:38:21 <ningxin_hu> q+
15:38:22 <anssik> ack dom
15:39:28 <anssik> dom: if you're doing security analysis of a spec, having an easy way to figure out what op creates a risk for computation because the shape may change between input and output, that may make it easier to write your attack test suite to verify you haven't missed something
15:39:37 <anssik> ... not sure if metadata approach is a practical one
15:39:55 <anssik> ... trying to say, not just guiding implementers only, but guiding reviewers
15:40:23 <anssik> q?
15:40:50 <dom> chai: the graph builder approach is actually a very robust approach to harden this problem space
15:41:43 <dom> ... because you don't specific the size of the output of the ahead of time, instead based on the operation semantics
15:42:04 <dom> ... the output size is not in control of the attacker with such a model
15:42:56 <dom> ... this allows to calculate output size at every single step of the graph
15:43:07 <dom> ... adding metadata is a security hole
15:43:07 <dom> q+
15:43:11 <anssik> q?
15:43:14 <dom> qq+
15:43:22 <anssik> ack dom
15:43:22 <Zakim> dom, you wanted to react to dom
15:44:15 <anssik> dom: we are not required for data itself to provide metadata
15:44:33 <anssik> chai: in that case read the definition of each op
15:44:55 <anssik> dom: not needing to dive into each op algorithm to understand if boundary risks might be created
15:45:58 <anssik> q?
15:46:01 <dom> chai: for simple operations, it's trivial, for complex operations like conv2d or recurring networks, you can't simplify it
15:46:04 <anssik> ack ningxin_hu
15:46:04 <dom> q-
15:46:22 <anssik> Present+ Wang_Xiaojian
15:46:36 <dom> ningxin_hu: for conv2d, we have a formula to calculate the output size based on the parameters of the operations
15:46:56 <dom> ... would this be useful to the reviewers, or for machine processing?
15:47:56 <anssik> q?
15:48:41 <anssik> dom: perhaps highlighting such formulas in the spec would work?
15:49:05 <anssik> Subtopic: Running timeable things out of process
15:49:18 <anssik> anssik: WG asked: "The group would like to understand how running timeable things out of process works as a mitigation?"
15:49:24 <anssik> ... Alex responded: "This is a concern of lower importance - mainly adding an ipc step will confuse any accidental high precision timers. Realistically there is not a lot to be done here!"
15:49:55 <anssik> q?
15:50:04 <RafaelCintron> q+
15:50:16 <anssik> dom: IPC step mitigates the issue, but is it guaranteed to happen in all implementations?
15:50:20 <anssik> ack RafaelCintron
15:50:49 <anssik> RafaelCintron: I agree with Dom, we should not add IPC as a mitigation
15:51:42 <anssik> dom: my suggestion is to mention the problem and mention the same mitigations as with high precision timing might apply here
15:52:09 <anssik> q?
15:52:20 <anssik> Subtopic: Next step: Wide review for security
15:52:37 <anssik> -> Wide review tracker https://github.com/webmachinelearning/webnn/issues/239
15:53:10 <dom> -> https://w3c.github.io/hr-time/#sec-security Security Considerations of "High Resolution Time" spec
15:53:12 <anssik> -> Update Security Considerations per review feedback PR #251 https://github.com/webmachinelearning/webnn/pull/251
15:53:19 <anssik> -> Responses to the Self-Review Questionnaire: Security and Privacy https://github.com/webmachinelearning/webnn/issues/119
15:53:48 <anssik> -> All security-tracker issues https://github.com/webmachinelearning/webnn/issues?q=label%3Asecurity-tracker+
15:54:28 <dom> anssi: we have ongoing discussions on all issues except for #175
15:54:47 <anssik> -> WebGPU Security Considerations https://gpuweb.github.io/gpuweb/#security-considerations
15:55:11 <dom> ... the WebGPU WG has done quite a good job on this topic, we should re-use what we can from them
15:55:36 <dom> Rafael: I know the WebGPU has gone through Google internal security review, and is about to go through the W3C review
15:55:51 <dom> ... it has been under TAG review for quite some time
15:56:16 <dom> anssi: overall, it feels we're on good track to resolve these security issues
15:56:36 <anssik> q?
15:56:38 <dom> ... once the pull request lands and hopefully addresses the issues, we can start the W3C security review
15:56:48 <anssik> Topic: Candidate Recommendation technical scope
15:57:19 <anssik> Subtopic: The baseline implementation of WebNN ops [cr]
15:57:23 <anssik> -> The baseline implementation of WebNN ops https://github.com/webmachinelearning/webnn/issues/245
15:57:38 <anssik> -> WebML WG Teleconference - 10 February 2022 resolutions https://www.w3.org/2022/02/10-webmachinelearning-minutes.html#ResolutionSummary
15:57:47 <anssik> ... review in progress, anything to report Ningxin?
15:57:56 <anssik> q?
15:58:17 <dom> ningxin_hu: we're waiting for a review from chai & his team on the conv2d implementation
15:58:21 <anssik> Subtopic: Should WebNN support async APIs? [cr]
15:58:28 <anssik> -> Should WebNN support async APIs? https://github.com/webmachinelearning/webnn/issues/230
15:59:06 <dom> chai: I'm working on a PR for this
15:59:21 <dom> anssi: thanks, very important work
15:59:22 <anssik> Subtopic: Should restrict the sync APIs to only exist in Workers? [cr]
15:59:32 <anssik> -> Should restrict the sync APIs to only exist in Workers? https://github.com/webmachinelearning/webnn/issues/229
16:00:06 <anssik> anssik: on our 13 January 2022 call we decided to request feedback from ML framework authors, how this proposed change would impact the frameworks, in particular their Wasm backends.
16:00:09 <anssik> ... no responses received
16:00:13 <anssik> ... any new information to be shared?
16:00:18 <anssik> ... are we still happy to keep this issue in scope for CR?
16:00:21 <anssik> -> https://www.w3.org/2022/01/13-webmachinelearning-minutes.html#t03 WebML WG Teleconference – 13 January 2022
16:00:30 <anssik> q?
16:00:37 <anssik> Subtopic: Integration with real-time video processing [cr]
16:00:41 <anssik> -> Integration with real-time video processing https://github.com/webmachinelearning/webnn/issues/226
16:00:55 <anssik> anssik: it seems the outstanding topic is the support for async API and interaction with GPU timeline, these changes will have an API shape impact
16:01:05 <anssik> ... is it fair to say this issue #226 is blocked on issue "Should WebNN support async APIs? #230"
16:01:11 <anssik> -> Should WebNN support async APIs? #230 https://github.com/webmachinelearning/webnn/issues/230
16:01:20 <anssik> anssik: Ningxin, do you want to give an update on the Chromium prototype for WebNN/WebGU interop?
16:01:28 <anssik> -> [Chromium Prototype] WebNN / WebGPU interop #929 https://github.com/otcshare/webnn-native/issues/929
16:01:57 <anssik> q?
16:02:38 <anssik> ningxin_hu: formulating a plan for the prototyping assignment, working on that along with spec discussion
16:02:39 <anssik> q?
16:02:44 <anssik> Subtopic: Add method steps to operations [cr]
16:02:46 <chai> i should have a PR out for #230 soon.
16:02:54 <anssik> -> Add method steps to operations https://github.com/webmachinelearning/webnn/issues/210
16:03:07 <anssik> anssik: this is just work that needs to happen, any questions to be recorded in the issue
16:04:10 <anssik> q?
16:04:40 <anssik> anssik: Thanks everyone for joining and for your contributions!
16:04:45 <anssik> RRSAgent, draft minutes
16:04:45 <RRSAgent> I have made the request to generate https://www.w3.org/2022/02/24-webmachinelearning-minutes.html anssik