IRC log of webmachinelearning on 2020-04-16

Timestamps are in UTC.

13:54:24 [RRSAgent]
RRSAgent has joined #webmachinelearning
13:54:24 [RRSAgent]
logging to https://www.w3.org/2020/04/16-webmachinelearning-irc
13:54:26 [Zakim]
Zakim has joined #webmachinelearning
13:54:31 [anssik]
RRSAgent, make logs public
13:54:37 [anssik]
Meeting: WebML CG Teleconference – 16 April 2020
13:54:43 [anssik]
Chair: Anssi
13:54:51 [anssik]
Agenda: https://github.com/webmachinelearning/meetings/blob/master/telcons/2020-04-16-agenda.md
13:54:52 [anssik]
Scribe: Anssi
13:54:57 [anssik]
scribeNick: anssik
13:55:07 [anssik]
Present+ Anssi_Kostiainen
13:55:15 [anssik]
Present+ Rafael_Cintron
13:56:46 [anssik]
Present+ Greg_Whitworth
13:57:46 [RafaelCintron]
RafaelCintron has joined #webmachinelearning
13:58:17 [gregwhitworth]
scribenick: gregwhitworth
13:59:25 [Rama]
Rama has joined #webmachinelearning
13:59:37 [ningxin_hu]
ningxin_hu has joined #webmachinelearning
14:00:08 [anssik]
Present+ Ganesan_Ramalingam
14:00:13 [anssik]
Present+ Ningxin_Hu
14:00:18 [baul_eun]
baul_eun has joined #webmachinelearning
14:00:56 [anssik]
Present+ Baul_Eun
14:02:22 [Rama]
Present+ Rama
14:03:23 [Nikhil]
Nikhil has joined #webmachinelearning
14:03:35 [anssik]
Present+ Nikhil_Thorat
14:04:52 [gregwhitworth]
Topic: New ops for WebNN API
14:05:13 [gregwhitworth]
anssik: as you know we have added conf and matmul to the API
14:05:27 [daniel_smilkov]
daniel_smilkov has joined #webmachinelearning
14:05:28 [gregwhitworth]
anssik: now is the time to discuss what the next ops we think we should add
14:05:52 [gregwhitworth]
anssik: the proposal is, prioritize ops that address identified use cases
14:05:55 [anssik]
-> https://webmachinelearning.github.io/webnn/#usecases WebNN API use cases
14:06:48 [paul_mcdaniel_msft]
paul_mcdaniel_msft has joined #webmachinelearning
14:06:50 [gregwhitworth]
anssik: the space of usecases is expanding rapidly, we should also maintain these use cases that inform the API priorities to ensure we target the future and not the past
14:07:25 [gregwhitworth]
anssik: currently the usecases informantly reference papers from publicly available locations
14:07:42 [gregwhitworth]
anssik: we should continue to bring to the group's attention the latest developments in this space
14:08:00 [gregwhitworth]
anssik: with that background, let's hear proposals for new ops to add to the WebNN API
14:08:26 [gregwhitworth]
anssik: the floor is open
14:08:36 [anssik]
ack anssik
14:08:36 [Zakim]
anssik, you wanted to ask
14:09:09 [ningxin_hu]
q+
14:09:15 [anssik]
ack ningxin_hu
14:09:48 [gregwhitworth]
ningxin_hu: I think I like the approach to identify the models and usecases and we can use operators that are used in those models
14:10:09 [ningxin_hu]
https://github.com/huningxin/webnn/wiki/WebNN-first-wave-models-and-ops
14:10:11 [gregwhitworth]
ningxin_hu: with that, I recommend to do some homework and make a table with the operators for the models
14:10:54 [gregwhitworth]
ningxin_hu: on each row there is an operator and models
14:11:10 [gregwhitworth]
ningxin_hu: this does not fully cover all usecases but does cover a lot of them
14:11:58 [gregwhitworth]
ningxin_hu: add, multiply and some activations as well as pooling ops (avg pool)
14:12:09 [gregwhitworth]
ningxin_hu: there are some that we already support, like convulution
14:12:28 [anssik]
s/convulution/convolution
14:12:40 [gregwhitworth]
ningxin_hu: I left some questions open in the table as well. That's my initial try to identify the ops
14:12:45 [anssik]
q?
14:13:40 [RafaelCintron]
q+
14:13:49 [gregwhitworth]
anssik: thank you ningxin_hu, this is very helpful
14:14:12 [gregwhitworth]
anssik: I think we could add this content into the spec as informative content
14:14:47 [gregwhitworth]
ACTION: ningxin_hu submit a PR for model ops table into spec or wiki
14:15:00 [gregwhitworth]
q?
14:15:26 [gregwhitworth]
Nikhil: this is a bigger question, we've been spending quite a bit of time about what's going on with TF ops
14:15:47 [gregwhitworth]
Nikhil: is there any other standard that is going to make its way into an open body
14:16:05 [gregwhitworth]
Nikhil: I'm slowly thinking we should target a lower level of abstraction
14:16:44 [gregwhitworth]
Nikhil: we'll be chasing the ops forever and they'll be iterating at a high rate. I know this is kind of a left turn but this at about the level of XLA
14:17:08 [gregwhitworth]
Nikhil: the 80 or so ops within XLA covers the majority of neural nets they've seen
14:17:20 [gregwhitworth]
Nikhil: I'd challenge us to consider lower level abstraction
14:17:34 [gregwhitworth]
Nikhil: there are only 80 where as in TF there are 1K
14:17:51 [gregwhitworth]
Nikhil: there are tensor compute primitives and XLA, another Google one...
14:18:03 [gregwhitworth]
Nikhil: something at the same level of XLA
14:18:36 [gregwhitworth]
Nikhil: that's my current thinking, but to answer your question regarding ops. I have intuitions but I think looking at real models and consider the next steps
14:18:59 [gregwhitworth]
Nikhil: interested in hearing people's thoughts
14:19:03 [anssik]
q?
14:19:08 [anssik]
ack RafaelCintron
14:20:07 [gregwhitworth]
RafaelCintron: I acknowledge what Nikhil is saying, my worry if we get too low level we at the browser can't make them performant. But we'd have to see what they are, you say matmul and conf2d are included. What would be the other ones?
14:20:24 [anssik]
s/conf2d/conv2d
14:20:50 [gregwhitworth]
Nikhil: that's a great point, what you mentioned. I'm not saying this is 100% what we do, but I think it will be easier if we have a smaller opset
14:21:09 [gregwhitworth]
Nikhil: If we ride the wave of another standard looking at ops
14:21:45 [paul_mcdaniel_msft]
q+
14:22:03 [gregwhitworth]
Nikhil: there's the other approach is that you would never go upwards, you'd ship that lower level ops and it would go directly to x86, arm, etc. If we have to call something higher than that - that's an interesting question
14:22:08 [gregwhitworth]
q+
14:22:24 [gregwhitworth]
RafaelCintron: looking at the ops that ningxin_hu proposed are those too high level?
14:23:12 [anssik]
ack paul_mcdaniel_msft
14:23:37 [gregwhitworth]
paul_mcdaniel_msft: Nikhil one thing that may help, low level vs high level. Some scenarios we've hit in the past if we go too low you hit real issues with optimizations
14:24:16 [gregwhitworth]
paul_mcdaniel_msft: there is an lstm sitting in there but there was no way of knowing this and inferring the structure. Keras added to the TF model the information to allow for optimization
14:24:58 [gregwhitworth]
Nikhil: yeah, there are a lot of problems here and that's a valid point. There is no golden bullet. That said it is valid to have lower level, smaller opset that is pretty solid and not moving
14:25:28 [gregwhitworth]
paul_mcdaniel_msft: the ONNX group has been trying to do, is function which allows you to go down to these lower level ops while retaining the higher level concept of what it is
14:25:57 [gregwhitworth]
paul_mcdaniel_msft: ningxin_hu is talking about activation but you don't want to know its a mul and add because the GPU can do things when it knows it's an activation
14:26:14 [gregwhitworth]
daniel_smilkov: just wanted to add something to Nikhil suggestion
14:26:26 [anssik]
Present+ Daniel_Smilkov
14:26:51 [gregwhitworth]
daniel_smilkov: I don't think we want to go extremely low level, we want to find intermediate so that we don't have to add ops every 3 months
14:27:14 [gregwhitworth]
daniel_smilkov: XLA for example has LTSM information to allow for optimization on GPU/TPU, etc
14:27:16 [paul_mcdaniel_msft]
FWI: how the linux foundation is starting to craft "how to propose new ops"
14:27:18 [paul_mcdaniel_msft]
https://github.com/onnx/onnx/blob/master/docs/AddNewOp.md
14:27:52 [gregwhitworth]
daniel_smilkov: issue with high level is we'll need to keep adding ops. Whatever we standardize today, a year from now we'll have a new architecture that will have more opsets that we'll need to add. It's just the growth
14:28:01 [gregwhitworth]
daniel_smilkov: my worry is us keeping up
14:28:09 [ningxin_hu]
q+
14:28:12 [anssik]
RRSAgent, draft minutes v2
14:28:12 [RRSAgent]
I have made the request to generate https://www.w3.org/2020/04/16-webmachinelearning-minutes.html anssik
14:28:24 [gregwhitworth]
daniel_smilkov: we'll have to add custom ops if we don't which we've lost the hardware benefit
14:28:28 [gregwhitworth]
paul_mcdaniel_msft: that's a good point
14:28:51 [gregwhitworth]
paul_mcdaniel_msft: the ONNX foundation, we were worried that we'd have unchecked ops - but it's slowed down and we've started to deprecate ops
14:29:12 [gregwhitworth]
paul_mcdaniel_msft: I don't know if we're at steady state, but we were worried about that as well and I don't think we've seen that
14:29:14 [anssik]
q+ to re-ask this question from RafaelCintron: looking at the ops that ningxin_hu proposed are those too high level?
14:29:19 [paul_mcdaniel_msft]
onnx model architectures
14:29:19 [paul_mcdaniel_msft]
https://github.com/onnx/models
14:29:23 [gregwhitworth]
ack gregwhitworth
14:29:35 [gregwhitworth]
daniel_smilkov: the issue is we're trying to guess it
14:29:47 [gregwhitworth]
paul_mcdaniel_msft: I'm saying one step stronger, we're not guessing in ONNX
14:30:01 [gregwhitworth]
paul_mcdaniel_msft: we have over two dozen model arch that have fed the opsets in ONNX
14:30:15 [gregwhitworth]
paul_mcdaniel_msft: it's worth looking at as a case study
14:31:06 [gregwhitworth]
daniel_smilkov: it would be a good exercise to go back a year and half ago to understand what models wouldn't run due to missing ops in ONNX
14:31:18 [gregwhitworth]
anssik: is that something you all can look at? Or folks in ONNX
14:31:24 [gregwhitworth]
daniel_smilkov: yep we can look at that
14:31:33 [Jonathan]
Jonathan has joined #webmachinelearning
14:31:56 [gregwhitworth]
ACTION: daniel_smilkov to look at the opsets from a year ago to see what couldn't be ran on ONNX opsets
14:32:20 [gregwhitworth]
Nikhil: the goal wasn't to derail this, just to propose this thought because others at Google is considering this
14:32:36 [anssik]
q?
14:32:41 [anssik]
ack anssik
14:32:41 [Zakim]
anssik, you wanted to re-ask this question from RafaelCintron: looking at the ops that ningxin_hu proposed are those too high level?
14:33:07 [anssik]
https://github.com/huningxin/webnn/wiki/WebNN-first-wave-models-and-ops
14:33:57 [gregwhitworth]
ACTION: Nikhil and daniel_smilkov to compare ningxin_hu ops and XLA to see what, if any, are too high level
14:34:04 [RafaelCintron]
q+
14:34:13 [anssik]
ack ningxin_hu
14:34:40 [Nikhil]
XLA HLO: https://www.tensorflow.org/xla/operation_semantics
14:35:14 [gregwhitworth]
ningxin_hu: want to mention that I, the operator list, I did initial investigation into XLA. I reviewed that and there are some overlaps for example convolution and matmul
14:35:35 [gregwhitworth]
ningxin_hu: there are some other things such as batch normalization and concat, elementwise, etc
14:35:54 [paul_mcdaniel_msft]
NIkhil do you have a copy to that other lower level op you mentioned? was it "TCP"? thx !
14:35:56 [gregwhitworth]
ningxin_hu: we have something common - so my initial take is to grow that set slowly doing our due diligence by studying
14:36:12 [gregwhitworth]
ningxin_hu: so we can bring in other element wise ops
14:36:44 [gregwhitworth]
ningxin_hu: ningxin_hu will make a PR that will allow Nikhil and daniel_smilkov comment on
14:36:45 [anssik]
q?
14:36:56 [anssik]
ack RafaelCintron
14:37:33 [gregwhitworth]
RafaelCintron: just the question we have from earlier, ONNX is not in the thousands but in the hundreds and XLA seems to be similar. Maybe we can look at an intersection of those
14:37:56 [gregwhitworth]
RafaelCintron: that we can agree on that allow for hardware optimizations but not high enough where we're adding numerous ops every year
14:39:51 [gregwhitworth]
Rama: I feel that actually it is helpful to look at it as tensor and scalar ops
14:40:03 [gregwhitworth]
Rama: you can identify key categories
14:40:26 [gregwhitworth]
Rama: you end up covering a large aspect of ops
14:40:42 [gregwhitworth]
Rama: one way to limit it is to identify the op categories
14:41:09 [gregwhitworth]
Rama: I think if we do that exercise we'll find the commonalities across XLA and ONNX
14:42:24 [gregwhitworth]
Rama: the question that arises is, that if you express it this way can you reference them at a higher level?
14:42:44 [gregwhitworth]
Rama: there is value in doing that because of the existing implementations
14:43:13 [gregwhitworth]
Rama: hand crafted compilers is normally better, while there is innovation in MLIR, custom is normally better
14:43:22 [gregwhitworth]
Rama: so there is value in defining higher level ops
14:46:15 [gregwhitworth]
ACTION: paul_mcdaniel_msft to do an intersection between XLA & ONNX
14:46:56 [anssik]
RRSAgent, draft minutes v2
14:46:56 [RRSAgent]
I have made the request to generate https://www.w3.org/2020/04/16-webmachinelearning-minutes.html anssik
14:47:09 [gregwhitworth]
TOPIC: Inference API to load and run a model
14:47:35 [Jonathan]
Jonathan has joined #webmachinelearning
14:47:44 [gregwhitworth]
anssik: to quickly summarize where we are
14:48:32 [anssik]
https://github.com/webmachinelearning/model-loader
14:48:50 [gregwhitworth]
anssik: we reached consensus on incubating this idea in this group
14:49:01 [gregwhitworth]
anssik: next steps is to start drafting the actual spec text
14:49:11 [gregwhitworth]
anssik: the use cases are already covered in the existing explainer
14:49:50 [RafaelCintron]
q+
14:49:56 [gregwhitworth]
anssik: can anyone speak to prototyping? Possibly folks on the call
14:50:04 [anssik]
ack RafaelCintron
14:50:19 [gregwhitworth]
RafaelCintron: we can help with the mission of the load model API
14:50:58 [gregwhitworth]
RafaelCintron: I think the group needs to define a canonical format, we don't want to have fragmentation which is not good for web developers.
14:51:09 [gregwhitworth]
RafaelCintron: that obviously will bring us full circle to the ops
14:51:41 [gregwhitworth]
anssik: yeah that's the good long term goal but the API shape doesn't require us to have that
14:52:06 [gregwhitworth]
anssik: our current charter blocks us from making a declaration of a model format
14:52:23 [gregwhitworth]
RafaelCintron: really, we can't define a format but we can define an opset
14:52:24 [anssik]
https://webmachinelearning.github.io/charter/#out-of-scope
14:53:55 [anssik]
https://github.com/webmachinelearning/charter/issues
14:54:08 [gregwhitworth]
gregwhitworth: I recommend we do start the charter changes, but yes it shouldn't block the API
14:54:22 [gregwhitworth]
ACTION: anssik create a strawman spec for others to iterate on
14:54:52 [gregwhitworth]
TOPIC: Virtual workshop agenda building
14:55:27 [Jonathan]
Sorry, my internet is down. Catching up on meeting notes. @Anssi
14:55:29 [anssik]
-> https://bit.ly/webml-workshop-talks Virtual workshop agenda
14:55:34 [gregwhitworth]
anssik: I'd like this group to help figure out what to discuss on the virtual workshop
14:55:42 [gregwhitworth]
Jonathan: no worries :)
14:56:01 [Jonathan]
Hard to wfh with no internet and poor cell phone reception
14:56:02 [gregwhitworth]
anssik: Nikhil it would be good to have a talk around different abstractions, such as XLA, MLIR, etc
14:56:39 [gregwhitworth]
anssik: we would have offline short talks that would allow for async consumption
14:56:49 [gregwhitworth]
anssik: Nikhil & daniel_smilkov do you all think that's a good topic?
14:57:10 [anssik]
-> https://bit.ly/webml-workshop-talks Virtual workshop agenda
14:57:18 [gregwhitworth]
ACTION: Nikhil and daniel_smilkov to put together abstraction talk proposal
14:57:41 [gregwhitworth]
anssik: I'm trying to figure out which parts of the topics would make good for a virtual workshop
14:58:25 [gregwhitworth]
anssik: I'd expect us to try and cover 30-50% of the content. We would have 4 90minute sessions to discuss these topics
14:58:44 [gregwhitworth]
anssik: for example have folks from Google & XLA to answer questions and have a discussion around these topics
14:59:01 [gregwhitworth]
anssik: if you can just vote for which talks you think should be created for offline consumptions
14:59:32 [gregwhitworth]
anssik: for now just add a comment of support, if necessary I'll create a more formal voting process
15:00:04 [gregwhitworth]
anssik: we have settled on Zoom + Slack for infra
15:00:10 [anssik]
-> https://www.w3.org/Guide/meetings/zoom.html Zoom @ W3C
15:00:30 [anssik]
anssik: On that page there are tips how to mitigate some of the known security/privacy issues raised recently
15:01:03 [anssik]
anssik: Slack has been used in the past workshops with success, it is more accessible to people than IRC
15:01:32 [anssik]
... I will setup our next CG call with Zoom so we can dogfood the system and iron out any kinks
15:01:52 [gregwhitworth]
anssik: we're trending for a time in June
15:03:16 [anssik]
TOPIC: Adjourn
15:03:25 [anssik]
anssik: thanks Greg for scribing!!
15:03:30 [anssik]
RRSAgent, draft minutes v2
15:03:30 [RRSAgent]
I have made the request to generate https://www.w3.org/2020/04/16-webmachinelearning-minutes.html anssik
16:58:25 [Zakim]
Zakim has left #webmachinelearning
18:25:18 [zkis_]
zkis_ has joined #webmachinelearning
19:58:58 [zkis_]
zkis_ has joined #webmachinelearning
20:08:30 [zkis__]
zkis__ has joined #webmachinelearning