20:03:00 <RRSAgent> RRSAgent has joined #low-latency
20:03:04 <RRSAgent> logging to https://www.w3.org/2024/03/12-low-latency-irc
20:03:04 <tpac-breakout-bot> RRSAgent, do not leave
20:03:05 <tpac-breakout-bot> RRSAgent, make logs public
20:03:06 <tpac-breakout-bot> Meeting: Low-latency input events in workers
20:03:06 <tpac-breakout-bot> Chair: Marcos Caceres, Reilly Grant, Matthew Reynolds, 7ombie
20:03:06 <tpac-breakout-bot> Agenda: https://github.com/w3c/breakouts-day-2024/issues/2
20:03:06 <Zakim> Zakim has joined #low-latency
20:03:07 <tpac-breakout-bot> Zakim, clear agenda
20:03:07 <Zakim> agenda cleared
20:03:07 <tpac-breakout-bot> Zakim, agenda+ Pick a scribe
20:03:08 <Zakim> agendum 1 added
20:03:08 <tpac-breakout-bot> Zakim, agenda+ Reminders: code of conduct, health policies, recorded session policy
20:03:08 <Zakim> agendum 2 added
20:03:08 <tpac-breakout-bot> Zakim, agenda+ Goal of this session
20:03:10 <Zakim> agendum 3 added
20:03:10 <tpac-breakout-bot> Zakim, agenda+ Discussion
20:03:10 <Zakim> agendum 4 added
20:03:10 <tpac-breakout-bot> Zakim, agenda+ Next steps / where discussion continues
20:03:11 <Zakim> agendum 5 added
20:03:11 <tpac-breakout-bot> tpac-breakout-bot has left #low-latency
20:03:12 <tidoust> tidoust has joined #low-latency
20:05:18 <Norren> Norren has joined #low-latency
20:56:37 <reillyg> reillyg has joined #low-latency
20:57:14 <marcosc> marcosc has joined #low-latency
20:59:01 <marcosc> present+
20:59:35 <Norren> Greets!
20:59:41 <flackr> flackr has joined #low-latency
21:00:59 <adamscott> adamscott has joined #low-latency
21:01:01 <flackr> present+
21:01:10 <adamscott> present
21:01:25 <tidoust> present+ Francois
21:01:34 <tidoust> present+ adamscott
21:02:01 <tidoust> present+ ShawnC, GeneGirard
21:02:32 <marcosc> zakim, take up item 2
21:02:32 <Zakim> agendum 2 -- Reminders: code of conduct, health policies, recorded session policy -- taken up [from tpac-breakout-bot]
21:02:43 <Gene> Gene has joined #low-latency
21:02:57 <Gene> +present
21:02:58 <marcosc> https://www.w3.org/policies/code-of-conduct/ CEPC
21:04:06 <tidoust> scribe+ tidoust
21:04:17 <tidoust> present+ reillyg
21:04:48 <tidoust> present+ Mustaq_Ahmed
21:05:09 <girard> girard has joined #low-latency
21:05:22 <marcosc> Zakim take up item 3
21:06:06 <tidoust> zakim, take up item 3
21:06:06 <Zakim> agendum 3 -- Goal of this session -- taken up [from tpac-breakout-bot]
21:06:23 <mustaq_> mustaq_ has joined #low-latency
21:06:23 <marcosc> zakim, close item 2
21:06:23 <Zakim> agendum 2, Reminders: code of conduct, health policies, recorded session policy, closed
21:06:25 <Zakim> I see 4 items remaining on the agenda; the next one is
21:06:25 <Zakim> 1. Pick a scribe [from tpac-breakout-bot]
21:06:35 <marcosc> zakim, close item 1
21:06:35 <Zakim> agendum 1, Pick a scribe, closed
21:06:36 <Zakim> I see 3 items remaining on the agenda; the next one is
21:06:36 <Zakim> 3. Goal of this session [from tpac-breakout-bot]
21:07:21 <tidoust> present+ Matt_Reynolds
21:07:22 <marcosc> zakim, take up item 3
21:07:22 <Zakim> agendum 3 -- Goal of this session -- taken up [from tpac-breakout-bot]
21:07:53 <girard> present+
21:07:56 <tidoust> marcosc: I don't think we'll reach a conclusion on the topic but it's just to discuss whether we can route some input events (mouse, keyboard, etc.) into workers.
21:08:04 <tidoust> ... And we'll talk about why that's needed.
21:08:44 <tidoust> ... As a little segway, for folks creating multimedia, games applications, all input events are being routed to the main threads, which causes latency issues.
21:09:06 <tidoust> ... At the same time, all other things now run in workers, e.g., audio processing.
21:09:12 <tidoust> ... That creates a mismatch.
21:09:39 <tidoust> ... Devs have come up with various different ways to solve the issue. They can be hacky. postMessage, SharedArrayBuffer are the main techniques.
21:09:48 <tidoust> ... postMessage obviously also comes with latency.
21:09:58 <tidoust> ... The SharedArrayBuffer technique requires polling.
21:10:10 <tidoust> ... Wouldn't it be nice if the events could be sent directly to workers?
21:10:47 <marcosc> See https://lists.w3.org/Archives/Public/public-webapps/2023OctDec/0011.html for more context
21:10:54 <tidoust> reillyg: A piece of context that this feedback came initially in the Gamepad API. The API has this nice feature that events are not tied to the DOM per se. It feels obvious that the events could be triggered in workers.
21:11:33 <tidoust> ... Could we even send DOM events to workers as well? That's more complex. Maybe worth starting from Gamepad API events.
21:11:53 <tidoust> [round of introductions]
21:13:40 <tidoust> mustaq_: From Google. Did some exploration. Challenges include when a part of the DOM is handled by the main thread while another part is handled in a worker.
21:14:22 <tidoust> ... Explored the possibility to create forks.
21:14:51 <reillyg> q+
21:15:27 <mustaq_> present+
21:16:24 <tidoust> marcosc: For some inputs, it's really clear what the interaction with the page is. There may be an interesting distinction to be made with abstract input events.
21:16:41 <flackr> q+
21:16:48 <tidoust> ... There may or may not be overlap there, but Mustaq's perspective is interesting. I hadn't thought about that this way before.
21:17:24 <tidoust> adamscott: Work on the Godot engine. We currently use the main loop to poll and receive input events from the Gamepad API and channel them through EMScripten.
21:17:44 <tidoust> ... If we could directly have access to events in workers, that would be very nice.
21:18:12 <tidoust> mustaq_: Happy with polling? Or do you need events?
21:18:29 <marcosc> q?
21:18:29 <tidoust> adamscott: I'll take a look at the code, I don't remember on top of my head.
21:18:32 <marcosc> ack reillyg
21:19:13 <tidoust> reillyg: I wanted to mention that there is a long standing issue aroudn implementation gamepad events. Whether it's polling or events, the same problem arises on whether to expose that in workers.
21:19:15 <marcosc> q+
21:19:50 <tidoust> ... Some games have historically been built using a polling approach.
21:20:01 <tidoust> ... Web developers usually are used to handling events.
21:20:19 <tidoust> ... For this discussion, I don't think it really matters. Either way, the point is exposure to workers.
21:20:31 <marcosc> q?
21:20:38 <marcosc> ack flackr
21:20:57 <tidoust> flackr: On the Chrome interaction team as well. Events are much more power efficient when you're not in a tied loop.
21:21:03 <tidoust> s/tied/tight/
21:21:11 <reillyg> q+ to discuss Observables
21:21:34 <tidoust> flackr: Current direction in the Gamepad API implies the forking model that Mustaq was mentioning.
21:22:12 <tidoust> ... Many games support keyboard on top of gamepad. I'm very much in favor of finding a way to support this more generally.
21:22:18 <reillyg> ack marcosc
21:22:45 <tidoust> marcosc: Hearing what Adam's needs are, building a game engine, I'm wondering what should the priority be.
21:23:09 <tidoust> ... How do we want to generalize?
21:23:11 <Norren> q+
21:23:53 <tidoust> ... I don't think we'll be able to get rid of polling. But we might be able to offer an opt-in model into the polling or event-based model in the Gamepad API.
21:24:16 <tidoust> adamscott: I just checked the polling and, yes, we do use polling.
21:24:27 <marcosc> q?
21:24:36 <reillyg> ack reillyg
21:24:36 <Zakim> reillyg, you wanted to discuss Observables
21:24:43 <reillyg> -> https://github.com/WICG/observable
21:25:15 <tidoust> reillyg: Thinking about this event forking concept reminded me of observable. I don't think that's immediately a solution here.
21:25:29 <tidoust> ... But something shaped like this might work.
21:25:55 <flackr> q+
21:25:57 <tidoust> ... This would give you a context of where the event comes from. Up to the implementation to optimize that.
21:26:12 <marcosc> q?
21:26:28 <tidoust> ... The implementation could in particular push things down the stack to make sure that the fork does not block the main thread.
21:26:50 <mustaq_> q+
21:27:03 <marcosc> ack Norren
21:27:03 <tidoust> Norren: I'm the motivated person in this space who posted comment issue in the GitHub repository.
21:27:18 <tidoust> ... I've developed a lot of web pages, games.
21:27:54 <tidoust> ... The polling stuff makes a lot of sense. The event stuff makes less sense as I've seen events being dropped.
21:28:25 <tidoust> ... I don't expect to contribute a lot myself, but I can give you an amateur perspective on what's bugging me and how I would envision things work.
21:28:51 <tidoust> marcosc: What you describe looks like a terrible bug. But you make a lot of good points.
21:29:00 <marcosc> q?
21:29:15 <adamscott> q+ to discuss exposing current behavior to workers vs. invent a new API
21:29:24 <tidoust> reillyg: Maybe there are corner cases where when you lose focus and you're in keydown, you don't get keyup.
21:29:29 <marcosc> ack flackr
21:29:33 <tidoust> flackr: That's exactly what happens here.
21:29:43 <tidoust> ... It is a weirdness.
21:30:14 <tidoust> ... I wanted to respond to the observable API. One of the problems I see is that it's based on running script. By its very nature, it has to run in the main thread.
21:30:37 <tidoust> ... We could design an API which is not script-evaluated. But the way it's written there where it does some evaluation wouldn't work.
21:30:38 <Norren> Focus Loss on things like Click and drag across a canvas is also very weird.
21:30:58 <tidoust> reillyg: I agree. This is the vague shape, but we would need to rework it.
21:31:18 <marcosc> ack mustaq_
21:31:19 <tidoust> flackr: Just wanted to point that requiring access to all of the local context wouldn't work.
21:31:51 <tidoust> mustaq_: The proposal we have, on the API side, is similar to Observable without anything tied to the main thread context.
21:32:00 <mustaq_> myWorker.addEventTarget(t1);
21:32:03 <tidoust> ... You can imagine the worker becoming an observer to the event.
21:32:21 <tidoust> ... You just add an event target.
21:32:38 <mattreynolds> mattreynolds has joined #low-latency
21:32:50 <tidoust> ... The main thread creates the worker.
21:33:15 <Norren> q+
21:33:21 <tidoust> reillyg: The general concept seems good.
21:33:27 <marcosc> ack: adamscott
21:34:28 <tidoust> adamscott: Wondering whether exposing the cache of last polling from the Gamepad API could be exposed to workers.
21:34:52 <tidoust> ... If we could just expose things to start with and then generalize through a new API.
21:35:42 <tidoust> mattreynolds: Working for Google. I don't see for Gamepad why we couldn't expose things to workers.
21:35:51 <adamscott> q+ about depending on SharedArrayBuffers
21:35:57 <tidoust> ... In Chromium, that's already implemented using SharedArrayBuffer.
21:36:12 <flackr> q+
21:36:15 <adamscott> q+
21:36:26 <tidoust> marcosc: Yes, that seems fairly direct. At the same time, I don't want to lock ourselves in another corner.
21:36:35 <marcosc> ack adamscott
21:36:35 <Zakim> adamscott, you wanted to discuss exposing current behavior to workers vs. invent a new API and to
21:36:49 <tidoust> q+
21:37:31 <tidoust> adamscott: We have issues with SharedArrayBuffer because of CORS requirements. For games, lots of sites want to show ads and you cannot do that with SharedArrayBuffer. Not only about ads in practice.
21:37:42 <marcosc> ack Norren
21:37:51 <tidoust> ... Would it be possible to make this accessible through another mechanism?
21:38:24 <tidoust> Norren: Any thought on being able to set priority to threads? To give priority to tasks where latency is most felt?
21:38:48 <tidoust> ... I can only relate to the comment on CORS settings. That's a headache.
21:39:01 <marcosc> ack flackr
21:39:23 <tidoust> flackr: The Gamepad API has two events. Presumably, we wouldn't do anything to them until we figure out a suitable solution.
21:39:48 <tidoust> ... navigator.getGamepad would be what developers use not to use SharedArrayBuffer, I would assume.
21:40:05 <marcosc> q?
21:41:04 <marcosc> tidoust: six years ago I organized a game dev workshop. Navid from Google presented about latency and it may be helpful to look up what he said
21:41:30 <tidoust> https://www.w3.org/2018/12/games-workshop/report.html#latency
21:42:28 <marcosc> zakim, close item 3
21:42:28 <Zakim> I see a speaker queue remaining and respectfully decline to close this agendum, marcosc
21:42:39 <marcosc> q?
21:42:43 <marcosc> ack tidoust
21:42:50 <marcosc> Zakim, close item 3
21:42:50 <Zakim> agendum 3, Goal of this session, closed
21:42:52 <Zakim> I see 2 items remaining on the agenda; the next one is
21:42:52 <Zakim> 4. Discussion [from tpac-breakout-bot]
21:43:04 <marcosc> zakim, close item 4
21:43:04 <Zakim> agendum 4, Discussion, closed
21:43:05 <Zakim> I see 1 item remaining on the agenda:
21:43:05 <Zakim> 5. Next steps / where discussion continues [from tpac-breakout-bot]
21:43:08 <reillyg> q+
21:43:25 <marcosc> ack reillyg
21:43:41 <marcosc> zakim, take up item 5
21:43:41 <Zakim> agendum 5 -- Next steps / where discussion continues -- taken up [from tpac-breakout-bot]
21:43:43 <girard> To clarify, the proposal given from the workshop report was the same one that mustaq referenced in the issue / preread for today's session.
21:44:45 <tidoust> reillyg: [summarizes points of discussion]. Question is what working group is going to pick up the input side of things?
21:44:59 <tidoust> marcosc: I think it would be WebApps.
21:45:13 <tidoust> reillyg: Other than for Gamepad API I mean.
21:45:41 <tidoust> flackr: Technically speaking, it would be the UI Events spec, because we'll have to define the integration there.
21:45:47 <tidoust> marcosc: And UI Events falls under WebApps as well.
21:46:20 <Norren> q+
21:46:25 <tidoust> ... Taking this for the Gamepad API within the implementers team would be a good first step.
21:46:31 <flackr> q+
21:46:53 <tidoust> ... Good question for Shawn and Adam is: is that a good start? You wouldn't have the keyboard events, but you'd have the Gamepad events.
21:46:57 <marcosc> ack Norren
21:47:09 <adamscott> q+
21:47:20 <flackr> q-
21:47:42 <tidoust> Norren: Getting gamepad events is a great start. Some games are going to be mixed input by nature, e.g., analog stick in one hand and drawing on the screen with the other hand.
21:48:22 <tidoust> ... When I play a game and can either hook a controller OR mouse/keyboard often annoys me. It would be fantastic to have a general solution, of course.
21:48:58 <tidoust> ... I'll take piecemeal, but the whole thing would be great.
21:50:01 <tidoust> adamscott: If it were to happen, we could take one piece at a time, for sure. Maybe expose a raw input like handler to poll key, mouse, that you don't necessarily link to the DOM. Inputs captured by the web site directly.
21:50:18 <marcosc> q?
21:50:23 <tidoust> ... I haven't thought of possible consequences, including privacy and security.
21:50:27 <marcosc> ack adamscott
21:50:31 <tidoust> ... The more raw, the better!
21:51:03 <tidoust> Norren: It might make more sense to have some kind of key bind API. That's how many games work.
21:51:08 <reillyg> q+
21:51:12 <tidoust> ... That might help with security concerns.
21:51:25 <tidoust> ... Restricting capture to only a set of keys.
21:51:53 <tidoust> marcosc: You could key bind and the browser could prevent you from binding sensitive keys (e.g., OS-specific keys, etc.)
21:51:57 <flackr> q+
21:52:07 <tidoust> ... And then having some mechanism to release those keys.
21:52:07 <marcosc> ack reillyg
21:52:42 <tidoust> reillyg: This feels a bit like entering the Keyboard Lock territory. The proposal we were considering was more tied to the DOM model.
21:53:35 <marcosc> ack flackr
21:53:42 <tidoust> ... In the discussion with the Gamepad API, one way to solve the problem would be to redo a mouse or keyboard API that would be disconnected from the DOM, but then that would have these security issues. And I think we can avoid that with the proposal that we've been discussing.
21:54:00 <tidoust> flackr: I think that covers what I was about to say :)
21:54:21 <tidoust> reillyg: In some scenario, you do want to capture OS-specific keys, e.g., for remote desktop apps.
21:54:51 <Norren> Related but different is the bane of designing things. :)
21:55:41 <tidoust> marcosc: In terms of next steps, I think we'll continue on GitHub, in WebApps WG. Good first step is the action to get back to our implementation teams to evaluate the impact for the Gamepad API.
21:56:14 <Norren> On gamepads being used to enter passwords- think about all those TV remote based "enter password" screens where you arrow between letters and hit okay/A. XD
21:56:39 <tidoust> mustaq_: We had a WICG repository at some point. WICG/input-for-workers. It's now archived.
21:57:06 <mustaq_> https://github.com/WICG/input-for-workers
21:58:02 <tidoust> marcosc: Thanks all for joining today!
21:58:13 <marcosc> https://github.com/w3c/gamepad/issues/37
21:58:28 <tidoust> ... please follow along that issue. Cheers!
21:58:29 <tidoust> RRSAgent, draft minutes
21:58:31 <RRSAgent> I have made the request to generate https://www.w3.org/2024/03/12-low-latency-minutes.html tidoust
22:44:20 <tidoust> RRSAgent, bye
22:44:20 <RRSAgent> I see no action items