Fserb: starting presentation. Going to intro how we came to the problem, use-cases, the background of solution space, demos, open issues/high level problems. Don’t need to solve them, just present them to the group. Interested participants can join the WICG to develop further. Please interrupt if needed.
There is a link to the explainer on the WICG repo.
Anne: Louder please
Fserb: historically authors have been trying text on canvas and current support is limited to 2 APIs. filltext/strokeText, one style, one font. No support for multi-line text, have to layout yourself. Line of text with bold word: not possible. Authors have asked for this capability, result is canvas text is worse than html text.
Also limits adoption from accessibility. Even if canvas works for your use-case, text missing features limits it
What does good text mean? There’s more to it than accessibility, which also encompasses many things. Text on the web has qualities of interaction (list on the slide) which is not supported on canvas at all. Select, translate, find, autofill, spell check, canvas is just pixels on screen for the text.
Avoiding being too extensive about all sites that use canvas, but wanted to highlight some sample sites. First type is rich applications using either a full site or subset canvas. Miro is onscreen which is representative of a set of sites with that use-case. This is done by sites manually (cursor), a bunch of features don’t work. Accessibility comes in late.
Interactions in gaming with WebGL/WebGPU/WebXr.
Authors want interesting WebGL/WebGPU UXs with regular website like text. activetheory.net is an example, beautiful UI with text which is not selectable/translatable etc.
These are the use-cases we want to fix. You have a game or something with a paragraph of text, no browser first class support.
Fserb: The first requirement of the solution, we need a language to describe style/layout. Either reuse what we have (html/css) or design something bespoke. There are other advantages of allowing html/css to describe text: already supported and familiar with authors.
If it’s something bespoke, it also has to support every new feature that gets added to the web.
Forcing canvas to be a blackbox texture completely outside the normal web is not great.
Fserb: Design wise, we don’t want to use html just to describe it. Having nodes in the DOM is how interaction works. Canvas element has children but they are currently ignored, it was spec’d as fallback content. All browsers which support it ignore them, don’t layout them. So that would be a good place to have nodes which we want in canvas.
The children can be laid out same as rest of the DOM. Main difference is instead of painting them as regular, they are painted inside the canvas. What that looks like in practice.
Fserb: First mode is a high level `placeElement` API which only works on 2D. Add the children to the canvas, initially ignored. When this API is called the elements are live in the canvas. Browser is responsible for interaction and repainting them into the canvas. Once it’s there, it behaves as normal element: events work as usual. It only works on 2D, that’s where we know where the element is inside the canvas. 3D goes through texture, browser can’t know where the element is.
Second half of the API is drawElement, paints the element into a texture. You’re just drawing it, no interactivity. We have one more API which allows you specify where the element ended up on canvas which allows browser to do hit testing. Manual in that, you need to tell the browser where element ended up. Also need an invalidation event to tell the author to repaint the element. It’s a more cumbersome way than placeElement but is the only way that works for 3D. So a path which does right thing automatically, and manual mode which is lower level and supports 3D.
Fserb: showing some live demos. A basic example, shows canvas element with a child div. Code shows adding event listeners. Scaled the canvas to a size, important part is here: get the context and call placeElement.
You get a canvas with an element which has normal behaviour: shows scrolling, find-in-page. A click goes to the event listener. You can add more interesting things to it.
*shows another demo*: It’s a rain shader. Render the element into a texture with a shader on top. It’s using WebGL. This could be cheating since rain could be on top of a div.
*shows another demo*: a dragon which reflects the text.
*shows mario kart demo*: a mario kart 3D impl in WebGL. I added a texture which is html. This has a form element with css background being rendered as a part of the 3D world.
The demos are from a prototype implementation from the APIs described. Here’s another demo where text follows your cursor’s path. But now with text selection.
Fserb: There’s issues and tradeoffs to consider. Not looking for a solution, just want to highlight the problems.
- The placeElement challenge is changing the expectation of how canvas works. A browser can be in immediate mode, allocate a texture and render commands as they are called. Commands can be buffered but not required. But placeElement requires redrawing when the element changes. Re-compositing is needed.
A few impl options, store layers of texture or commands. But state will have to be kept around. Reason is placeElement does a high level thing: redraw. Otherwise author will have to redraw themselves.
On the flip side, construct mentioned that they have layering of canvases on top of each other. So they are doing it themselves. It’s difficult for the browser but eliminates author complexity.
- Second issue, especially 3D case, rendering has to be privacy preserving. 2D can taint the canvas. But 3D any shader implies that info can leak. WebGL/WebGPU requires that texture pixels don’t leak private data. Visited links, spell check, cross-origin content is example of cases which are a problem.
Fserb: Other issues which are not insurmountable but subtle. Hit-testing occlusion (from canvas commands) is a problem.
- Size of the texture is unclear.
- Which box are we talking about: border box, when paint.
- What to do with content like video/out of process iframes.
Happy with the direction and it’s worth solving these problems.
Link to the WICG in the presentation, there’s a pinned issue to indicate you’d like to participate in discussions. The demos you saw should be working in Blink soon. Questions..?
—The end!---
Questions
Ben: focus management. Will elements inside the canvas have focus.
Fserb: it’s normal html, tabing goes through the elements. The children get tabing the same way.
???: focus enables accessibility. That still works like regulat html.
Fserb: yea
???: CSS animations are supported?
Fserb: they keep re-animating. Canvas updates itself. Canvas had to change a bit for supporting the feature, rest of the html code didn’t need any changes. The elements get invalidated. drawElement will work with a raf based redraw.
???: one reason to use canvas without this feature is to improve perf where html is slow. Lot of style/relayout. Any benchmarking? If canvas has a lot of DOM elements how does that affect perf.
Fserb: The layout perf of those children is the same as regular html. So if you didn’t use canvas for that reason, like layout of a lot of nodes, it’ll be the same. But there’s more control on which elements are alive at a particular time.
One option is you have nodes but don’t care for them to be interactive all the time. You could have one element which is live, rest is stale text. So it’s faster because less interactive nodes.
Fserb: what type of layout happens for the canvas children. There’s nothing in this proposal which is inherently slower and can make it faster.
Smfr: The magic of repainting automatically is that we’ll get large display lists. Authors won’t realize they have to manage that. It just accumulates. Unless you have a reset API I don’t know how to make it work.
Fserb: Because with placeElement you have to retain info. Authors might not reset and you have an infinitely growing display list. The perf is going to be much slower.
You can collapse the display list into a texture between placeElement calls. So you’re only bound by the number of commands themselves. Right now even for clearRect, i don’t think we do a lot of optimization to detect when to ignore it. You can still do these things with placeElement. Everything before/after and it would still kinda work. But this is a problem.
Smfr: i don’t understand how clearRect approach works. If you do placeElement and then clearRect, you have to erase the placed element and the texture behind. If you clear half the canvas, causing it to be cut in half that won’t work.
Chrishtr: concern is that it interferes with being able to auto repaint.
Smfr: it’s compatible with pages which redraw every frame.
Dbaron: IIUC the place element calls has 2 effects: one is drawing the canvas. It’s acting like a drawing operation except it gets updated. Another is event handling, a later placeElement moves the element, that’s just for the event handling aspect?
Fserb: No, there’s a single element and you’re saying where it’ll be drawn.
Dbaron: So the second call makes the first one a no-op. It’s unpainting it. So you have to think of it is a display list.
Can you only call it on the children on the canvas, no descendants. Yes.
Alice: You talked about a list of elements: live and stale. Fallback content is exposed to assistive tech. This means you’re encouraging authors to think of fallback content as staging area when it’s used by assistive tech.
Fserb: wouldn’t call them staging.
Alice: Elements which are not placed or drawn. How are they affected?
Fserb: Elements which are now part of the canvas. But not placed elements will continue to behave same as today.
Alice: My concern was the elements which are not placed. How they are exposed to the accessibility tree is not obvious.
Fserb: It’s not incompatible with the proposal?
Chrishtr: My hope is this will improve accessibility outcomes. Like you said, they are given to accessibility tech but misaligned with the visuals.
Alice: The not placed elements could encourage a pattern of …
Anne: If you want to enforce that only placed elements in the fallback then allow only one.
Alice: You do want fallback content which is not placed to pass to ARIA.
Chrishtr: A mode for the canvas element which says that non-placed elements are for ARIA.
Alice: Would be interesting to puzzle out how to encourage authors beyond spec recommendation.
Cynthia: We could work on test cases.
Ben: Most common things are 30 SVG charts which are slow. Fancy text layout where text is not selectable. Get a screenshot of just one element on the page, otherwise have the element go into an SVG which is painted onto a canvas which is then screenshotted. That’s really slow so this helps.
Ydaniv: How do we interact with elements inside the shaders. Just a texture?
Fserb: WebGL flow is an overload of textImage2D. You call it and get a texture of the element’s painting. Once it’s on screen and you want interaction, you call the update transform API to pass a matrix to the browser telling it where this element is in the canvas buffer. That allows it to participate in hit testing.
Because you’re redrawing the texture every frame, it repaints and looks interactive.
Fserb: We’ll start meetings to discuss the API questions.
How does styling of the placed content works? Do we need a canvas-penetrating CSS selector?
Does hit-testing test pixels, or boxes of the rendered elements?
Is there a way to support asynchronous scrolling? Encouraging use of active wheel event handlers is not great.
Painting selection is tricky on platforms where the caret and selection exist in a different process (iOS)
What happens if you put a CSS filter or opacity on the placed element?