06:38:47 RRSAgent has joined #interop-2023-a11y 06:38:51 logging to https://www.w3.org/2023/09/13-interop-2023-a11y-irc 06:39:57 Ian has left #interop-2023-a11y 07:10:45 Ian has joined #interop-2023-a11y 07:10:57 RRSAgent, do not leave 07:10:59 RRSAgent, make logs public 07:11:03 Ian has left #interop-2023-a11y 10:13:56 wendyreid has joined #interop-2023-a11y 10:19:00 CharlesL has joined #interop-2023-a11y 10:19:38 present+ 10:19:42 Louay_ has joined #interop-2023-a11y 10:19:47 Fritz has joined #interop-2023-a11y 10:19:51 present+ 10:20:02 present+ Louay_Bassbouss 10:20:43 alexrudenko_ has joined #interop-2023-a11y 10:20:43 jgraham_ has joined #interop-2023-a11y 10:20:43 present+ 10:20:43 present+ Fritz_Heiden 10:20:43 present+ 10:20:43 gpellegrino has joined #interop-2023-a11y 10:20:43 jamesn has joined #interop-2023-a11y 10:20:43 Wilco has joined #interop-2023-a11y 10:20:43 present+ 10:20:43 spectranaut_ has joined #interop-2023-a11y 10:20:43 RRSAgent: make minutes 10:20:44 I have made the request to generate https://www.w3.org/2023/09/13-interop-2023-a11y-minutes.html jgraham_ 10:20:51 present+ 10:22:01 present+ 10:22:01 aaronlev has joined #interop-2023-a11y 10:22:01 atai has joined #interop-2023-a11y 10:22:01 zcorpan has joined #interop-2023-a11y 10:22:01 whimboo__ has joined #interop-2023-a11y 10:22:01 matatk has joined #interop-2023-a11y 10:22:01 jcraig has joined #interop-2023-a11y 10:22:01 scribe+ 10:22:01 present+ 10:22:01 present+ 10:22:01 present+ 10:22:01 jarhar has joined #interop-2023-a11y 10:22:01 wfilipek has joined #interop-2023-a11y 10:22:01 present+ Andreas_Tai 10:22:01 Wolfgang has joined #interop-2023-a11y 10:22:01 meeting: Cross-Browser Automated Accessibility Testing in WPT Interop 2023 and Beyond 10:22:01 Roy has joined #interop-2023-a11y 10:22:01 BrianE has joined #interop-2023-a11y 10:22:44 jcraig: Welcome to the automated testing session 10:22:44 Chair: James Craig 10:22:44 ... Simon and others kicked off a 2023 interop acessibility investigation 10:22:44 ... we've made a whole lot of progress 10:22:44 ... I want to share what we've done this year 10:22:44 ... integration into the servers and talk about the scope of what's testable 10:22:47 ... some new things that people would like to do 10:22:54 Matt_King has joined #interop-2023-a11y 10:22:55 present+ 10:22:58 ... a proposed solution we've been discussing with the browser testing and tools group 10:23:00 mrobinson has joined #interop-2023-a11y 10:23:22 ... webdriver, scheduled for later this week 10:23:22 ... any questions? 10:23:22 present+ 10:23:22 present+ 10:24:50 jcraig: I always like to start with a little bit of background 10:24:50 ... I know this is a web group 10:24:50 ... [sharing slides] 10:24:50 ... the web stack, on one side we have the web author, the other we have the end user 10:24:50 ... web APIs, html css javascript, then the webkit 10:24:50 ... the implementation inside webkit - webcore 10:24:50 ... how pixels are rendered, keyboard and mouse events 10:24:54 ... implementations of accessibility differ 10:24:57 ... change by platform 10:25:07 daniel-montalvo has joined #interop-2023-a11y 10:25:09 ... inside the rendering engine there is an accessibility runtime 10:25:20 ... it bends properties to a platform-specific context 10:25:35 ... that API is accessed by assistive technologies 10:25:35 Jean-Yves_ has joined #interop-2023-a11y 10:25:36 ... screen reader, switch control, etc 10:25:43 ... that renders the interface to the end user 10:26:26 ... and it's 2-way, the AT will send events back from the user 10:26:26 ... the web stack will test things along the way 10:26:26 ... many platforms have platform-specific automation 10:27:00 ... windows has UI Automation, there are accessibility hooks for that automation 10:27:00 ... there is client-side automation 10:27:00 ... Deque's axe-core 10:27:00 ... accessing the DOM, what is available via the DOM 10:27:07 ... slides will be shared later 10:27:18 ... every engine has it's own specific internal automation 10:27:23 ... WebKit has Layout Test 10:27:41 ... as of this spring, WebKit had 83k tests 10:27:46 ... WPT has 1.8 million tests 10:27:53 ... shared between different runtimes, engines 10:28:33 ... goal of these tests is a write-once-run-anywhere 10:28:33 ... until this year, there was very little testing related to accessibility 10:28:46 ... who here has heard of WPT, and Interop 2023 10:29:01 ... goal is to achieve interoperability and testing for it in 2023 10:29:15 ... this includes flexbox, video codecs, accessibility investigation 10:29:29 ... in april, we did some infrastructure work to hook into web driver properties 10:29:46 ... at TPAC 2019 Chrome and WebKit had implemented, we have a path forward here 10:29:51 ... let's see how much we can test 10:30:05 ... Firefox and Gecko hadn't implemented then, but has now 10:30:23 ... as of today, we have about 620 web accessibility tests running in CI builds 10:30:26 ... this is HUGE 10:30:37 ... it opens up so many different possibilities for reliability 10:31:24 ... we started testing the accessibility names spec, AAM, ARIA, we've hit about 600 tests 10:31:24 ... thousands of tests that are now unblocked, and we'll likely clear 1000 this year 10:31:30 ... we've discovered many bugs, and cleared many of them 10:31:38 ... discovered spec ambiguities 10:31:46 ... good benefit as well 10:32:08 ... one of the differences between this and other tests, there are new infinitives to writing these tests 10:32:22 ... web developers have little incentive to write browser-specific tests 10:33:00 ... it's a lot of work 10:33:00 ... learn how to do all these custom specific things that browser engineers should be doing 10:33:00 ... write-once-run-anywhere tests are more appealing, they can test issues they run into 10:34:03 ... it's using web technologies, HTML and JS, I hope that the web developers in the audience, see this and want to contribute 10:34:03 ... you're making the web better 10:34:03 ... extend testing beyond role and label 10:34:03 ... we have a team of about a dozen 10:34:03 ... we could use some help 10:34:03 ... next steps beyond what we can do this year 10:34:03 ... we need to extend our testing 10:34:17 ... beyond role and label, there are incubator issues in the AOM incubator (#197 and #203) 10:34:34 ... that's the gist of accessibility investigation this year 10:34:48 ... show you one more thing [changes shared screen] 10:34:54 https://github.com/WICG/aom/issues/197 10:35:12 https://github.com/WICG/aom/issues/203 10:35:51 jcraig: [new screen] Scoring criteria for the testing 10:36:06 ... we have a list of what we're working on [walks through list] 10:36:17 ... there are a number of role tests remaining, and app name tests 10:36:22 s/app/acc/ 10:36:37 ... if there's time at the end, we'll do SVG ARIA and DPUB ARIA 10:37:16 ... [WPT dashboard] Label:accessibility 10:37:50 ... this is the WPT dashboard showing the accessibility tests 10:37:50 ... [grid of different tests] 10:37:52 ... failures and different ones in each browsers 10:38:08 https://wpt.fyi/results/wai-aria/role?label=master&label=experimental&aligned&q=label%3Aaccessibility 10:38:19 ... [return to slides] 10:38:32 ... we'd like to extend accessibility testability beyond what we're able to currently test 10:38:53 ... 53 ARIA attributes in the editor's draft and role 10:38:58 ... how many of those have we been able to test? 10:39:01 Wilco: 3? 10:39:06 jcraig: You knew the answer 10:39:12 ... yes, 3 / 53 10:39:17 ... we have a lot more to test 10:39:18 blfeigel has joined #interop-2023-a11y 10:39:26 ... we got 600 tests out of 3 attributes 10:39:36 ... all of the reigning ARIA attributes and associated features 10:39:51 ... if there are similar host semantics, there is reconiliation of which wins 10:40:04 ... ideally we could synthesize associated accessibility actions and events 10:40:09 ... there are notable differences 10:40:16 ... there are different interoperability issues 10:40:35 ... registering to receive notifications, it would be good to test this and intercept in testing 10:40:44 ... the accessibility tree one kicked off this project 10:40:54 ... could we have an interoperable accessibility tree? 10:41:06 ... I don't think it will happen this year, but I think we can work towards it 10:41:20 ... it's a worth test interface, even if we don't end up with an interoperable tree 10:41:30 ... we can work towards testability 10:41:46 ... we can avoid posting the tests that have irrelevant details 10:42:00 ... AOM #197, this is the discussion we've been having 10:42:01 https://github.com/WICG/aom/issues/197 10:42:08 ... new test-only web API 10:42:17 ... all of the browsers agree on the following details 10:42:22 ... we need test-only web apis 10:42:40 ... we agreed that it would be too problematic to open this up to the public for use 10:42:48 ... remain test-only for the foreseeable future 10:43:12 ... within that test-only API, we need a way to get to an accessibility object and its attributes 10:43:27 ... a way to reconcile accessibility objects with elements 10:43:35 ... synthesize user events from AT 10:43:54 ... as if a user were to press something, click, focus, these events differ 10:44:01 ... if ARIA ships aria-actions, that is a different event 10:44:08 ... there is no way to test that yet 10:44:18 ... a way to register for outbound accessibility notifications 10:44:22 ... non goals 10:44:38 ... refs to accessible nodes are not expected to be writeable 10:44:46 ... we should nto be able to change its state 10:44:50 ... not expected to be live 10:44:58 ... don't think the implementability will be there 10:45:05 ... a snapshot or property back instead 10:45:22 ... refs to accessible node are not expected to be persistent after DOM changes 10:45:31 ... no relation to the previous one 10:45:47 ... tree identicality, we are not expecting tree interoperability at this point 10:46:00 ... platform-specific accessibility APIs are not in scope, yet 10:46:15 ... testing accessibility features or ATs is not part of this effort 10:46:21 ... we're not automating VoiceOver for example 10:46:40 q+ 10:46:43 ... another group has a presentation today working on something called AT Driver, there's overlap here, but we're not doing that 10:46:56 ... this portion of the presentation will be more fleshed out tomorrow, when we talk to browser testing tools 10:47:05 ... one discussion of what web driver might look like 10:47:34 Matt_King: It's not expected to be live, I was not following what you meant 10:47:58 jcraig: Thanks, I'm trying my best to avoid words like interface or method, I'm talking about these things conceptually 10:48:15 q? 10:48:16 daniel-montalvo has joined #interop-2023-a11y 10:48:19 ... sometimes when you explore a new web interface, there are refs or things you can call into 10:48:44 ... in the case of WebDriver, it's a server-client relationship where you call out to the backing web server and giving an ID and getting a property back 10:48:57 BrianE has joined #interop-2023-a11y 10:48:57 ... it's not like you could observe a change on that element, to do so you'd need to make a request back 10:49:40 ???: You have a test, you get an object, you go to the node interface, test runs, the accessibility interface may destroy the old content, and you'd need to re-request 10:49:52 Matt_King: The objects you're getting are from the accessibility tree? 10:50:03 jcraig: The browser's representation 10:50:10 ???: It's not platform specific 10:50:23 jcraig: Hidden and shown again, a completely new identifier 10:50:34 Matt_King: You can't rely on the unseen behaviour of the layer 10:50:44 ???: You can't rely on the underlying DOM 10:50:54 ack CharlesL 10:50:56 q+ 10:51:14 s/???/aaronlev/g 10:51:41 CharlesL: On the last slide, one of the points mentioned platform specific was not going to be test 10:51:49 ??? = Aaron Leventhal 10:51:52 ... a chrome implementation, you're not testing windows vs mac 10:51:59 ... would the tests really show that? 10:52:23 ... for role, you're expecting doc-chapter, depending on mac or windows, those results would be identifiable 10:52:30 ... just trying to understand why that would not work? 10:52:50 jcraig: I'll go back to the slide on the web automation stack 10:53:41 ... four grey boxes, different types of automation, platform-specific (Mac AX roles), client-side automation (axe-core), in the browser internals it would be the same for both 10:53:52 ... the only standard we have is what is the computed ARIA role for this element 10:54:00 ... does this element have a role of heading 10:54:13 ... but it's different from an AX role of heading or a Windows role of heading 10:54:25 ... we're testing the web specific portion of the stack, not the platform on 10:54:29 s/on/one/ 10:54:47 ... here is a specific web test, WPT can't do this kind of platform testing well 10:54:58 ????: We try to avoid that kind of thing 10:55:19 jcraig: The CI implementation on that is a linux machine, so it's not even running those platform APIs 10:55:27 Wilco: I had 2 questions 10:55:35 s/????/jgraham/ 10:55:58 ... this sounds to me like [?] has with (sorry I missed this) 10:56:10 ... have you thought about what ARIA is doing with pulling language or table structure? 10:56:20 jcraig: I hadn't personally thought of the lang attributes 10:56:29 ... I know VO will switch languages 10:56:36 s/????: We/jgraham: We/ 10:56:40 q+ 10:56:49 ... it's doing that because there is a WPT for it 10:57:22 ... anything with reference to accessibility, and 10:57:41 ... repeating the first question, CP, the Chrome Dev Tools, James Graham, does that align? 10:57:51 s/CP/CDP/ 10:57:52 ... this particular section about the web driver potential extension 10:58:02 ... I don't know as much about it as the others 10:58:13 ... one recommendation was to put this on the bidi roadmap 10:58:28 jgraham_: I can answer but it might not be worth taking up time 10:58:33 jcraig: We can discuss tomorro 10:58:59 Simon: Web Driver bidi, (missed) 10:59:14 Simon: adding to the language question, the content language, set with the lang attribute 10:59:18 ... it's already testable 10:59:26 ... it can be tested in WPT today 10:59:48 jcraig: Browser internal accessibility it's going to be platform specific 11:00:00 ... you can test it in the browser, but should we test it in accessibility 11:00:09 [cross-chatter] 11:00:31 %%%: There are bugs and ambiguities to test 11:00:34 ack zcorpan 11:00:43 s/%%%/jamesn/ 11:00:45 jcraig: This will be more fleshed out in the webdriver session tomorrow 11:00:58 ... we need either a new method for specific web driver extension 11:01:15 ack wilco 11:01:24 jcraig: Potentially be a web opt-in extension 11:01:40 ... because there is so much in accessibility, we don't want to ask web driver to update piecemeal 11:01:55 ... we need a method on the existing web driver interface to get a node 11:02:10 ... this is a way to reconcile if you have an accessibility object 11:02:24 ... if there is a relationship between an object and node we need to get at it 11:02:39 ... ways to get an web driver ID from a node 11:03:04 ... this is a potential sample JSON object [slide] 11:03:10 ... label, role, parent 11:03:13 ... ids for each 11:03:19 ... checkstate, requiredstate 11:03:29 ... way to synthesize an event on an accessibility node 11:03:47 ... minimum payload would be the notification type 11:03:54 q+ 11:03:54 ... focus, press, actions 11:04:03 ... open questions 11:04:19 ... why return property bag instead of calls per property 11:04:32 ... there could be a lot of property values, especially for some tests 11:04:41 ... would fit closer with current web driver interface 11:04:58 ... are accessibility id refs implementable in an interoperable manner? 11:05:12 ... depending on the longevity of the references, some aren't possible yet? 11:05:33 ... what about testing accessibility of related native features like `inert`? 11:05:40 ... good questions everyone 11:05:42 q? 11:05:56 ack jgraham_ 11:06:20 jgraham_: Make a few comments, first one is that if you have time for discussion, let's make it happen? 11:06:27 [scheduling] 11:07:34 jgraham_: Of the details, you mentioned bidi, a lot of this is a fit for bidi, for the DOM tree, we more accurately represent that in bidi 11:07:34 ... it becomes complicated to work with 11:07:35 ... I will definitely think about this, classic web driver will be the way to get this working in all browsers 11:07:53 ... its' also worth thinking about what you actually want this to look like, then determine protoals 11:08:02 ... looks at existing [?] actions 11:08:22 ... it moves from point to point, presses, it might fit with that instead of a new endpoint? 11:08:33 jcraig: It's critical we go through the accessibility node 11:08:51 jgraham_: If you want it to be this source, performs this action, it can represent that 11:09:03 jcraig: For bidi vs web driver, I'll do what Sam things is best 11:09:22 jgraham_: Do what's best for browsers, what worries me, if we have this stuff, web authors may use it for their own tests 11:09:35 ... but not accessibility, it gives them information that's hard to get other ways 11:09:53 ... having things like "is this completely implemented, defined", it could be a risk 11:10:10 ... be more specific about what the integration is or browser specific 11:10:16 jcraig: I think we're align on what we want 11:10:23 ... webkit has an intermediary accessibility node 11:10:39 ... differences would be tree relationships, different parent, but can still get into ut 11:10:56 ... if a web author hits that, it may need notes in the TODOs, they can fix their tests 11:11:02 jgraham_: [missed this] 11:11:08 q? 11:11:20 jcraig: Anyone jones-ing to write accessibility WPT tests? 11:11:24 [hands!] 11:11:25 s/[missed this]/it's definitely difficult if it's different tree structures/ 11:11:42 Wilco: Question, accessibility conformance testing taskforce 11:11:59 ... writes a lot of rules that are picked up in testing, we're having a hard time to track what works where 11:12:10 ... this will help us figure that out, and we can help with gaps 11:12:19 jcraig: The arrangement of testing is by spec 11:12:27 ... line-by-line we go through the requirements 11:12:37 ... it would be good to have tests for WCAg 11:12:50 ... the tests are similar if structured differently 11:13:00 ... ARIA vs WCAG, we say a lot of bugs in testing 11:13:06 bfeigel has joined #interop-2023-a11y 11:13:06 Wilco: I believe that! 11:13:13 CharlesL has left #interop-2023-a11y 11:13:16 jcraig: Thanks everyone for your time, email me! 11:13:28 ... contact me via the W3C slack too 11:13:45 present- 11:13:54 rrsagent, please draft minutes 11:13:56 I have made the request to generate https://www.w3.org/2023/09/13-interop-2023-a11y-minutes.html wendyreid 11:30:19 Slides: https://www.icloud.com/keynote/0eciRqzy6aGWyffs7OXZbkz8w#AccessibilityAutomation_2023SeptTPAC 11:30:38 Roy has joined #interop-2023-a11y 11:30:49 rrsagent, make minutes 11:30:50 I have made the request to generate https://www.w3.org/2023/09/13-interop-2023-a11y-minutes.html jcraig 11:34:04 s/doing that because there is a WPT for it/doing that because there is an attribute for it in the platform accessibility API/ 11:34:21 s/acessibility/accessibility/g 11:35:54 s/a proposed solution we've been discussing with the browser testing /a potential solution we'll be discussing tomorrow with the browser testing / 11:36:44 s/web group/web group so many of you know this background already/ 11:37:36 matatk has joined #interop-2023-a11y 11:38:32 s/ javascript, then the webkit/ javascript, then the rendering engine's implementation/ 11:38:58 s/the implementation inside webkit/in Safari's case, the implementation inside webkit/ 11:39:35 s/bends properties/vends properties/ 11:40:40 s/windows has UI Automation/both Windows and Mac have UIAutomation (though different ones)/ 11:41:06 s/Deque's axe-core/such as Deque's axe-core/ 11:42:14 s/video codecs, accessibility investigation/video codecs, inert, pointer events, accessibility investigation/ 11:43:32 s/Gecko hadn't implemented then/Gecko hadn't implemented the webdriver interface in April, but did in May/ 11:44:19 s/about 620 web accessibility tests running in CI builds/620 web accessibility tests running in the CI environments of every major browser engine on the planet/ 11:45:26 s/accessibility names spec, AAM, ARIA,/accessibility names spec (AccName), HTML-AAM, ARIA,/ 11:45:58 s/new infinitives to writing these tests/new incentives to writing these tests/ 11:48:07 s/we have a team of about a dozen/we have a team of about a dozen from all over: Bocoup, Igalia, Apple, Adobe, Google, Salesforce, etc/ 11:48:33 matatk has joined #interop-2023-a11y 11:49:18 s/reigning ARIA attributes/remaining ARIA attributes/ 11:50:03 s/could we have an interoperable accessibility tree?/People wondered: could we have an interoperable accessibility tree in 2023?/ 11:50:27 s/I don't think it will happen this year,/It won't happen this year,/ 11:51:04 s/worth test interface/worthy test interface/ 11:51:55 s/browsers agree on the following details/WPT Accessibility stakeholders agree on the following details/ 11:53:18 s/not expected to be live/not expected to be live (probably a snapshot object aka "property bag")/ 11:53:37 s/property back instead/property bag instead/ 11:55:03 s/there's overlap here, but we're not doing that/there's overlap with some of the people working on the other one, but it's not in scope for this effort/ 11:57:27 s/The objects you're getting are from the accessibility tree?/The objects you're getting are from the platform accessibility tree?/ 11:58:14 s/The browser's representation/No, it's getting the browser's internal representation of the tree, before it makes it out to the platform./ 11:59:32 matatk has joined #interop-2023-a11y 11:59:56 s/Hidden and shown again, a completely new identifier/another example, if you were to hide an element and show it again, the backing accessibility node would likely have a completely new identifier, and bear no relationship to the previous accessibility node that was exposed./ 12:00:39 Roy has joined #interop-2023-a11y 12:02:04 s/AX role of heading or a Windows role of heading/AXRole of AXHeading on Mac, or a Windows role of heading in the windows-specific platform accessibility API/ 12:02:33 s/We try to avoid that kind of thing/WPT tries to avoid that kind of thing/ 12:02:47 rrsagent, make minutes 12:02:48 I have made the request to generate https://www.w3.org/2023/09/13-interop-2023-a11y-minutes.html jcraig 12:04:10 s/we're testing the web specific portione of the stack, not the platform on/in this case it's testing the web-engine-specific portion of the stack, not the platform one/ 12:05:01 s/anything with reference to accessibility/so anything with relevance to accessibility should be tested here, not just the ARIA stuff/ 12:05:39 s/I don't know as much about it as the others/I don't know as much about it as the others, maybe James Graham or Sam Sneddon?/ 12:10:12 s/web opt-in extension/web driver extension/