16:02:41 <RRSAgent> RRSAgent has joined #aria-at
16:02:41 <RRSAgent> logging to https://www.w3.org/2019/09/25-aria-at-irc
16:02:49 <Jean-Francois_Hector> scribe: Jean-Francois_Hector
16:02:58 <Zakim> Zakim has joined #aria-at
16:03:12 <Matt_King> rrsagent, make log public
16:03:34 <Matt_King> MEETING: ARIA and Assistive Technologies Community Group for Sep 25, 2019
16:03:44 <Matt_King> CHAIR: Matt King
16:03:47 <Matt_King> present+
16:03:47 <spectranaut> present+
16:03:48 <jongund> jongund has joined #aria-at
16:03:52 <Matt_King> rrsagent, make minutes
16:03:52 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/25-aria-at-minutes.html Matt_King
16:03:59 <jongund> I will be a few minutes late
16:05:25 <Matt_King> Matt_King has joined #aria-at
16:05:48 <Matt_King> rrsagent, make minutes
16:05:48 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/25-aria-at-minutes.html Matt_King
16:08:11 <Jean-Francois_Hector> TOPIC: Test case management systems
16:08:55 <Jean-Francois_Hector> V: Started a wiki page to summarise findings so far. Documented 2 options, can talk about the 3rd one
16:08:57 <spectranaut> https://github.com/w3c/aria-at/wiki/Test-Case-Management-System-Research
16:10:42 <Jean-Francois_Hector> Looked at 3 options, including a test harness maintained by CSS Working Group.
16:11:50 <Jean-Francois_Hector> Kiwi and TestLink are written in the paradigm that they're testing a product. Both have lots of users. They expect that the tests will be written in the application.
16:12:37 <Jean-Francois_Hector> CSS Working Group Test harness is much simpler. Tests are all written in HTML files that get loaded in an iFrame.
16:12:57 <Jean-Francois_Hector> Kiwi has a hosted version of Kiwi for people to look at and play around with.
16:13:09 <Jean-Francois_Hector> The CSS WG Test harness is open to the public.
16:14:10 <Jean-Francois_Hector> What's a good way to test these test harnesses for accessibility?
16:16:11 <Jean-Francois_Hector> MCK: I don't know how far we want to go with that. These are open source. So we might give them a sniff test: is it a disaster or is it workable? The test harness should be practical for screen reader users to use. But if something is reasonably successful but includes some mistakes we could fork it or improve it.
16:17:24 <Jean-Francois_Hector> JG: Kiwi's pull down menus don't have ARIA, but they do seem to support the keyboard (open and close menu, tab to features).
16:18:02 <Jean-Francois_Hector> JG: Kiwi not using landmarks. Might be easy to get them to add those things.
16:18:31 <Jean-Francois_Hector> V: You have to create a test plan and page before you can get to the page where you can run tests, which is probably the most important to test.
16:19:29 <spectranaut> https://public.tenant.kiwitcms.org/plan/2429/menubar#testcases
16:19:38 <spectranaut> an example test plan
16:20:30 <Jean-Francois_Hector> V: Clicking on this, log in, then click 'run'
16:21:06 <Jean-Francois_Hector> JG: Some controls don't seem to be accessible (eg the switchers at the bottom)
16:22:03 <Jean-Francois_Hector> MF: Looks pretty good overall, except for some critical instances where it's not looking good
16:23:09 <Jean-Francois_Hector> MF: Ran axe, found a few violations (especially bypass blocks and colour control). This is better than most
16:24:06 <Jean-Francois_Hector> V: Each test case is a markdown text field. It should have the instructions and the expectations.
16:24:30 <Jean-Francois_Hector> MCK: How customisable is the data that we're collecting (eg pass field and the notes)
16:25:28 <Jean-Francois_Hector> V: At the moment it's just 'pass or fail'. To be investigated. There's the concept of extensions in Kiwi.
16:26:08 <Jean-Francois_Hector> We can import tests. Kiwi provides an Python API to import or update tests, if we want to save them separately outside the system.
16:27:02 <Jean-Francois_Hector> To run a test, there's a span that's a button called run. It activates a dropdown on hover. Couldn't make it work with the keyboard.
16:28:54 <Jean-Francois_Hector> MCK: Clicking on run didn't seem to do anything. I had to move the mouse to hover on it, then a button appears.
16:29:05 <Jean-Francois_Hector> Then hitting that button, I can't tell what happened.
16:29:12 <Jean-Francois_Hector> This is a little problematic.
16:29:21 <Jean-Francois_Hector> MF: Yes
16:31:30 <Jean-Francois_Hector> V: Should I investigate whether there are ways to extend the view that we have when going through test cases? Should I look into the code and at the extension paradigms that they have and talk to the developers?
16:32:38 <Jean-Francois_Hector> MCK: Not feeling super optimisting about Kiwi, given the amount of work it'd take to make it just accessible enough. It may be a fair amount of work.
16:33:29 <Jean-Francois_Hector> When it comes to actually running a test, we might want to layer questions so that (eg in the case of failures). I don't see enough flexibility here to do branching
16:33:55 <Jean-Francois_Hector> V: I haven't found branching in the open source softwares that I've investigated, that are being actively maintained
16:34:15 <Jean-Francois_Hector> What we're doing is branching a bit from what some of these applications have begun as
16:34:44 <Jean-Francois_Hector> I can look at TestLink later today, for us to discuss next week.
16:34:56 <spectranaut> css working group test harnesS: http://test.csswg.org/harness/
16:35:02 <Jean-Francois_Hector> Let's look at the CSS Working Group Harness (link just above)
16:35:19 <Jean-Francois_Hector> This test harness is a lot simpler
16:35:37 <Jean-Francois_Hector> it would also need to be extended for our particular use case
16:35:45 <Jean-Francois_Hector> it's nice that it's a simple interface doing a simple thing
16:36:00 <Jean-Francois_Hector> One active maintainer, who's enthusiastic about it being used
16:37:03 <Jean-Francois_Hector> Login is authenticated with the W3C log in system. Logging in only gives you the ability to review test results. The rest doesn't require logging ing
16:37:34 <Jean-Francois_Hector> Can mark test as Pass / Fail / Skip / Can't tell
16:39:16 <Jean-Francois_Hector> MF: Generally looks pretty good. But there's a keyboard trap (testing on Chrome). Stuff like that might be an easy fix, it depends
16:39:41 <Jean-Francois_Hector> V: This is a PHP application (Kiwi was a Django/Python application).
16:39:55 <Jean-Francois_Hector> This has little JavaScript in it.
16:40:12 <Jean-Francois_Hector> MCK: If you don't need to log in, does that mean that anyone can come in and run test cases?
16:40:25 <Jean-Francois_Hector> V: Yes
16:41:29 <Jean-Francois_Hector> MCK: We want to know who runs the test. Especially if we contract people to run tests. We may want to only have results from people that we have verified / qualified.
16:41:45 <Jean-Francois_Hector> If we don't know who the test results are coming from, that seems like a shortfall.
16:42:08 <Jean-Francois_Hector> V: In result of tests, you can see IP address, or if they have logged in you can see their name
16:42:21 <Jean-Francois_Hector> MCK: I wonder whether we could modify this to see Github login
16:43:02 <Jean-Francois_Hector> V: It's designed for humans to run the test.
16:43:25 <Jean-Francois_Hector> MF: Some tests are automated.
16:44:02 <Jean-Francois_Hector> V: This works with WPT tests. It's in the same repository
16:44:22 <Jean-Francois_Hector> JG: It's be a plus if we were using WPT – if this platform was connected to that.
16:44:53 <Jean-Francois_Hector> JG: I expect there'd be willingness within W3C to make changes for a W3C project.
16:45:26 <Jean-Francois_Hector> MCK: If the tests are written in HTML, does it have to be just static test in the HTML or could it include questions, or prompts, or branching? Could you put an HTML form into that HTML?
16:45:29 <Jean-Francois_Hector> V: Don't see why not
16:45:34 <Jean-Francois_Hector> JG: What would a form do?
16:46:29 <Jean-Francois_Hector> MCK: E.g. if there was a failure, gathering more information about the failure. So that we can you 'everything was good', or if it didn't pass, we'd want to collect more detailed information. This would help speed things up.
16:47:01 <Jean-Francois_Hector> JG: In WPT, I know that a test can have multiple parts.
16:47:34 <Jean-Francois_Hector> V: WPT is mostly for automated tests. There's a JavaScript test harness that'll execute tests.
16:48:57 <Jean-Francois_Hector> MCK: E.g. if the test is 'navigating to a checkbox in reading more' and the expectation 'read name role and state': if it read name role and state with up and down arrows, but only role and state with current line key, you'd want to capture that it didn't read the state with the current line key.
16:49:38 <Jean-Francois_Hector> If you tested that and none of them failed, it'd be nice to not have to mark 9 passes.
16:50:56 <Jean-Francois_Hector> We could require super granular input (eg per command and separately for name, role and state). But if they all work, it seems like a lot of work. I'm trying to think about how to make the testing easier.
16:51:30 <Jean-Francois_Hector> JG: E.g. what if there was a 'partial' success button, that'd take you to another page?
16:51:52 <Jean-Francois_Hector> JG: I like the simplicity of the CSS WG harness. Without much orientation I can go to tests and test something
16:53:01 <Jean-Francois_Hector> V: Another nice thing is that tests are associated with the specifications and grouped that way. We could make it possible to group all the tests related to an aria attribute, or an example.
16:53:51 <Jean-Francois_Hector> Should we focus on the investigating the TestLink option, or on CSS WG Harness?
16:54:38 <Jean-Francois_Hector> MCK: Requiring our testers to have W3C logins wouldn't be too big a problem. But I am hoping that we could get some tight integration with Github. E.g. for discussing.
16:55:39 <Jean-Francois_Hector> I'm not yet sure at what level we'd want to raise Github issues. But if there was the ability within the harness to press a button, and create a Github issue, to allow discussions of specific results or the design of a test, that'd be good
16:56:17 <Jean-Francois_Hector> V: Currently all the tests are in the WPT repository. That'd be a Github repository. If people wanted to file issues on tests, they'd probably do it there.
16:56:33 <Jean-Francois_Hector> Every WPT test is a test file in the WPT Github repository.
16:56:46 <Jean-Francois_Hector> A test file could be in the ARIA-AT repo.
16:57:38 <Jean-Francois_Hector> MCK: If a test is a test file in our repo, ... where would the code for the harness run?
16:58:01 <Jean-Francois_Hector> V: It's not in WPT or in Github. it's in Mercurial (note taker: not sure of spelling)
16:58:17 <Jean-Francois_Hector> It's a PHP application.
16:58:34 <Jean-Francois_Hector> (Not sure, based on comment from MF)
16:59:08 <Jean-Francois_Hector> MF: I think that this CSS WG solution is promising. I'd like to see a proposal of what it'd take to modify the CSS platform to our needs
16:59:11 <Jean-Francois_Hector> MCK: Agree
16:59:42 <Jean-Francois_Hector> V: Kiwi have an orientation that's different from us
17:00:10 <Jean-Francois_Hector> Will get deeper into the CSS WG Harness. E.g. more structured information, partial passing.
17:00:25 <Jean-Francois_Hector> MCK: Hosting and log-in requirements
17:00:52 <Jean-Francois_Hector> V: I think that this would also come with proposing a design for what our tests would look like.
17:01:16 <Jean-Francois_Hector> MCK: That's where looking at what both what JF and Michael have done could hopefully be useful.
17:01:43 <Jean-Francois_Hector> MF: That you so much Valerie.
17:02:07 <Jean-Francois_Hector> rrsagent, make minutes
17:02:07 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/25-aria-at-minutes.html Jean-Francois_Hector
17:04:23 <jongund> present+ jongund
17:04:40 <jongund> rrsagent, make minutes
17:04:40 <RRSAgent> I have made the request to generate https://www.w3.org/2019/09/25-aria-at-minutes.html jongund
20:24:30 <Zakim> Zakim has left #aria-at
20:40:00 <jongund> jongund has joined #aria-at
20:40:54 <jongund_> jongund_ has joined #aria-at