07:36:18 RRSAgent has joined #wcag-act 07:36:23 logging to https://www.w3.org/2023/09/15-wcag-act-irc 07:36:23 RRSAgent, make logs Public 07:36:24 Meeting: Accessibility Conformance Testing Teleconference 07:37:04 Jean-Yves has joined #wcag-act 07:41:52 enrico has joined #wcag-act 07:44:11 giacomo-petri has joined #wcag-act 08:02:31 jamesn has joined #wcag-act 08:03:04 spectranaut_ has joined #wcag-act 08:15:08 daniel-montalvo has joined #wcag-act 08:26:47 AWK has joined #wcag-act 08:48:48 jcraig has left #wcag-act 09:31:21 daniel-montalvo has joined #wcag-act 09:51:41 Jean-Yves has joined #wcag-act 09:51:50 present+ 09:51:50 CarlosD has joined #wcag-act 09:52:00 present+ 09:52:02 Topic: Empty Headings Rule 09:52:44 MarkRogers has joined #wcag-act 09:52:44 Wilco: We said yesterday that the rule would be deprecated because we were failing it under 1.3.1 and there are no longer cases where this is an issue 09:52:48 ... Deprecated may be an option, but ARIA requires headings to have an accessible name 09:53:08 ... We could have a rule that checks that from the ARIA perspective, much more generic 09:53:38 ... It could cover headings, buttons, column headers, etc 09:53:58 ... But we cannot transform the current rule 09:54:33 ... We could for the moment map this rule to ARIA until we have a new rule 09:55:36 ... But it creates a situation where implementations that currently fail this under 1.3.1 would be inconsistent 09:55:48 Jean-Yves: That would not be a problem for mee 09:56:09 Carloss: Same here, good to have it from an ACT perspective 09:56:27 Markr: Same here, would be easy to change 09:57:04 Jean-Yves: Currently I wouldn't deprecate the rule only tool. I would do when we have the new one 09:57:38 Mark: Deprecating does not make much difference for us, we just continue using it until wee have the new rule 09:58:09 Wilco: We'll need to check with all tools explicitly 09:58:29 Wilco: I like that approach much better 09:59:03 q+ 09:59:31 giacomo-petri has joined #wcag-act 10:00:07 Daniel: What about Trusted Tester? 10:00:07 Wilco: They do not implement this rule, they are going to have opportunity to review PR anyway 10:00:54 Wilco: Do we think that's a 2-week Call for Review? 10:00:54 Jean-Yves: Yes. 10:00:54 Carlos: Yes, significant change. 10:01:12 Rrsagent, draft minutes 10:01:13 I have made the request to generate https://www.w3.org/2023/09/15-wcag-act-minutes.html daniel-montalvo 11:52:31 thbrunet has joined #wcag-act 11:57:19 daniel-montalvo has joined #wcag-act 12:02:28 trevor has joined #wcag-act 12:02:35 Jean-Yves has joined #wcag-act 12:02:46 present+ 12:03:48 kathy has joined #wcag-act 12:03:58 suji has joined #wcag-act 12:03:58 present+ 12:04:01 CarlosD has joined #wcag-act 12:04:01 Wilco has joined #wcag-act 12:04:04 present+ 12:04:05 present+ 12:04:07 agenda? 12:04:07 present+ 12:04:08 scribe+ 12:04:13 enrico has joined #wcag-act 12:04:20 clear agenda 12:04:59 agenda+ Acc support & assumptions in the background 12:05:05 zakim, take up next 12:05:05 agendum 1 -- Acc support & assumptions in the background -- taken up [from Wilco] 12:05:16 agenda+ Optional test cases 12:05:21 agenda+ Machine learning and ACT Rules 12:05:26 agenda+ Transition states 12:05:28 https://www.w3.org/TR/act-rules-format/#act-rule-structure 12:05:40 present+ 12:05:55 present+ 12:06:01 giacomo-petri has joined #wcag-act 12:06:27 kathy: Looking at the structure of an ACT rule. As you get through the list assumptions and accessibility support are both required 12:06:47 https://www.w3.org/WAI/standards-guidelines/act/rules/09o5cg/ 12:07:51 ... pages or rules have a navigation sidebar. Interested in moving some things to be under background. Thinking specifically assumptions and accessibility support could go under there. 12:08:22 ... right now the background has the bibliography as a subsection, interested to move a few more things. 12:08:55 q+ 12:09:49 daniel: Would probably be less cluttered, might want to rename background if we put these as subsections 12:10:06 ... don't think assumptions or accessibility support are background because they are required 12:11:04 ack daniel-montalvo 12:11:15 wilco: Thought this might be okay since we mix up these section anyways. Some times accessibility support background shows up in the accessibility support 12:11:33 ... find that people outside this group are regularly overlooking the information. 12:11:48 ... if you don't know what is expected in accessibility support you might just check the background 12:12:00 daniel: agree with the thinking, but don't like the naming. 12:12:11 ack d 12:12:14 ack j 12:12:48 q+ 12:12:49 Jean-Yves: No strong opinions, might need to make other changes, like background becomes required if we put assumptions and accessibility in there 12:13:30 ... will need to have a good structure since this will become a large section. 12:14:02 ... might have times where accessibility support and assumptions are more normative than what is usually included in the background. 12:14:29 ack t 12:14:31 ... slight concerns with blurring everything together 12:14:48 thbrunet: Would it help to borrow a term from WCAG and use the term "understanding" 12:15:13 daniel: Sounds better to me, gives a bit more idea of what to find there 12:15:25 Jean-Yves: I think that could be a good name 12:15:40 thbrunet: And I think from WCAG people are trained to go look at the understanding documents 12:16:08 kathy: I think "Understanding" works, might need to look at other categories that might fall under this and make sure they work as well. 12:16:40 ... Thought rule versions or implementations could potentially also be included. 12:16:55 wilco: Would like to leave implementations where they are personall. 12:17:07 kathy: Currently not in the format 12:17:30 wilco: The PR adds it to the rule structure and defines it. 12:18:05 kathy: Rule structure also has change log and live site has rule versions. 12:18:20 wilco: I like rule versions better 12:18:23 giacomo-petri_ has joined #wcag-act 12:18:42 kathy: would that be something we could put under the background? 12:18:51 wilco: I think that is a separate thing. 12:19:40 Jean-Yves: Would also put bibliography and related rules could also make it into the ACT structure. 12:19:52 wilco: Sounds like we have a good direction here, just need to write it down. 12:20:08 zakim, take up next 12:20:08 agendum 2 -- Optional test cases -- taken up [from Wilco] 12:21:03 Jean-Yves: Our rules have two uses to make sure that humans understand what the rule is testing and then we also care about tools and methodologies getting correct results. 12:21:22 https://github.com/act-rules/act-rules.github.io/issues/1954 12:21:54 ... when we add things that clash with the accessibility support. 12:22:15 ... according to the spec, the alt is empty and go to the title, others just say that is now name 12:22:32 ... if you take a rule like image button has non-empty accessible name 12:22:59 ... for the specification, it should be passed, but it does result in an actual accessibility violation 12:23:41 ... in this case, we ended up removing the example from the rule because it couldn't really be a pass or fail example 12:23:59 ... but it is a very valuable test case that we would like to have included in the rule. 12:24:30 https://github.com/act-rules/act-rules.github.io/pull/2077#discussion_r1246410253 12:24:32 ... We have had multiple other times where we remove things because of how they were affected by accessibility support. 12:25:14 ... Had a rule targetting empty headings, example was an empty div 12:25:47 ... we also have problems in some rules of having too many examples, and it might be bad for the human, but also worthwhile having for the computer 12:26:08 q+ 12:26:12 ... the solution that was mentioned for the accessibility support problem would be optional test cases that don't affect the implementation 12:26:45 ... specs may say pass, but some accessibility support problem could cause it to fail 12:26:58 ... optional cases to test some corner cases 12:27:40 q? 12:27:43 ... those are the two sides of it. Problem with accessibility support is much more important. Want to enforce standard, but understand where accessibility support lies. 12:27:47 ack t 12:28:26 thbrunet: Would propose to change the wording to use 'extended' instead of 'optional'. So you have a 'core' test set and an 'extended' test set 12:29:35 wilco: I see value in this tracking problems around accessibility support. Have had problems with test cases changing depending on how the accessibility support. Causes problems with tracking. 12:30:15 ... that doesn't necessarily need to be part of the rules though. In adding this, it might add more work on maintaining on each rule. 12:30:19 MarkRogers has joined #wcag-act 12:30:24 q+ 12:30:44 ... will move test cases around and then implementers would need to agree to that 12:30:54 q+ 12:31:03 ack m 12:31:10 q+ 12:31:17 Jean-Yves: This follows spec, but then some browser doesn't support it. 12:32:18 MarkRogers: XML1.1 is really hard to test due to all of the mimetypes, its very easy to mess up some configuration and that could affect results. Need to be very specific about versions when testing, things update all the time 12:33:10 ... versions don't need to be in the rule, but we might should have some kind of metadata. 12:33:59 wilco: Agree, think there are better places to put the information. Second thing is that if the spec and browser are in conflict, it doesn't directly mean the browser is wrong and the spec is right 12:34:04 ack w 12:34:49 q+ 12:35:14 wilco: Have had talks of making a repo with recording this information and including test instructions to verify that the accessibilty support problems 12:35:25 ... not saying there is no room to optional test cases 12:36:10 ... as for having more test cases, and having an extended set of test cases and using them for automation. Not sure I am keen on that either, since it makes manual implementation of those rules more burdensome 12:36:29 ack k 12:36:30 ... would like for us to stay selective of our rules 12:37:02 kathy: I like the idea of tracking these separately, but if they are included in the rule, I think if people see optional they are just gonna not do it. 12:37:26 ... I think there needs to be some significant description for why the other test cases are separated and not included in the consistency examples 12:37:53 ... if they are optional, what result should they get? What would their results be used for? 12:37:53 ack j 12:38:05 q+ 12:38:50 Jean-Yves: Should we include them in the rule, should we link to them from the rule, then always a question of if we want detailed test instructions, specifically which assitive tech to consider (e.g., browser) 12:39:23 ... that might be a bit on the line. Its something that has blocked us before. Think it makes sense to not necessarily include them in the rule 12:40:12 ... could have some use linking them from the rule, could use them to demonstrate accessibility support issues. 12:41:03 ... not exactly sure how we can manage on it. Maybe have example without detailed instructions on the rule page and example with full instructions on other site. 12:41:54 wilco: Was some work, AT drivers which wants to programmatically run screen readers and cross-browser platform testing which is increasingly incorporating more accessibility feature testing. Being used for role and name computation 12:42:08 ... those are resources that I'm hoping we can be using instead of having to do it ourselves 12:42:39 ... the only reason I think we might want to do something like this, is to create transparency about how different implementers report these different test cases 12:42:59 ... have a sense that people are not digging that deep into our rules today, except for maybe besides ourselves. 12:43:46 q? 12:43:46 ... a possible example would be to have a dummy rule, but not have it as a rule that is tracked. Uses the systems we have already in place, but doesn't show up on the implmentations list 12:43:46 ack w 12:44:36 Jean-Yves: Even without the full test example suite, we still need to udpate the accessibility support. We can't just point to those resources and say to check if they are conforming 12:45:18 ... hope that it will bring some harmonization at that level. Might be able to link specifically to web platform examples that demonstrate the issues 12:45:34 ... might be a way to track it without pointing fingers since it is a separate group 12:46:28 wilco: Is this something we want to continue pursuing, just keep thinking about it? Put it on the agenda? 12:46:50 suji: Would like to discuss it a bit more regularly to get a conclusion on it. 12:47:39 Jean-Yves: From the accessibility support side, using the web platform tests might be the nicest way and require less maintenance. 12:48:05 ... need to investigate further before commiting to optional cases further 12:48:31 q+ 12:48:45 wilco: Proposal is to look at the platform tests for accessibility support. Which means for the moment if we see major accessibility support issues we don't include test cases for them. 12:49:40 ... for the web platform, talked with member that it might be possible for us to have a section where we could contribute our own test cases to. 12:50:03 daniel: It would be on a browser, it doesn't tell us what the assistive technology might do. 12:50:45 wilco: I initially really liked the idea of optional test cases, but think there might be better ways to help our accessibiltiy support problems. 12:51:36 Jean-Yves: Goal to aim for is finding accessibility support, try to find web platform test, submit one that is relevant to the rule. Finding the appropriate place and method for contribute to the web platform test 12:51:45 ... same goes for the AT driver test 12:52:08 ... something where we add this guidance and know what we should do. 12:52:23 ... something that would make the revisit test much easier. I think that should be our goal 12:54:36 zakim, take up next 12:54:36 I see a speaker queue remaining and respectfully decline to close this agendum, trevor 12:54:45 q? 12:55:20 ack j 12:55:20 zakim, take up next 12:55:20 agendum 3 -- Machine learning and ACT Rules -- taken up [from Wilco] 12:55:53 https://github.com/act-rules/act-rules.github.io/discussions/2113 12:55:53 wilco: Setup a discussion. 12:56:59 ... brief introduction, ML is starting to change our industry. Seems that the rules we have been writing, especially for ones that haven't traditionally been automatable, we don't handle predictive models well 12:57:32 ... on top of that, our examples are very minimal, so they may not work as well on our minimalist examples. 12:58:00 ... even if they did, not clear that they would get these right in the real world. 12:58:37 ... three topics, Do ACT test cases work with ML, Training on ACT test cases, How does confidence fit with ACT consistency 12:59:26 Jem has joined #wcag-act 12:59:28 MarkRogers: Would problem could be people poisoning the datasets, such as incorrectly labelling an email as spam on purpose 13:00:21 scribe+ 13:00:59 ... add "Poisoning data" topic title 13:01:04 Zakim, agenda? 13:01:04 I see 2 items remaining on the agenda: 13:01:05 3. Machine learning and ACT Rules [from Wilco] 13:01:05 4. Transition states [from Wilco] 13:01:10 TOPIC: Do ACT test cases work with ML? 13:01:25 wilco: first topic - test cases 13:01:43 https://github.com/act-rules/act-rules.github.io/discussions/2113#discussioncomment-6948559 13:03:59 q+ 13:04:40 q+ 13:04:51 trevor: ML systems can work with small test cases if trained, and others can use the entire page 13:05:20 ... if they can't handle the test case, change the configuration 13:06:04 ... no guarantee on how well ML will perform in real world scenario 13:06:19 ack m 13:06:39 q+ 13:07:06 ack j 13:07:07 q+ 13:07:10 markrogers: may be difficult to understand a whole page for pass or fail 13:07:50 q+ 13:07:57 q+ 13:08:37 q+ 13:08:58 ack t 13:09:06 jean-yves: real web page many issues while test cases are one 13:09:06 ack tr 13:09:20 qq+ thbrunet 13:09:50 trevor: even with large data sets, some may be simple like our test cases 13:10:34 q? 13:10:37 ack th 13:10:37 thbrunet, you wanted to react to trevor 13:11:11 thbrunet: having multiple targets within the same test has been questionable 13:11:14 ack c 13:12:14 carlos: don't think we should build test cases for predictive models, but other way around 13:13:44 ... train on our test cases 13:13:59 q+ 13:15:42 enrico: future implementations using ML may identify a portion of the page 13:17:14 ... technology should be able to handle simplicity 13:17:29 ack w 13:18:11 wilco: summarizing, general agreement that ML should be able to handle simple test cases 13:18:45 ... have a different category of implementations. talk about this in third topic 13:20:49 q+ 13:20:49 q? 13:20:54 ack m 13:20:56 ... contextual applicability 13:21:04 ... confidence of ML in real world 13:21:29 q+ 13:21:31 markrogers: don't train on ACT examples 13:22:21 q+ to mention need of context for "look like" rules, and validity on any tool based on implementation report 13:23:23 ack th 13:23:23 ... how much is needed for rule examples is needed for testing depends on requirement 13:23:33 ack j 13:23:33 Jean-Yves, you wanted to mention need of context for "look like" rules, and validity on any tool based on implementation report 13:24:52 jean-yves: examples in rules that require context will have bigger examples for ML 13:25:27 q+ 13:26:29 ... validity of ML in real world, assume tool wasn't trained just on test cases 13:26:59 Q+ 13:27:01 wilco: concerned ACT test cases are so simple the ML tool isn't challenged 13:27:27 jean-yves: not a problem 13:28:15 ... tool only trained on test cases won't be good in real world 13:28:21 q? 13:28:51 wilco: we never meant for ACT to be a quality checker 13:29:14 trevor has joined #wcag-act 13:30:25 Q? 13:30:25 ack w 13:30:25 ack th 13:30:46 thbrunet: split test cases to explanatory and test examples 13:31:06 q+ 13:31:17 ... explanatory doesn't have to be real world 13:31:32 ack d 13:31:35 ... have tool vendors agree to not train on ACT 13:33:06 daniel: examples are granular, not worried. more worried about context 13:33:09 q- 13:33:41 trevor: if it doesn't work in real world, not a big concern. 13:33:59 q+ 13:34:20 ... not guaranteeing performance outside of test cases 13:34:43 ... don't think it's our job 13:34:51 ack me 13:35:23 wilco: conclusion sounds like we don't need to pivot our approach 13:36:06 ... also not worried about quality in real world 13:36:18 q+ 13:36:41 scribe+ 13:36:52 Kathy: Coming from manual, when I go through test caes 13:37:04 ... The applicability is spelled out for me so I know there is something to test here 13:37:13 ... We're expecting testers will be able to find the applicability 13:37:28 ... It sounds similar to ML concern, ability to find what needs testing. 13:37:49 ... I would agree that we're not responsible for how well the methodology performance, but we are with what should pass and what not? 13:38:32 markrogers: ML can be scripted for the test cases 13:38:44 wilco: answer is known 13:39:13 TOPIC: Training on ACT test cases 13:39:59 wilco: on the links in context question, TF asked about Power Mapper implementation 13:40:24 markrogers: we used the test cases 13:40:30 q+ 13:40:33 ... was that bad or good to do? 13:40:38 ack m 13:40:39 ack j 13:41:13 jean-yves: we can't prevent use of test cases for learning 13:42:08 q+ 13:42:12 ... if overtrain on test cases, it could be bad. 13:42:17 q+ 13:42:45 ack c 13:42:47 q+ 13:43:12 carlosd: should be a big concerned if only train on test cases, but don't think we need to change 13:43:19 q+ 13:43:29 ... but we can't prevent it 13:43:29 q+ 13:44:21 ack tr 13:44:28 jean-yves: implementers can also lie on their reports 13:44:50 trevor: no way to test an ML model even with a giant data set 13:45:20 ... over training on examples isn't always a negative 13:45:58 q? 13:46:01 ack w 13:46:41 wilco: existing models train on specific web sites 13:47:04 ... open source Crest trains against the website first 13:47:27 ... doesn't seem right to train on model being evaluated 13:47:40 ack th 13:47:42 ... not opposed but not in favor 13:48:26 thbrunet: can't govern tool vendors to overfit 13:48:27 ack m 13:48:45 q+ 13:52:20 wilco: proposal is to recommend against training on test cases, but not to regulate 13:52:27 +1 13:52:30 +1 13:52:31 +1 13:52:32 +1 13:52:33 +1 13:52:34 +1 13:53:30 TOPIC: How does confidence fit with ACT consistency? 13:53:30 https://github.com/act-rules/act-rules.github.io/discussions/2113#discussioncomment-6950358 13:54:30 q+ 13:55:11 q+ 13:55:26 q+ 13:55:30 ack w 13:55:41 ack m 13:56:11 q+ 13:56:11 markrogers: can ask implementation to report confidence. 13:56:38 ack t 13:56:44 ack tr 13:56:46 ... number as a percentage or probability since it's available with their tool 13:56:47 qq+thburnet 13:57:09 trevor: confidence is arbitrary across ML models 13:57:56 ... an overconfident model vs conservative model 13:58:44 ack th 13:58:44 thburnet, you wanted to react to trevor 13:58:49 ... can be a user setting for confidence threshold can be followed for implementation report 13:59:31 thbrunet: more of an issue for subjectivity. don't need confidence for objective pass/fail. 13:59:40 ack j 13:59:41 ... maybe another category of test cases 14:00:10 jean-yves: agree to request a tool to give yes/no answer 14:00:14 q+ 14:01:20 ... can request confidence to evaluate inconsistent results 14:01:51 q+ 14:03:01 ... don't want to overload manual testers with too many test cases 14:03:05 ack c 14:03:37 carlosd: request pass/fail with confidence to measure consistency 14:04:47 ... develop a different metric for AI-based tools 14:05:04 q+ 14:05:21 ack tr 14:05:45 trevor: less concerned about things testing differently on different days 14:06:28 ack w 14:06:33 q+ 14:06:38 ... think testing of ACT test cases will be consistent 14:07:10 wilco: expect with limited ACT test cases, tools won't vary day to day 14:08:05 ... reporting confidence could be an option 14:08:20 ... but might not be meaningful 14:08:37 ... if not standardized 14:09:12 q? 14:10:22 ... testing in its own category 14:10:22 ack th 14:10:22 q+ 14:10:22 thbrunet: implementers should be doing their own checks 14:10:22 q+ 14:10:22 q+ 14:10:56 ... implementation note on type might be good to have 14:11:18 wilco: check level and not at the tool level is good 14:12:54 ... we expect tools to run in default configuration 14:13:34 ... axecore doesn't look at fallback roles by default, but we can turn that on. run ACT on default. 14:14:14 ... can ask implementer to run in default for ACT 14:14:17 ack tr 14:14:44 trevor: allow people to define their own confidence 14:15:47 ... if new implementer class, metadata used for decisions could be interesting for users looking at implementation reports 14:15:49 q+ to talk about open fields on wai site 14:16:00 ack me 14:16:00 Wilco, you wanted to talk about open fields on wai site 14:16:04 q+ 14:17:04 ack MarkRogers 14:17:43 markrogers: link text goes to same page or similar may be hard 14:17:52 scribe+ 14:18:19 Wilco: i like to see how we can get more metadata, more info from implementers. 14:19:02 ... implementers already want to report why things are different that they should, can't do that because WAI doesn't allow random text to be injected. 14:19:24 q+ 14:19:27 ... would need pre approved messages, numbers, ... but not a plain text. 14:19:44 ... but I'm in favor of more metadata. 14:19:54 ack Wilco 14:20:08 ... We expect reports to use the default setting (confidence threshold) 14:20:11 Proposal: ML based tools must use "default" confidence threshold 14:20:19 +1 14:20:20 +1 14:20:23 +1 14:20:54 +1 14:20:54 +1 14:20:54 +1 14:20:54 +1 14:20:54 +1 14:20:54 +1 14:21:52 ack m 14:22:43 MarkRogers: we can put metadata in a github issue, then only store the number on WAI site. 14:23:36 Wilco: It seems we do not need a new category of tools. These are automated tools. 14:23:54 Proposal: ML based testing fits in the automated testing category. 14:24:09