Meeting minutes
Review agenda and next meeting dates
Matt_King: The agenda is available at https://
Matt_King: Anyone want to modify it?
lola: Could we talk about the ARIA projects and how we want to sequence them--support tables included?
Matt_King: This is what I had in mind with the "support tables" item on the agenda. We can expand on that a bit during that discussion
James_Scholes: I seem to remember that we wanted to debrief following our attendance at the Browser Testing and Tools ("BTT") meeting. This may also be related to that
Matt_King: jugglinmike is awesome
Current status
Matt_King: We now have 5 plans in Candidate Review
Matt_King: two of them are approved by Vispero
Matt_King: Thanks to everybody for all the hard work!
Matt_King: The test queue is now empty, which is an interesting state of affairs. We'll address that soon
Matt_King: The radio test plan had one open issue which was feedback from Vispero. I've added a comment for them asking them to revisit now that we have a new plan
Matt_King: I've created a pull request, linked in the agenda. It's a one-character change needing reviewers. The goal is to fix the "alert" plan so we can close issue 1032
Matt_King: With that, we'll bereally well-positioned to meet with Apple
IsaDC: I can review that; please assign me
Matt_King: Thank you!
Matt_King: I don't know how the app will treat that change because it doesn't impact any test results. I think it should be one of those changes that "just happens", i.e. without creating any new plan version, because it is essentially editorial
Matt_King: At least, that's what I'm hoping
howard-e: It will indeed create a new version
howard-e: It will be named according to the date of the merge. The current version is already in candidate. When you press "advance" to the new version from "R&D", it will copy the old version's test results
howard-e: From there, you can immediately finalize and advance to candidate
howard-e: I was anticipating this specific question, so I tested it locally to verify that this will work as I've just described
Matt_King: Our next meeting with Vispero is on Wednesday of next week
howard-e: In that case, I'd be more comfortable planning a deployment for Monday
howard-e: Also, there's a changelog file in the ARIA-AT repository that will be automatically updated with the relevant information at the moment of the deployment
Matt_King: That sounds good. We'll get this merged, and we'll test out this new process
Enhancements to APG support tables
https://
Matt_King: This is in response to the feedback that we received last year when we added AT support tables to the APG for the first time
Matt_King: Lots of people were involved in that
Matt_King: There was some public feedback about the understandability of those reports and what the high-level data meant
Matt_King: one of the changes we've been working on in response to that concerns "MUST", "SHOULD", and "MAY"
Matt_King: We're trying to surface two numbers at the highest level for each pattern
Matt_King: The AT support table will show both the numbers for the "MUST"s and the "SHOULD"s
Matt_King: ...and we will have a link in the tables to a page which explains what this data means
Matt_King: It sounds like the code could be ready as soon as end-of-day tomorrow. That's what we were discussing yesterday
carmen: That's correct; we're testing today and anticipating a release tomorrow
Matt_King: So we should be able to make a pull request to APG after that and review it in the APG meeting next week
Matt_King: That will also enable us to create this page with the explanation of how to read the report
howard-e: Someone at Bocoup will create the pull request to APG once the requisite changes are ready on the "staging" server
Matt_King: We also want to create a page for the "about" section of the APG that explains how to read the reports
carmen: Boaz is working on the copy for that page. I will ask him about it today
Matt_King: Okay, maybe just add a link to Boaz's document in the issue you've created about this, carmen
carmen: Got it
Matt_King: Ideally, we'll get this into the May 7th publication of APG. That's my target
Matt_King: We can think about one or more blog posts after that. I was hoping for Global Accessibility Awareness Day, but that might be difficult... Especially since publishing a post will require coordination with W3C
Matt_King: We want to make sure that the content of the reports is understandable by anybody who might come to the APG and read it. I really want to make the editorial text as understandable and approachable as possible
Automation V2 project
<SamShaw> jugglinmike: Matt created an issue for this
https://
jugglinmike I shared this to the mailing list a few weeks ago, I'm not sure we are ready to discuss today. It would be good to get some confirmation from people that this is inline with their understanding
Matt_King Can you give people an understanding of what its all about?
jugglinmike Sure this is a description of the project to enhance the integration of automated AT response collection so
that it can be more fully integrated into the community group's processes.
jugglinmike It lays out the problem, background, goals, deliverables, and requirements
Matt_King VO and Firefox are listed as separate items. If you add firefox support, and you add JAWS support, would the automation be ready for both? Or would you have to create a pairing for automation to use?
jugglinmike On the technical level, it could be either way. I was considering it would be available across the board. I think we can decide what we want to do for each case
Matt_King I wasn't sure when you mention Firefox, is that available for Mac too?
jugglinmike No, we are still working on Mac
Matt_King Okay that part was unclear in the deliverables section
Matt_King okay there weren't any priorities on things, like reusing verdicts.
Matt_King Alyssa I was curious of your experience with the NVDA bot?
Alyssa: Tests 1 through 4 for tab and shift tab with focus mode on, for radio button, the app didn't record the output. I had to manually add it
Matt_King I thought Mike when the output wasn't captured, that it actually had collected responses
Alyssa: should I send you the test that it didn't record the output?
jugglinmike yes please
Matt_King for the tests that it did collect output, was that helpful? Was that faster?
Alyssa: Yes, I found it to be very helpful
jugglinmike: In this case the bot recorded that there was no output, (rather than missing the output)
Matt_King So we need a way to make that more reliable?
jugglinmike yes. There is one area we know certain keys aren't being recorded and we are investigating
jugglinmike We don't think this issue is limited to firefox
Matt_King So one of the next things it could do, if the output is the same as what was previously recorded, then it can also check off the verdicts
Matt_King When I was thinking about this I thought of something worth discussing
Alyssa: for tests 1 through 4 for tab and shift+tab with focus mode the bot recorded no output
Matt_King We changed our form to be checkboxes for assertions, before there were 3 choices, when we made it two choices we changed to check boxes
Alyssa: 5 through6 bot recorded no output for up arrow with browse mode and shift tab with focus mode
Alyssa: 11 bot reported no output for space with focus mode
Alyssa: 12 and 13 bot reported no output for both cases
Matt_King We actually don't know if one has been checked. Should we go change to yes/no answers for everything? For the bots, we would know which ones were and were not recorded. and if a human tester missed one
Matt_King Not checked could mean you skipped it
jugglinmike Glad you raised this Matt, there is an issue for this
jugglinmike UI cannot accurately describe the state of assertion verdicts #1045
Matt_King So we are both thinking along the same lines
Matt_King I think the reason it was complicated with the radio buttons was that it was confusing with a table [to show our report], but now we dont have a table so I think we could change this
Matt_King Do other people running tests thing this would be a good change?
IsaDC: I think so
Cool I will use IsaDC
Joe_Humbert So I should be capturing the default output with usage hints
Matt_King Okay so there is this document with the goals for the next phase of the automation project, please review and comment
Matt_King Mike lets work on 1045 and create an action
jugglinmike I created another issue that we need to fix as well, I'll paste a link to it
jugglinmike I think we need to discuss how to modify the statuses
Matt_King Okay we can discuss
Matt_King lets move on to the Dialog Test plan
Dialog test plan
Matt_King: one of the things that we're talking about testing is related to testing the boundaries of a modal dialog
Matt_King: e.g. not letting you fall outside of it when you are reading
IsaDC: had a question in the issue, so I added an agenda item to discuss it here today
IsaDC: How should we structure the test? How far do we want to test that?
Matt_King: I encourage people to open the link in the agenda to review the draft test plan
https://
Matt_King: Test six is "navigate to the end of the modal dialog"
Matt_King: if you open the page for test six, run the test setup, I think one thing we might want to change is that right now, the focus goes to the input field labeled "street"
Matt_King: I think if we were to set the focus on the button "verify address", then screen readers would automatically be in reading mode rather than browse mode
Matt_King: And this test is about behavior in reading mode, after all
Matt_King: I've been thinking that we could have another command, called something like "navigation beyond dialog boundary" where of course what we want is to observe this as being impossible
Matt_King: The command sequence would be CTRL+END followed by down arrow, followed by up arrow
Matt_King: That would have one assertion: "the name of the button 'add' is conveyed" or, more explicitly, "screen reader cursor is on the 'add' button" (possibly with better wording)
Matt_King: Does that make sense to people
IsaDC: I think it could be confusing
Alyssa: What about trying to tab or shift+tab out of it?
Matt_King: We have a separate test for that. A test for navigating to the beginning and the end
James_Scholes: I don't know if we need such a test because that's controlled by the page
James_Scholes: Oh, so we're not testing that the tab trapping is implementing correctly; we're testing the screen reader's response to the tab trapping
Matt_King: Correct
IsaDC: Yes
James_Scholes: Ctrl+End is a special case because the dialog appears at the bottom of the page. Maybe we need to move the dialog so that something else exists below it
jugglinmike: We're out of time
James_Scholes: I think this requires additional discussion
IsaDC: Matt_King can you draft your idea as a proposal?
Matt_King: Sure