19:57:57 <RRSAgent> RRSAgent has joined #aria-at
19:58:01 <RRSAgent> logging to https://www.w3.org/2023/12/14-aria-at-irc
19:58:01 <Zakim> RRSAgent, make logs Public
19:58:02 <Zakim> please title this meeting ("meeting: ..."), Matt_King
19:58:19 <Matt_King> MEETING: ARIA and Assistive Technologies Community Group
19:58:29 <Matt_King> CHAIR: Matt King
19:58:35 <Matt_King> present+
19:58:42 <Matt_King> rrsagent, make minutes
19:58:43 <RRSAgent> I have made the request to generate https://www.w3.org/2023/12/14-aria-at-minutes.html Matt_King
20:00:11 <jugglinmike> jugglinmike has joined #aria-at
20:01:54 <Joe_Humbert> Joe_Humbert has joined #aria-at
20:02:18 <jugglinmike> present+ jugglinmike
20:02:22 <jugglinmike> scribe+ jugglinmike
20:02:29 <Joe_Humbert> present+
20:02:51 <jugglinmike> Topic: Review agenda and next meeting dates
20:04:11 <murray_moss> murray_moss has joined #aria-at
20:04:25 <jugglinmike> Matt_King: Starting in January, we will have to use a different Zoom account for this meeting
20:04:27 <howard-e> howard-e has joined #aria-at
20:04:37 <howard-e> present+
20:04:58 <jugglinmike> Matt_King: We're still deciding which account to use, but we'll update the meeting invitation once we do
20:05:05 <murray_moss> present+
20:08:05 <jugglinmike> jugglinmike: I will lead a meeting of the ARIA-AT Automation on this Monday, December 18
20:08:56 <jugglinmike> Matt_King: Great! I can't tell you how excited I am to get this into production. It's so cool to tell the Bot to go collect responses, and then to verify that the data it reported is accurate
20:09:45 <jugglinmike> Matt_King: We have a meeting for this group scheduled for next Wednesday, December 20. Should we cancel? I know that PAC will be on holiday at that time.
20:09:59 <jugglinmike> Matt_King: Hearing no response, I will cancel.
20:10:38 <jugglinmike> Matt_King: This means that the next meeting of this group will be Thursday, January 11, 2024. The W3C calendar will reflect this.
20:12:20 <jugglinmike> Matt_King: It's possible that Vispero will approve some test plans before the end of the year. That would be extremely helpful. However, there's no hope of getting any feedback from Apple by the end of the year
20:12:55 <jugglinmike> Topic: Matt_King: If possible, I w
20:13:10 <jugglinmike> s/Matt_King: If possible, I w/Manual testing progress check-in/
20:14:17 <jugglinmike> Matt_King: If we could get these four test plans into "candidate review" by the end of the year, that would be great: Alert, Command Button, Link, and Toggle Button
20:14:31 <jugglinmike> present+ IsaDC
20:14:52 <jugglinmike> IsaDC: I'm unable to access all of the data
20:15:21 <jugglinmike> Matt_King: What we have right now with Toggle Button is an instance of the bug that's later in the agenda
20:16:08 <jugglinmike> Matt_King: But for Alert, if you access the Test Plan through the "Data Management" page, then in the "draft review" column, you'll find that the required reports are in progress
20:16:50 <jugglinmike> Matt_King: Clicking through to that, it appears that testing for VoiceOver is 100% complete by two testers. That's surprising
20:16:58 <jugglinmike> present+ murray_moss
20:17:10 <jugglinmike> murray_moss: Oh, that was me--I just finished testing today
20:17:14 <jugglinmike> IsaDC: That's great!
20:17:56 <jugglinmike> IsaDC: I think we can mark "Alert" as final, now
20:18:28 <jugglinmike> Matt_King: I will do that, now...
20:19:07 <jugglinmike> Matt_King: It should say "Required reports complete" on the "Data Management" page, now
20:19:12 <jugglinmike> present+ Joe_Humbert
20:19:15 <jugglinmike> Joe_Humbert: It does!
20:19:37 <jugglinmike> Matt_King: Great, then we can advance "Alert" to the "candidate" phase
20:21:09 <jugglinmike> Matt_King: Okay, so, "Command Button" for JAWS has a problem. The version that's in the test queue, and the one that folks recently ran, is a deprecrated version of the plan
20:21:36 <jugglinmike> Matt_King: I don't know what to do about this
20:21:50 <jugglinmike> Matt_King: There's a bug: the deprecated version of the test plan should not show up in the test queue
20:22:05 <jugglinmike> Matt_King: More immediately, though, we should figure out what our options are
20:22:28 <jugglinmike> Matt_King: We could add the December 13th version of the plan to the test queue. Then both versions will be there, but the December 13th version will be empty
20:22:45 <jugglinmike> Matt_King: Nothing changed about the JAWS tests for that version
20:23:49 <jugglinmike> Matt_King: howard-e, is there a better way to manage this state rather than just throwing away IsaDC's work and asking her to re-do it
20:24:27 <jugglinmike> howard-e: Without explicit assistance from the Bocoup team, the so-called "Brute force" solution that you just described is the only option
20:26:03 <jugglinmike> howard-e: I think the reason why this didn't work is a recent settings change affects a top-level value in the test
20:26:46 <jugglinmike> howard-e: I think we may have a backup of the verison before; we may be able to roll back, but it makes me uneasy to manipulate the data in that way
20:27:34 <jugglinmike> Matt_King: If there's no straightfoward way to do this on the backend, then IsaDC: how long would it take to re-run the test plan?
20:27:57 <jugglinmike> IsaDC: The issue is not only Toggle Button. That's just the one that triggered the bug. We made changes to Link, Command, and Toggle Button
20:28:53 <jugglinmike> Matt_King: You could just copy your output manually. You don't have to re-run the test; all you have to do is look at your results, copy the output strings, and select "pass" for the rest
20:29:15 <jugglinmike> Matt_King: Because JAWS passed everything in your report and because nothing has changed
20:29:53 <jugglinmike> IsaDC: I'm concerned about "link" because I marked as final a deprecated version for which we had results
20:30:00 <jugglinmike> IsaDC: This was for NVDA
20:31:08 <jugglinmike> IsaDC: the changes you made to the setting are not present in the VoiceOver test plan run
20:31:28 <jugglinmike> Matt_King: I don't see "toggle button" with VoiceOver in the test queue right now
20:31:41 <jugglinmike> IsaDC: It seems like the test queue is behaving strangely today
20:32:22 <jugglinmike> Matt_King: If I go to "toggle", it shows the December 13th version in "Draft review". It shows in the "report status" dialog, "VoiceOver for macOS" is just empty
20:32:38 <jugglinmike> Matt_King: I'm going to add that to the test queue. Now, we can look at it in the test queue to see if the new instructions show up
20:33:24 <jugglinmike> Matt_King: It should be on the first test (for navigating to the toggle button under "J"--it should say "j command with single quick key"
20:34:05 <jugglinmike> IsaDC: Nope, the runner is not rendering the instructions
20:34:16 <jugglinmike> Matt_King: Okay, I guess we're blocked, then
20:35:02 <Joe_Humbert> Just FYI. I filed a new issue this morning: https://github.com/w3c/aria-at/issues/1024
20:35:40 <Joe_Humbert> Issue # 1024 since the chat blocked my link
20:38:55 <jugglinmike> Matt_King: I'm going to close issue 1019
20:39:08 <jugglinmike> IsaDC: Yeah, that was mine. And you created issue 1023
20:39:25 <jugglinmike> howard-e: Except 2023 is a change to Link, and 1021 is a change to "Command button"
20:41:32 <jugglinmike> IsaDC: It appears that I missed merging a pull request.
20:41:58 <jugglinmike> Matt_King: Ahah, that may explain the problem we've been discussing. That would be nice because it would mean there is no software bug to fix
20:42:23 <jugglinmike> Matt_King: There are no conflicts in "Toggle Button", so I should just merge this pull request...
20:43:00 <jugglinmike> Matt_King: We have to merge the "toggle button" pull request, it has to show up in the app, we have to add the newer version to the test queue, and then it needs to be run
20:43:32 <jugglinmike> Matt_King: For the folks who have already run the test, I recommend copying the output you've already collected and pasting it into the new version (rather than manually re-running the test)
20:44:43 <jugglinmike> Topic: App gbug impacting manual test running
20:44:52 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/877
20:45:39 <jugglinmike> Matt_King: We have the December 6th verison of a test plan in "draft review". We have results for some of those tests. While we were running the test, we found a problem. So we made a new version, and that new version shows up as an "r&d" versino
20:45:47 <jugglinmike> s/versino/version/
20:46:08 <jugglinmike> Matt_King: So we have three active versions of this test plan. We advanced the r&d version to draft review.
20:46:24 <jugglinmike> Matt_King: The december 13th version is now in "Draft" and the december 6th version becomes "deprecated"
20:46:35 <jugglinmike> Matt_King: But the december 6th version is still in the test queue and it has results
20:46:45 <jugglinmike> Matt_King: The December 13th version is not in the test queue at all
20:47:08 <jugglinmike> Matt_King: This is not the situation we want, because if people continue testing, we'll have to throw their work away
20:47:41 <jugglinmike> Matt_King: The bug is that whenever a test plan is in the test queue and we advance a new version of that plan to the same phase as the version that is in the test queue--well, what should happen?
20:47:55 <jugglinmike> Matt_King: In this bug report, I proposed what should happen
20:48:43 <jugglinmike> Matt_King: The first thing is that if a plan becomes deprecated, any runs that exist in the test queue for the deprecated plan should be remoeved from the test queue (not thrown away, just removed)
20:50:06 <jugglinmike> Matt_King: Then, since there were runs, the runs should still be there--we should add a new test run for the new version of the test plan, retaining the metadata from the prior version (e.g. AT version, browser version). Then, for any test for which the results would still be valid, copy the results into the new test plan run
20:50:22 <jugglinmike> Matt_King: Is this what people would expect?
20:50:41 <jugglinmike> IsaDC: That's what I already expected would happen, especially because we didn't change anything for VoiceOver
20:50:55 <jugglinmike> howard-e: I do agree with the scenario that you've outlined. That's what should have happened
20:51:23 <jugglinmike> howard-e: The issue is that I think there was a failed comparison. After these new changes, there's a part of that test which should be excluded during the comparison, but it isn't being excluded
20:51:59 <jugglinmike> Matt_King: Let's say all the tests changes and none of the data is transferrable, should we still do everything else? The deprecated version should definitely be removed from the queue, right?
20:52:28 <jugglinmike> Matt_King: It seems like there are two different kinds of failure here
20:52:50 <jugglinmike> howard-e: If every test changed, then if you transitioned a test plan version to that phase, then every tester would have zero results
20:53:06 <jugglinmike> Matt_King: But the deprecated version should also be replaced by the newer version, right?
20:53:09 <jugglinmike> howard-e: Yes, that's right
20:53:37 <jugglinmike> Matt_King: The new V2 format separates one version of a screen reader with another version very cleanly
20:53:55 <jugglinmike> Matt_King: We have an open issue related to updating the comparison algorithm...
20:55:09 <jugglinmike> Matt_King: There's another scenario, here. Let's say we had marked "final" the JAWS test for this plan.
20:55:33 <jugglinmike> Matt_King: So it's not in the test queue. But now that test plan is deprecated, and we advance a test plan from one phase to the next phase.
20:56:01 <jugglinmike> Matt_King: Should we automatically create a new draft report for any report that was previously marked "final" for that plan?
20:56:11 <jugglinmike> Matt_King: This is a harder question, especially in the world of automation
20:56:22 <jugglinmike> Matt_King: If it's in draft review, maybe this is an easier question to answer
20:58:30 <jugglinmike> IsaDC: I think the data should be copied into the new version, if there are no other changes.
20:59:08 <jugglinmike> Matt_King: If we had made a change to something that affected JAWS, should we not just keep it and also automatically add it to the test queue...?
20:59:14 <jugglinmike> howard-e: Only if it's in draft
21:02:56 <jugglinmike> [the group discusses further edge cases and potential remediations]
21:03:15 <jugglinmike> Matt_King: I'm getting a sense for what we need to do, here. I'll follow up with some more information on the bug report itself
21:03:20 <jugglinmike> Zakim, end the meeting
21:03:20 <Zakim> As of this point the attendees have been Matt_King, jugglinmike, Joe_Humbert, howard-e, murray_moss, IsaDC
21:03:23 <Zakim> RRSAgent, please draft minutes
21:03:24 <RRSAgent> I have made the request to generate https://www.w3.org/2023/12/14-aria-at-minutes.html Zakim
21:03:30 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
21:03:31 <Zakim> Zakim has left #aria-at
21:03:36 <jugglinmike> RRSAgent, leave
21:03:36 <RRSAgent> I see no action items