17:00:42 <RRSAgent> RRSAgent has joined #aria-at
17:00:46 <RRSAgent> logging to https://www.w3.org/2023/09/27-aria-at-irc
17:00:52 <jugglinmike> rrsagent, make log public
17:08:10 <Sam_shaw> Sam_shaw has joined #aria-at
17:08:30 <Matt_King> Matt_King has joined #aria-at
17:08:30 <jugglinmike> Zakim, start the meeting
17:08:30 <Zakim> RRSAgent, make logs Public
17:08:32 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
17:08:38 <Matt_King> present+
17:09:03 <jugglinmike> meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference
17:09:07 <jugglinmike> present+ jugglinmike
17:09:10 <jugglinmike> scribe+ jugglinmike
17:10:25 <jugglinmike> Matt_King: We have four topics on the agenda today, all all related to the app
17:10:51 <jugglinmike> Matt_King: the most meaty topic is the final one
17:10:58 <howard-e> howard-e has joined #aria-at
17:11:05 <howard-e> present+
17:11:11 <jugglinmike> Matt_King: Are there any additional topics anyone would like to bring?
17:11:20 <jugglinmike> Matt_King: Hearing none, we'll stick with these fours
17:11:38 <jugglinmike> Matt_King: the next meeting is Thursday October 5
17:12:03 <jugglinmike> Topic: Issue 795: Discuss changing title of reports page
17:12:17 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/795
17:13:03 <jugglinmike> Matt_King: We talked about chaging the title of the "Candidate Review" page, but I think that the title of the "Test Reports" page could also use improvement
17:13:18 <jugglinmike> Matt_King: If you saw that title in Google search results, it would tell you nothing
17:13:59 <jugglinmike> Matt_King: Because so much of what we talk about is about AT interoperability
17:14:35 <jugglinmike> So my proposal for that page is "Assistive Technology Interoperability Reports". In contexts where we need an abbreviated version, we could use "AT Interop Reports"
17:14:53 <jugglinmike> s/So my/Matt_King: So my/
17:15:02 <jugglinmike> Matt_King: Any thoughts?
17:16:04 <jugglinmike> present+ Joe_Humbert
17:16:40 <jugglinmike> Joe_Humbert: I like your title, and I agree it will make the page's meaning more clear when it appears in search results
17:16:48 <jugglinmike> present+ IsaDC
17:16:52 <jugglinmike> present+ James_Scholes
17:16:59 <jugglinmike> James_Scholes: I support the new title
17:17:02 <jugglinmike> IsaDC: Me too
17:17:17 <jugglinmike> Matt_King: Great!
17:17:29 <jugglinmike> Topic: Issue 738: Changes to unexpected behavior data collection and reporting
17:17:43 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/738
17:18:16 <jugglinmike> Matt_King: I'd like to prepare a better mock up so there's no ambiguity in our future discussion
17:19:07 <jugglinmike> Matt_King: But one of the decisions we made last time is that when we record an unexpected behavior, we would assign one of two severities (rather than one of three)
17:19:41 <jugglinmike> Matt_King: I was working on this because I didn't want to change from "high", "medium", and "low" to just "high" and "medium". "medium" and "low" also felt wrong
17:20:01 <jugglinmike> Matt_King: For now, I've settled on "high impact" and "moderate impact"
17:20:49 <jugglinmike> James_Scholes: As you say, "moderate" is almost meaningless because it could apply to any level impact that is not "high"
17:21:23 <jugglinmike> Matt_King: We want to set a fairly high bar for something that is "high impact."
17:22:03 <jugglinmike> Matt_King: The assertion will read somthing like "The AT must not exhibit unexpected behaviors with high impact"
17:22:19 <jugglinmike> Matt_King: And "The AT should not exhibit unexpected behaviors with moderate impact"
17:22:39 <jugglinmike> Matt_King: I'm going to move forward with those terms for now and bring it back to this meeting next week
17:22:52 <jugglinmike> Topic: Issue 689: Data integrity strategy
17:23:01 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/689
17:23:35 <jugglinmike> Matt_King: One of the things that we have to have a lot of confidence in is that all the tallies and counts and information we present in reports is accurate--and that we don't break it
17:24:12 <jugglinmike> Matt_King: When you run a report, the system is going to count up the number of passes and fails, its going to calculate percentages, and it's going to capture dates and times
17:24:24 <jugglinmike> Matt_King: There are an awful lot of ways for things to go wrong in that process
17:24:53 <jugglinmike> Matt_King: And as we transfer data to APG in the form of support tables, I wanted to ask: how are we going to approach making sure that the system doesn't produce errors?
17:25:08 <jugglinmike> present+ Lola_Odelola
17:25:26 <jugglinmike> Lola_Odelola: Through some of the work that we've already done, some of these questions have already been answered
17:25:51 <jugglinmike> Lola_Odelola: An outstanding question is: do we have any ideas for the types of queries we'd like to preform to make sure there are no anomalies in the data?
17:26:10 <jugglinmike> Lola_Odelola: What kind of anomalies would we want to check before and after a deployment?
17:26:56 <jugglinmike> howard-e: For the most part, I'd want to be able to trust that the tests that are being added--that the system catches problems with those
17:27:42 <jugglinmike> Matt_King: I added quite a long list in the V2 format--a list of checks for the format
17:27:59 <jugglinmike> Matt_King: While I was doing that, though, I wasn't thinking about how mistakes in the plan could introduce inconsistencies in the data
17:28:24 <jugglinmike> Matt_King: There are some checks like, "every AT must have at least one command mapped to every assertion" or something like that
17:28:57 <jugglinmike> Matt_King: And I have a separate issue related to being able to specify that an AT has no command
17:29:17 <jugglinmike> Matt_King: But now, I'm thinking more about the data that's on the "reports" site
17:29:56 <jugglinmike> Matt_King: For instance, the number of assertions which have verdicts--that number shouldn't change after a data migration
17:30:34 <jugglinmike> Matt_King: I think it would also be important to check that for subcategories of the data (e.g. the total number of reports generated from recommended test plans, the total number of recomended test plans)
17:31:11 <jugglinmike> James_Scholes: Are we talking about validating user input? What are we validating against?
17:31:44 <jugglinmike> Matt_King: Against the data before the deployment. This is about ensuring that we maintain data integrity during deployment operations
17:32:04 <jugglinmike> Matt_King: Maybe we need to enumerate the scenerios where we believe data integrity could be compromised
17:32:22 <jugglinmike> Matt_King: I'm assuming that when you save and close in the test runner, that some checks are performed
17:32:33 <jugglinmike> James_Scholes: To what extent are checks like that already present?
17:33:10 <jugglinmike> James_Scholes: for example, during a test run, when a Tester provides results for a single Test, I assume that when they save those results, checks are made to verify that the data is correctly saved
17:33:49 <jugglinmike> Lola_Odelola: I think that part of the issue here (as Matt mentioned earlier), this is an oldish issue, and in the time since it was created, there have been a lot of improvement to the code and the processes
17:34:58 <jugglinmike> Lola_Odela: Now, what we want to identify is: are there scenarios that could cause inconsistent data? We're asking because we have seen inconsistent data in situations we didn't expect
17:35:09 <jugglinmike> s/Odela/Odelola/
17:35:33 <jugglinmike> Lola_Odelola: I'm happy for this to be put on the back burner until something happens in the future where we need to readjust
17:36:17 <jugglinmike> Matt_King: Okay, though I'm still interested in building/documenting a fairly rigorous set of checks to ensure data integrity before and after deployment
17:36:33 <jugglinmike> Matt_King: That is, any time we make changes to the data model
17:36:59 <jugglinmike> Matt_King: For instance, when we do the refactor, we want to make sure that the total number of assertions doesn't change, et. cetera
17:38:06 <jugglinmike> James_Scholes: I'm not necessarily advocating for tabling this discussion, but I do believe that we need to have the existing checks documented before we can have a constructive conversation on the topic
17:38:18 <jugglinmike> Lola_Odelola: That makes sense. We can put something together for the group
17:42:44 <jugglinmike> Topic: Issue 791: Generating reports for specific AT/Browser versions
17:42:53 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/791
17:43:11 <jugglinmike> Matt_King: Right now, everything we've been dealing with is reviewing drafts and getting them to candidate
17:43:29 <jugglinmike> Matt_King: We haven't cared too much about browser and AT versions; we've just recorded what Testers have been using
17:43:44 <jugglinmike> Matt_King: That's fine until we reach the "Recommended" phase of a report
17:44:01 <jugglinmike> Matt_King: At that phase, we're reporting to the world how this software works
17:44:28 <jugglinmike> Matt_King: The expectation we've had from the beginning is that once we have a "Recommended" test plan, we're going to keep that data current indefinitely
17:45:38 <jugglinmike> Matt_King: At that point, it becomes important for us to re-run the reports when each AT releases a new version and (perhaps to a slightly less critical extent) when each browser releases a new version
17:46:10 <jugglinmike> Matt_King: So, this can be a little bit challenging, and there are a lot of decisions to make in this space
17:46:16 <jugglinmike> Matt_King: Let's talk about screen readers, first
17:46:33 <jugglinmike> Matt_King: Right now, when we add a plan to the queue, we don't specify a screen reader version on purpose
17:46:49 <jugglinmike> Matt_King: (We had this ability three years ago, but we removed it because it was causing problems for us)
17:47:06 <jugglinmike> Matt_King: We still want to say "any recent version is fine" for draft and candidate
17:47:20 <jugglinmike> Matt_King: But when we reach "recommended", we need to enforce specific versions
17:47:51 <jugglinmike> Matt_King: One problem is that we don't have a way in the test queue that a specific version is required (that seems pretty easy to solve)
17:48:18 <jugglinmike> Matt_King: It also means requiring Testers to confirm that they're using the appropriate version
17:49:18 <jugglinmike> Matt_King: Thirdly, the Test Admin needs to choose a specific version make new Test Plans
17:49:28 <jugglinmike> James_Scholes: that all seems right to me
17:49:34 <jugglinmike> Matt_King: Okay, moving on to browsers.
17:50:41 <jugglinmike> Matt_King: All of this gets so much harder because (1) there are a lot more browsers, and (2) because we will need to maintain a list of browser versions, and (3) a lot of people don't have control over the version of the browser which is installed on their system
17:51:39 <jugglinmike> James_Scholes: for regular users who are not using corporate machines, it is quite difficult to install a specific version of a browser (first because they aren't always published--as in the case of Chrome). Secondly, even if you manage to do that, it's probably going to update itself, anyway
17:52:19 <jugglinmike> James_Scholes: I think that's separate from the "enterprise problem" where Testers are working in a corporate setting where there is strict external control over the applications installed on their machine
17:52:49 <jugglinmike> James_Scholes: I think it's really difficult to ask people to use a specific browser build. Even requiring just a major version is tricky
17:53:02 <jugglinmike> Matt_King: But we could ask machines to do this
17:53:34 <jugglinmike> Matt_King: I'm thinking about a decision that we could make. For the time being--until we have full automation across the board--we only ever require specific AT versions...
17:53:53 <jugglinmike> Matt_King: ...that we TRACK the browser versions (just like we do now), but we don't require it
17:54:30 <jugglinmike> James_Scholes: If we say that, "oh, we don't want someone testing with a very outdated version of Chrome", I think that may be a good middle ground
17:55:22 <jugglinmike> James_Scholes: Because if we go super-specific on browser version requirements, we'd probably also have to requirements on operating system versions
17:57:41 <jugglinmike> jugglinmike: We might want to track version release dates as well, because "four major versions old" means something very different for Chrome and for Safari
17:58:48 <jugglinmike> jugglinmike: The kind of warning we're discussing (telling Testers that they're browser is suspiciously old) will be hard to express for a single "version delta" across all browsers. Expressing it in terms of release date will be more meaningful
17:58:55 <jugglinmike> Matt_King: I agree!
18:00:04 <jugglinmike> Matt_King: So for the foreseeable future, we will not require updating the data for Recommended Reports in response to the release of new browser versions. We will only require updating in response to new versions of ATs...
18:00:18 <Joe_humbert> Joe_humbert has joined #aria-at
18:00:30 <jugglinmike> Matt_King: And in that case, we will accept "any recent" version of the browsers under test
18:00:47 <Joe_humbert> Present+
18:00:50 <jugglinmike> James_Scholes: sounds good
18:00:56 <jugglinmike> IsaDC: sounds good
18:01:08 <jugglinmike> Lola_Odelola: sounds good
18:01:58 <jugglinmike> Matt_King: Okay, that's good. This helps a lot. I'll update my work to consistently use the terminology "any recent browser"
18:02:30 <jugglinmike> Matt_King: Thanks, everyone!
18:02:36 <jugglinmike> Zakim, end the meeting
18:02:36 <Zakim> As of this point the attendees have been Matt_King, jugglinmike, howard-e, Joe_Humbert, IsaDC, James_Scholes, Lola_Odelola
18:02:38 <Zakim> RRSAgent, please draft minutes
18:02:40 <RRSAgent> I have made the request to generate https://www.w3.org/2023/09/27-aria-at-minutes.html Zakim
18:02:47 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
18:02:47 <Zakim> Zakim has left #aria-at
18:02:50 <jugglinmike> RRSAgent, leave
18:02:50 <RRSAgent> I see no action items