<Jemma__> scribe?
I can scribe
<scribe> scribe: michael_fairchild
<Jemma__> present_ rob-fentress
Matt_King: we are at the end of
Bocoup's contract
... our plan is to work on a v2 in the second half of the
year.
... our goal was to get the runner to a place where the CG can
use the app in a minimal capacity
Val: we still need to deploy the reports page, and the tests from ARIA-AT will be imported to the staging and prod servers on a regular basis
<spectranaut> Here is the application: https://github.com/bocoup/aria-at-app
Matt_King: the app itself is still in a bocoup repo. We will work on moving to the w3c org
<spectranaut> https://github.com/bocoup/aria-at-app/blob/main/docs/local-development.md
Val: There is a docs directory, with details on local development and the database
<spectranaut> https://github.com/bocoup/aria-at-app/blob/main/docs/database.md
Val: it's a node react app, using
Yarn as a package manager
... to set up and run locally, follow that documentation
(details on scripts and yarn commands)
... if there are new tests from ARIA-AT, you will need to
manually run a script to get the newest tests. This is in the
documentation, and needs to be done for local development
(questions about differences between aria-at-app and aria-at. aria-at-app is the production runner and aria-at is the test repo)
Val: structure of the app: front end is under 'clients', backend and API is under 'server'. We have instructions for managing the database.
<zcorpan> https://aria-at-staging.w3.org/
Val: there is a deploy directory as well with a readme. There is a script for deploying to staging. We also have a vagrant set up to test locally if you don't have access to the actual servers.
Jemma__: what caused the error that I ran into?
zcorpan: that was a bug in the ARIA-AT app
Val: thank you for finding that bug
Seth: In the long run, we want to transfer the ownership of the app from Bocoup to the CG and the W3C.
<Jemma__> app was very stable in core functions.
Matt_King: That's the key, this is a community group owned resource.
<Jemma__> Thanks for brining the fund for this project, Matt
<zcorpan> https://github.com/w3c/aria-at/issues/162#issuecomment-642199169
<Jemma__> s/bringing/bringing
zcorpan: today I worked on going
through all of the issues that were filed, and closing
duplicates. There were a few that were very similar issues, but
I didn't close those.
... there are 52 open issues, and I added a label for the
tests
Matt_King: that's good, thank you
zcorpan: 47 were about the tests or the test data. 4 issues were about the test runner.
Matt_King: do we want to be able to distinguish if an issue is related to the runner (aria-at-app) or the tests (aria-at)?
zcorpan: I think that makes
sense.
... and there is one issue about the process. We should make
sure that we respond to issues in the timely manner during a
test cycle.
... Seth and I have been working on a document today that goes
through our findings for the pilot. It's not quite ready
yet.
... I expect more test runner issues to be filed
... any questions?
Matt_King: thank you for going through all of those issues
rob-fentress_: sorry if I logged too many issues
zcorpan: no, not a problem. Thank
you for logging them.
... Michael found a bug in one of the Jaws tests that prevented
completion of the test and you made a fix with a PR
... the problem is that we won't be able to complete the pilot
test in the way the app currently works, since we are fixed to
a specific commit from the ARIA-AT (tests) repo
Matt_King: that should be one of our top priorities for v2, to allow us to pull in changes mid cycle
Val: you can still complete the run, but that skipped test will just be missing
rob-fentress_: is the plan to redo all of these tests now that we have done the pilot?
zcorpan: I think it makes more sense to address the issues we found before we start a new cycle
Matt_King: yes, there were a lot of issues in the tests themselves. We should also capture the lessons learned from those, so that as we create and review new tests, we don't repeat those same issues.
zcorpan: I don't think you need to wait for renewed work on the aria-at app to start a new test cycle
Matt_King: right, we didn't run
into any show stopper issues in the runner itself
... one show stopper was the limit on assertions, which we
already have a PR for
zcorpan: and we have been fixing
bugs in the runner which will be deployed soon
... I want to learn about how much time people think they spent
working on the pilot tests
jongund: It was slow at first, but it took about 40 to 50% less time after the first cycle. so I think it took about an 60 to 90min per run.
rob-fentress_: that's about what it took me to run the checkbox tests
(that should have been Joe_Humbert_ )
rob-fentress_: it was about 90min per test run for me
Matt_King: (to testers) your experiences are very helpful in knowing how we should approach onboarding and training
Jemma: I did checkbox for nvda/ff and it took me about 2 hours because I had to restart the test. So it only took me 40min for nvda/chrome. Menubar for nvda/chrome took about an hour.
Matt_King: I gave Jemma some feedback about how to interpret the word 'convey'
rob-fentress_: VO provides screen reader instructions for the focused element after pausing for a little bit, and it wasn't clear if I should use that information while determining if assertions were met
Matt_King: we should document and share with all of the testers what our expectations are regarding that additional output
Joe_Humbert_: one of the things that I found difficult was capturing output from JAWS. For one thing, non-printable characters were copied, and switching between the history in JAWS and the browser caused problems sometimes.
<Jemma> That is a good feedback, Joe.
Matt_King: I mentioned to several of the screen reader developers that it would be great to have a command to copy the last utterance and to append the last utterance to the clipboard
Joe_Humbert_: I also had one test that resulted in a conflict. How you navigated to the checkbox (arrow keys vs tab) changed what was conveyed, and could cause issues.
jongund_: this was the command to read the current item, but how you navigated to the form control affected what the read command said
Matt_King: are those conflicts still there?
(the last jongund_ comment should have been Joe_Humbert_)
Joe_Humbert_: yes
<Jemma> One minor feeback for chrome and nvda testing is that once the example was opened and the button is grayed out so I cannot open the example window again. The case of reopening example window is when I lost the example window by clicking external link accidently.
zcorpan: after the new deploy that will happen after this meeting, the new teams in the w3c org will need to be used to sign into the app. We will need to invite everyone from the old Bocoup team to the new W3C team
Zakim: that will happen later today
<Jemma> yes. I understand what Simon said.
Matt_King: to make sure that this is very clear to everyone. If you used the app during the pilot, you were added to a Bocup team. That wasn't ideal, so we created similar teams in the W3C org.
This is scribe.perl Revision of Date Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00) FAILED: s/bringing/bringing/ Present: spectranaut Jemma__ Joe_Humbert rob-fentress jongund Found Scribe: michael_fairchild Inferring ScribeNick: michael_fairchild WARNING: No date found! Assuming today. (Hint: Specify the W3C IRC log URL, and the date will be determined from that.) Or specify the date like this: <dbooth> Date: 12 Sep 2002 People with action items: WARNING: Input appears to use implicit continuation lines. You may need the "-implicitContinuations" option. WARNING: IRC log location not specified! (You can ignore this warning if you do not want the generated minutes to contain a link to the original IRC log.)[End of scribe.perl diagnostic output]