<scribe> scribe: michael_fairchild

<zcorpan> https://github.com/w3c/aria-at/issues/358

<zcorpan> scribenick: michael_fairchild

App dev update

Jes: we did some end to end testing of the app this morning. We uncovered a few minor issues and are working on a few final fixes.

MK: When do you plan the deployment to the production server?

Jes: all of it will be done at the end of the current sprint (end of the week)

MK: great. so the issues that you found, what work flows do they affect?

Jes: we found an issue with draft reports where you can get stuck on the loading screen.
... Another issue around the behavior of the confirmation modal on the configure active runs page.
... I believe that is when you are brining in more test runs
... The other one is the published result behavior is a little inconsistent. The results row can disappear and reappear on refresh.

MK: That sounds like we are very close here
... we discuss the app more later.

Sina: huge thanks for all of the accessibility work and design considerations

Jes: Ditto, we are just as appreciative

MK: I'm just loving the preview functionality

Sina: Should we discuss the issue with previews?

MK: Yes, the issue is that the previews are not always being regenerated

Jes: yes, its being worked on as we speak

Test writing update

Sina: the good news is that we met our goal. We have the 6.

James: one commit away from our goal
... we are also taking some time to write some test plans for other patterns (grid, tabs, etc).

Sina: that gets us to 10 test plans and 6 pull requests

MK: That's awesome.
... I haven't looked at all of it, but did a detailed review of the disclosure. It was hard to find anything wrong with it.
... I want to talk about reviews of your work and merging the PRs. We will do that when we discuss the working mode.

Sina: For everyone else, please feel free to review the PRs and provide feedback

MK: I do want to encourage people to look at the pull requests. It does take a bit of time to review them, but with the new preview capability it should go pretty quickly.
... The preview link opens a page and you need to choose the correct pattern from the table. I use the review link, rather than the index link.
... I wonder if we could provide the direct link to the review page in your top comment.

Sina: the link to what?

MK: the deep link

James: I can add it to the pull rquest

Rob: I mentioned that I might try to write a test plan, but haven't had a chance. I might have some time in the next few weeks.

James: Button is a good one. Nice and simple to get used to things.
... if you are working on developing tests, we have some templates and insights we have written down over the past few weeks. If you get stuck on anything or just want to chat, let me know.

MK: Rob, if you have any time to look at the pull requests that would also be helpful

Rob: Yes, I'll try to do that

James: the good news is that you now have more data to work with, because there are the two completed tests plans, plus all of the pull requests.

Rob: Great, thank you

Automation update

Simon: last week we had an extra call about the automation test format. Issue 349.
... changes since last week: I changed assert_checked to assert_states (missed the exact name) so it is more generic
... fixed press_until so it looks for the document boundary
... in issue 349 there has been a lot of good comments from everyone that is interested in the automation work. That has been very helpful.
... I'm working on writing a report with findings and recommendations for future work. We might also make a video that describes how automation works and how it differs from manual testing.

MK: I'm very excited about the report and video because I want to brag about what you have done.

Simon: I think if we get it working with all of the screen readers that we are interested in, it will let us scale

MK: When I was reviewing tests, I saw perfect evidence why this is needed badly
... Is the work you have been doing to integrate this into the NVDA CI part of this?

Simon: it's working in our fork. I'm not sure if it makes sense to merge our prototype into their repo

MK: how long do you think it will be before it is mature enough for it to be merged into their CI?

Simon: I think it could be done Q1 of next year

MK: any other questions?

Working mode changes

MK: with all of these awesome app changes, I was thinking that we might want to revisit the working mode
... part of the problem is that the pull requests are interdependent, and might result in merge conflicts
... That made me think a bit more about our working mode where we assumed that once a test was merged to master we don't review it and think its ready to run
... now we have the ability merge tests without losing data
... so we could just keep merging in new versions as we want until we finish the review and have the data that we want
... Then we could mark it as a draft and then bring the screen reader developer to review
... So we can agree on assertions and results at the same time and speed up the process
... What do people think of working in this way? Merge to master as quick as possible, do a quick review, and the full community group review would happen by people actually running the tests

Sina: I agree with that, especially from our end because our tests are reviewed by 3 PAC team members

Jon: this would work for me

Rob: do you mean running it locally?

MK: No, not locally. On the server.
... the first one would likely be the disclosure plan

Simon: would it be a nice feature to be able to select a commit from a particular branch in the app, so you can test from a PR before you merge?

MK: we currently only list the commits in master

Sina: the problem with that is that we want to highlight the need to merge quickly

Jon: +1 to that, we don't want to waste a lot of time trying to get things back in sync

MK: I don't think there would be a huge advantage of testing from a branch

Simon: I was thinking mostly as a preview

MK: You can actually do that today as part of the preview process today, using aria-at app

Simon: interesting

James: it's pretty great because you can do it locally or as part of the preview

MF: +1

MK: For good tests, that means we have everything that's needed for automation
... It sounds like people are okay with moving in this direction and at least experimenting with it.
... in the meantime, I'll propose some revisions to the working mode document
... After I've merged a couple of these, I'll update the app. Next week probably. Then, I can send an email to the list.

Issue 358 How to state state within a test

Simon: this is about how we want to reset states within a test
... if we take the checkbox test, the instructions are to navigate to the checkbox using different commands. But as a tester, if you want to navigate to the checkbox after you have already navigated to it you need to either reload the page or navigate somewhere else on the page first

MK: By state you don't mean the state of the checkbox, you mean the state of the test/page
... I was running into some problems with this recently

Simon: the approach that I've take so far is to use h/shift+h to navigate to the heading before the checkbox. But we don't have instructions for this and testers can be doing different things.
... I think we need to decide what the correct way should be and use that in the automated tests

MK: The setup script for the disclosure currently sets focus to a link before or after the disclosure. It seems to me that one approach could be to re-run the setup script.
... when you re-run the page, screen readers don't always behave the same as if you loaded the page for the first time. I had to close the window and re-open it to get it to run.
... do we know 100% that it doesn't run? Or is JAWS remembering where you are?
... Could we put a button on every page that says 'run setup' or a 'reset'

Simon: that's my proposal

Sina: I like that

MK: what if that button also had a key assignment
... Maybe the instructions are always the same.

Simon: that would be easier from an automation standpoint

James: do we need any sort of teardown script?
... there is one where we hide something via display:none
... we could also have a sequence of setup scripts

This is scribe.perl Revision of Date 
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Present: Matt_King jesdaigle zcorpan rob_fentress michael_fairchild
Found Scribe: michael_fairchild
Found ScribeNick: michael_fairchild

WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)