<zcorpan> https://github.com/w3c/aria-at/issues/145

Test format refactor (Issue 145)

<spectranaut> I'll explain: https://github.com/w3c/aria-at/issues/145

1

Valerie: Felt some need some change in test format.Detail in the issue above.
... Refactor will be easy to update Jon's script. None of the information in the test file is preserved in a different place, that is the goal.
... My proposal is to move all js and replace them with json file.
... So the information and the key to the information is the same, but should be much more easier to consume the test in the runner.
... Within the issue, I outlined three things to be changed with this porposal.

Matt: Does this create any limitation in the future regarding what you'd be able to include in a test plan?
... For example, we need discuss mode based on feedback from apple.

Valerie: I don't think that'll limit us that much. The original design was mainly because that was my initial idea. My assumption was that I thought if one part was made in javascript, I thought I needed to be consistent.

Matt: The way you gonna consume the test is that what will happen when you make the test? Will the script itself become json script?

Valerie: So far, it's not what I'm doing. Writing and performing the test is identical in this refactor. For script, I'm suggesting to keep the script in the test file.

Matt: Jon's script is already doing what I was thinking. So it's makes sense.

Valerie: I think this would make things easier in the long run. The only reason why now I feel make this change is to apply this concept. I feel like we'll eventually face this challenge, and this would be the good time. I need metadata in the long run, and json script is much more easier to retrieve the data.

Matt: Fixing issue of extracting the data by javascript makes sense.

Valerie: The relationship between the name of AT and the key you'd use to open that test(such as VoiceOver/MacOS), that infomration is in test.json.

<zcorpan> https://github.com/w3c/aria-at/pull/139 is the PR with the support file

Matt: Each AT is going to have these attribute about what it is, and how we categorize it.

Valerie: Essentially, this support file tells you what SR this test support.As we add support for new SR, file name needs to be updated. So I also have sub-category that cover new AT.
... It's gonna be slowly changing.When you're consuming test, all you have is directory name.
... Any questions/suggestions for the refactoring ?

<zcorpan> https://github.com/w3c/aria-at/issues/145

Valerie: In the issue 145, there's some list of changes I need to make change, and wondering if anyone take over some of the work.

Jon: I can update the create-tests.js file.

Simon: Why do you want to have json in the file itself, rather than separate file?

Valerie: I think it's also possible.We can discuss in the issue, since there's some concern.
... It's a refactor that I need to happen in the next few month.

Matt: 2 questions. 1.Test reviewer needs to go in Jon's script.I'm curious about this demodulization. I was looking Jon's script, something which we eventually replace.

Valerie: I can update that.

Michael: I guess it's not clear to me what's involved in that file.I'll take a look into the issue.

Matt: Let's discuss this again next week after investigating the issue.

Valerie: Yes.I want to come up with consensus for this around end of April for next step.

Jon: What you have in Issue 145, is that format you want?

Valerie: Yes

Jon: These help link are informative right?

Valerie: Yes, that's copied over.

Jon: This maybe something needs to be changed now.

Matt: Last week, we had a discussion that you shouldn't manually code links.

<zcorpan> https://github.com/w3c/aria-at/issues/138 - Ability to filter for roles, states, properties

Matt: We need to talk about how we want to reference stuff in APG. We don't have unique key in APG example.

Valerie: That's somethign we gotta figure out soon. So I can add another key and property.

Matt: Valerie, would you be able to spearhead that issue,and assigne someone if needed?

Valerie: Yes.

Simon: I'll create the issue.

Jon: Do you wanna have the author information?

Matt: In the APG, we didn't have attribute information because everything by community contribution.
... I don't think we need attribution metadata.

Valerie: Okay, we can remove that data.

Jon: For the creation script, I think change is trivial.
... Try to look at other scripts later this week.
... Can you merge the PR to solve the bug of test creation?

Michael: We need to fix the setup script before merge.

Wireframe

wire frames

<isaacdurazo> Wireframes: https://github.com/w3c/aria-at/issues/132

Matt: Your description of the UI was awesome.

Isaac: I saw some comment from Yohta and Michael, is there any other comments?
... Yohta's question:What is conflicting question?

Matt: My thinkign for resloving conflicting results.
... If we had conflicting results, there are several reasoning behind that.
... I wanna make sure all the testers go through and get the same results, as opposed to be persuaded.

Joe: From the scaling perspective. What if two people had valid opnion, which both seemed valid.

Matt: I expect we don't want to exactly compare record of SR output.

Joe: For future scalability, perhaps we might want to SR novice users.
... Will you have a conversation to decide which one is right, when you have 3,000 tests.
... If there's rejection,there's retest?
... It's one way to avoid conflict in a limited time. I think we should consider this scalability for future.

Michael: I like the idea.

Jon: For testers to understand why the test generating different results is useful.

Matt: Another possibility for conflicts is we facilitate the conversation by automatically collecting the data and stuff them into a github issue, where other people can jump in.

Jon: Bring in other people to come up with consensus sounds like a better way.

Michael: I agree, but I think we need to time-box that so that it won't harm the speed of publishing results.

Simon: We wouldn't delete the raw result

Matt: If two testers come up with diefferent results, it give me the chance to go back and review. It allow me to "publish conflict to github" that show automatically publicize the discussion.
... That public discussion can lead testers to rerun the test, so that we won't have conflicts, after github discussion.

------------------------------------ Simon:I like everything you said, except the fact that we don't need to go over the entire tests.We can pick the test that testers didn't agree.

This is scribe.perl Revision: 1.154  of Date: 2018/09/25 16:35:56  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Present: spectranaut zcorpan shimizuyohta isaacdurazo Matt_King michael_fairchild Joe_Humbert jongund
No ScribeNick specified.  Guessing ScribeNick: shimizuyohta
Inferring Scribes: shimizuyohta

WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.


WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)