Screen reader testing protocol simplification

Issue 5 in the repo:

https://github.com/w3c/aria-at/issues/5

Proposing a simpler approach to how we write assertions.

<michael_fairchild> scribe: MF

<michael_fairchild> mck: in this issue, there is an example spreadsheet. It's going to be a lot easier to talk about this in concrete terms.

<michael_fairchild> mck: this is for the checkbox example

<michael_fairchild> mck: only 1 pre-condition, which is the screen reader mode.

<michael_fairchild> mck: each test is focused on a kind of element in the example (not a specific instance of that element in the example). In other words, it applies to many elements.

<michael_fairchild> mck: only two kinds of tests - reading or interaction. You are either reading or operating the element.

<michael_fairchild> mck: I categorized all of the tests as "must have"

<michael_fairchild> mck: for each test, we would have only one "must have" assertion, or only one "should have" assertion, etc. This will help simplify testing.

<michael_fairchild> mck: the "tested attributes" column lists all of the attributes that will be tested on the element

<michael_fairchild> mck: the "Test methods" column lists all of the different screen reader commands to test with

<michael_fairchild> mck: I made 3 made reading mode and 3 interaction mode assertions

<michael_fairchild> mck: result: either pass or fail, but we could have different types of failures (no support vs incorrect or misleading information)

<michael_fairchild> mck: I should have used the words, "Supports", "partial support", "no support", "incorrect support"

<michael_fairchild> mf: does anyone have questions or feedback?

<michael_fairchild> yohta: if one assertion had two attributes failing, but 4 total attributes, would it be listed as 'partial' support?

<michael_fairchild> mck: yes

<michael_fairchild> mck: I don't think results at the attribute level will be helpful

<michael_fairchild> yohta: is the importance column a subjective call?

<michael_fairchild> mck: that's subjective, but we will need to agree and come to consensus

<michael_fairchild> JF: I've looked at it and have some thoughts that I'd like to talk about later

michael_fairchild: mf: Helping me think about how we could simplify.

mf: some hesitatations ...
... If we don't list support for each attribut, will it still be possible to aggregate the support value for each attribute separately.
... dev wants to know how well a specific attribute is supported?
... is that possible?

mk: good question.
... In order to do that, we would have tester associate a failure with a specific attribute. that would be more work.

mf: something to consider.

mk: Maybe worth while to have tester mark a reason for failure that is associated with an attribute

mf: other concern is about consistency of interpretation of assertions
... maybe not clear what the expected behavior is
... e.g., for group, is it OK for group name to be announced but not role?
... writing the assertion for the attrib could be more clear than putting it on the element

mk: my goal was to bury some of the variance as unimportant.
... I do agree with the concern/issue, but need to assess value of getting that gramular.

mf: makes sense to me
... Might be useful to track output, e.g., insert+tab yielded bad results vs insert+up was fine.

mk: Was thinking that would be captured in the result notes.
... if creating actual sr bug, we'd capture that.
... do you mean speech history

mf: could be or could be a summary of that.

jf: Matt explored simplifications, and as group we are trying to figure out if a tester can do it and does the captured data support our goals.
... capturing sr output could be very valuable.
... if we do this, if for a specific command, do we capture for each one, and could get very complex.

mk: thinking we capture or summarize output only for failures

mf: if was previously passing, now not passing, how do we compare results.

This is scribe.perl Revision: 1.154  of Date: 2018/09/25 16:35:56  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

Present: Matt_King JeanFrancois_Hector shimizuyohta Michael_Fairchild
No ScribeNick specified.  Guessing ScribeNick: mck
Found Scribe: MF

WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)