IRC log of aria-at on 2019-09-11

Timestamps are in UTC.

16:05:00 [RRSAgent]
RRSAgent has joined #aria-at
16:05:00 [RRSAgent]
logging to https://www.w3.org/2019/09/11-aria-at-irc
16:05:08 [mck]
rrsagent, make log public
16:05:28 [mck]
MEETING: ARIA and Assistive Tech CG for Sep 11, 2019
16:05:35 [mck]
CHAIR: Matt King
16:05:48 [mck]
present+ Matt-King
16:05:53 [mck]
rrsagent, make minutes
16:05:53 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/11-aria-at-minutes.html mck
16:05:56 [michael_fairchild]
present+
16:06:00 [spectranaut]
present+ Valerie-Young
16:09:31 [Jean-Francois_Hector]
Valerie we can't hear you
16:14:31 [Jean-Francois_Hector]
TOPIC: Research on packages
16:16:34 [Jean-Francois_Hector]
V: Been researching 4 options. Made progress. Let's Test Harness and Test vocabulary first.
16:16:44 [Jean-Francois_Hector]
TOPIC: Test harness and Test vocabulary
16:16:55 [mck]
Discuss wiki page: https://github.com/w3c/aria-at/wiki/Test-Harness-and-Test-vocabulary
16:18:20 [Jean-Francois_Hector]
Valerie describes the document
16:19:34 [Jean-Francois_Hector]
V: Set up instructions are about preconditions (rather than instructions to perform the test)
16:21:25 [Jean-Francois_Hector]
Harness displays the setup code to be tested. This could be through a URL link.
16:23:25 [Jean-Francois_Hector]
Each test will have a set of abstract operating instructions, and hopefully we can have a programatic mapping to specific operating instructions
16:23:41 [Jean-Francois_Hector]
The role of the test harness here is to present the specific operating instructions in a clear way
16:24:41 [Jean-Francois_Hector]
Hoping that this document will help us have a shared language.
16:25:46 [Jean-Francois_Hector]
MCK: I'm hoping that we can have some sort of translation system that'd take key words from abstract instructions and abstract expectations, and translate them into specific instructions and expectations for the particular screen reader
16:26:25 [Jean-Francois_Hector]
Which one people want to see in a report might depend on the user.
16:27:06 [Jean-Francois_Hector]
E.g. if you're a screen reader developer you might want to see the specific terms related to your screen reader. But if you're a web developer, you might prefer using the more abstract language
16:27:47 [Jean-Francois_Hector]
V: Another part of this test harness system is importing the test. Hoping that we can have a format for writing tests, that we can just import into the test harness.
16:29:17 [Jean-Francois_Hector]
MCK: Anybody familiar with the accessibility conformance test taskforce's? Would they have any useful vocabulary or test structure?
16:29:41 [Jean-Francois_Hector]
(Typo: 'task force', not 'test force')
16:30:13 [Jean-Francois_Hector]
https://www.w3.org/WAI/GL/task-forces/conformance-testing/
16:30:40 [Jean-Francois_Hector]
They've developed a test format. We should investigate whether this (or any other) test format would be useful to us.
16:31:51 [Jean-Francois_Hector]
MF: It's a good standard with good ideas. With a11ysupported.io I couldn't directly use it, but we might be able to use some of the ideas from the task force.
16:32:01 [Jean-Francois_Hector]
E.g. it has concepts around atomic rules and composite rules
16:32:37 [Jean-Francois_Hector]
(typo: a11ysupport.io)
16:34:36 [Jean-Francois_Hector]
V: Is 'put the AT into reading mode' a set-up instruction? It might need to be more specific for different assistive technologies
16:35:36 [Jean-Francois_Hector]
MCK: In a 'test run' (or 'test session'), I might have a set of set up instructions, and perform different tests
16:36:03 [Jean-Francois_Hector]
It's not clear to me where the boundaries are between setup and doing the tests. It depends how we define the test.
16:37:39 [Jean-Francois_Hector]
Eg. if testing an expectation, like "the screen reader announced the beginning and the end of the menu bar", if we give very detailed set instructions, then the test because just "and press the F key".
16:38:00 [Jean-Francois_Hector]
Or we don't have all these instructions for setting up for every test, because we assume that the user knows more about how to use the screen reader
16:38:12 [Jean-Francois_Hector]
So to some extend this depends on the level of knowledge / experience we assume
16:38:52 [Jean-Francois_Hector]
We need to assume some level of knowledge, otherwise the amount of detail that'll need to be generated would sink the project
16:39:14 [Jean-Francois_Hector]
V: This concept of session, or a group of tests using the same setup code and setup instructions, would be useful.
16:40:03 [Jean-Francois_Hector]
What MCK is talking about is similar to what I'm describing as 'abstract operating instructions'. It's also similar to 'user task' is github issue 5
16:40:20 [Jean-Francois_Hector]
e.g. an abstract operating instruction could be "Operate checkbox in reading mode"
16:41:56 [Jean-Francois_Hector]
MCK: I was thinking that, at a higher level, there's a 'user task' (eg navigate to checkbox), then one or more expectations (eg that the screen reader states the name state and role of the checkbox). And that expectation would correspond to a lot of assertions. E.g. for each of name/role/state, and for each specific commands
16:43:43 [Jean-Francois_Hector]
V: A test expectation could be "are the checkbox's name, role and state announced". It'd correspond to several test assertions.
16:44:36 [Jean-Francois_Hector]
MF: It'll be important to record success as well as failure. And the speech output that is success.
16:45:04 [Jean-Francois_Hector]
This would help us improve our credibility, and help future testers reference how it tested last time.
16:45:47 [Jean-Francois_Hector]
V: It'd be great if the test harness did that.
16:46:09 [Jean-Francois_Hector]
MCK: In JAWS and VO it's possible to capture the last utterance of the screen reader. NVDA might have a plug in to do it too.
16:47:01 [Jean-Francois_Hector]
The tester could record one of the instances of the screen reader fulfilling the expectations. Not necessarily capture it for every single assertion (e.g. name, state and role).
16:47:50 [Jean-Francois_Hector]
If we structure our high level expectations that way, it'll be easy to capture how a particular screen reader fulfils an expectation.
16:47:59 [Jean-Francois_Hector]
The same utterance could cover many assertions.
16:48:48 [Jean-Francois_Hector]
V: I might start doing another wiki or issue page that records the different features that we need a test harness to have
16:49:46 [Jean-Francois_Hector]
TOPIC: Research on packages
16:51:13 [Jean-Francois_Hector]
Tuleap was very promising, but it turns out the open source version is too limited. Add-ons need to be bought. Price is per user.
16:52:19 [Jean-Francois_Hector]
There are other two that I'm looking at together: Kiwi TCMS (Test Case Management System) and Nitrate. One is a fork of the other.
16:53:01 [Jean-Francois_Hector]
The last open source test management solution I'm looking at is TestLink
16:53:27 [Jean-Francois_Hector]
TOPIC: Screen reader terminology
16:53:50 [Jean-Francois_Hector]
JF: See page here: https://github.com/w3c/aria-at/wiki/Screen-Reader-Terminology-Translation
16:54:18 [Jean-Francois_Hector]
MCK: The definitions, if they include explanations, are getting too big for table cells
16:54:26 [Jean-Francois_Hector]
So now I'm thinking of providing links to a glossary
16:55:48 [Jean-Francois_Hector]
Maybe I could include some related key commands, but it gets complicated fast. And I'm not sure whether this should be in this table or somewhere else.
16:56:47 [Jean-Francois_Hector]
E.g. There are half a dozen things that would trigger you to get in and out of reading mode. So there's a main way to force JAWS into reading mode, but in practice, in most usage situations, you don't need to use it
16:57:31 [Jean-Francois_Hector]
But having the instructions might still be useful. E.g. at the moment with JAWS you can't browse grid elements without getting out of mode manually
16:58:05 [Jean-Francois_Hector]
V: Agree that this page should be focused on the language. But might be useful to record this knowledge
16:59:07 [Jean-Francois_Hector]
MCK: I want this to cover all the knowledge we're going to use when writing expectations
16:59:20 [Jean-Francois_Hector]
Should this cover the different ways that screen reader speak ARIA stuff?
16:59:53 [Jean-Francois_Hector]
E.g. JAWS calls a menubutton one thing and VO calls it something else
17:00:29 [Jean-Francois_Hector]
V: This should cover everything that would be needed to describe a test to a tester. This would cover instructions, but also expectations
17:01:03 [Jean-Francois_Hector]
MCK: I'm debating about that one (i.e. whether or not to cover instructions), because 90% of the time test results should be obvious
17:01:45 [Isaac]
I need to take off to attend another meeting. Don't want to interrupt so leaving a message here
17:02:00 [Jean-Francois_Hector]
MF: In a11ysupport.io I record successful output, and also create an array of examples for each expectation, across different screen readers
17:02:52 [Jean-Francois_Hector]
MCK: Not yet sure about the choice of verbs. Right now we use disparate language eg 'reading' 'perceiving'. It'd be good to use words that have specific meanings, so that when you write expectations it has an unambiguous meaning
17:03:52 [Jean-Francois_Hector]
A user test is always going to include some kind of verb. Eg "Navigate to a checkbox in reading mode" is a task.
17:03:55 [Jean-Francois_Hector]
E.g. "Perceive the group label of a group of checkbox in interaction mode".
17:04:15 [Jean-Francois_Hector]
Not sure whether we should use the word "read" for perceive.
17:05:00 [Jean-Francois_Hector]
Some things are only announced in a transient manner. Eg one screen reader might only tell you when you're entering or leaving a group as you are navigating. But not when you're in that group
17:06:15 [Jean-Francois_Hector]
V: language about instructions and language about expectations might be different. eg "perceive" is more about an expectation
17:06:41 [Jean-Francois_Hector]
MCK: there's an expectation that the screen reader informs you about where you are.
17:06:53 [Jean-Francois_Hector]
MF: Would be good to revisit this in the next all
17:06:58 [Jean-Francois_Hector]
(typo: call)
17:07:17 [mck]
rrsagent, make minutes
17:07:17 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/11-aria-at-minutes.html mck
17:07:57 [Jean-Francois_Hector]
MCK: No meeting next week. Next teleconference will be on Sept 25
17:08:06 [mck]
rrsagent, make minutes
17:08:06 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/11-aria-at-minutes.html mck
18:33:58 [Zakim]
Zakim has left #aria-at