<zcorpan> start meeting
<zcorpan> meeting: ARIA-AT one-off discussion on test automation
<zcorpan> scribenick: jesdaigle
<zcorpan> https://github.com/w3c/aria-at/issues/349
<zcorpan> https://docs.google.com/document/d/1nCzxRdufL1l_6WreHJjzSZcdF4lKTGRqMnBwr4noEDk/edit#
Simon Pieters: This week changed custom text format to json based on feedback from NVDA. Changed how output is collect. Two new commands. Press until and press until role that translates ARIA role to specific string.
Simon updated design doc to reflect the changes and documentation in NVDA. Is working. Only one test right now. May be more bugs as we add more tests.
James: Two new commands Press
until contains, what is the input? What is it looking for?
Simon: If you use contains you can look for anything that is
part of the output. Substring match. Most commands are
substring matches. Role does a conversion from ARIA role to AT
specific string for that role.
... could there be variations? Simon: there is no way to do
that. If NVDA changes their output the tests should be changed
to match.
... balance between tests that truly fails and false failures.
Will it be versioned? Simon: currently the translation lives in
NVDA repo. Test runner could look for both variants.
... it would be ideal if ARIA AT could drive different readers
and carry out the tests.
Simon: should be possible to automate VoiceOver with Apple Script.
James: How does the system go
through the test? Simon: the only new logic is in a loop, press
a key and collect the output until the system thinks the output
is complete and look for substring. If substring found loop
stops. Continues if not.
... How does it execute? Simon: There are two different
commands. Two loops. Use one or the other. James: Role, name,
state. Simon: Depends on where you navigate.
Simon: existing test for combobox, press x until reach combobox. All accumulated output is in the string.
<michael_fairchild> https://docs.google.com/document/d/1YxHAf6r3E2RG4REaYcmM14DuxfBgP4XoWRdfSk5aAQM/edit#heading=h.n40b82cyf9d8
<s3ththompson> scribenick: s3ththompson
James: can we keep comments in one place (GitHub)?
Simon: having repo root in tests would make it possible to change our minds about where files live later
James, Michael: that makes sense
James: prioritize adjacency
Simon: (summarizing Michael
comment) could add argument about how many times
`assert_contains` matches
... (summarizing comment) how is `assert_accname` different
from `assert_contains`
... it's not now, although it could be if it knew internal
state of AT (but we don't have internal support of vendors
yet...)
James: in general when we want to do things ourselves, can we run things as addons (rather than having nvda integrate it)
simon: yes, would probably be possible given how the python modules work
James: as an aside, i agree with reef from nvda about reading internal state...
simon: certainly we don't want to *only* read internal state. we could do both
michael: i like the idea of translating a role to a specific AT, but we can't just look at role value... have to consider context too
simon: what about
`assert_state_or_property`?
... if consensus is that option role should not be conveyed...
that's testable.
james: would this human also
understand nuance about when option role should be
included?
... e.g. do we consider hint text as important?
... is it possible for each screen reader to use different
assertions? we've been under the assumption that the answer is
no, but that could change
michael: other example was role mutation based on context: aria-pressed will mutate from button to toggle-button
simon: if it's a one-off we could
use assert_contains... or have a special role for toggle button
even if not standard per arai
... do we need to enumerate everything for
assert_state_or_property? (answer from group: probably
not)
... comment from weston thayer: How long does the test runner
wait before forwarding the next command?
james: this is a comment question
on nvda forum... the answer is usually "you don't." it's not
really possible in the wild because speech engines are all
different. but maybe in our tests we could pick one engine and
know
... (or a custom speak engine...)
... 17 instances of role with modifier... not sure if all of
them cause role mutation, but maybe more than we were
thinking
... (from core a11y mappings doc)
simon: so is the implication we don't have `assert_role` at all? or do we design it differently?
michael: my take would be leave
it as is and follow up after "beta" version
... weston brought up another good point: if we intercept an
output and nvda sends multiple chunks after that output, how
can we know for sure in the runner that speech output is
finished before we move on to the next test
... we really need to think about that more
james: for this and other
questions, nvda may have input on how this should be
designer... it's all fairly new
... need to continue convo about how test writing is
organized... csv files... reusable assertions, etc.
simon: my assumption is definitely that how we write the csv manual tests will need to be updated once we finish designing automated format... but that can come later
<zcorpan> thanks s3ththompson for scribing!
<zcorpan> also thanks jesdaigle for scribing :)
This is scribe.perl Revision of Date Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00) Default Present: zcorpan, jesdaigle, s3ththompson, michael_fairchild Present: zcorpan jesdaigle s3ththompson michael_fairchild Found ScribeNick: jesdaigle Found ScribeNick: s3ththompson Inferring Scribes: jesdaigle, s3ththompson Scribes: jesdaigle, s3ththompson ScribeNicks: jesdaigle, s3ththompson WARNING: No meeting chair found! You should specify the meeting chair like this: <dbooth> Chair: dbooth WARNING: No date found! Assuming today. (Hint: Specify the W3C IRC log URL, and the date will be determined from that.) Or specify the date like this: <dbooth> Date: 12 Sep 2002 People with action items: WARNING: Input appears to use implicit continuation lines. You may need the "-implicitContinuations" option. WARNING: IRC log location not specified! (You can ignore this warning if you do not want the generated minutes to contain a link to the original IRC log.)[End of scribe.perl diagnostic output]