Protocols and Formats Working Group Teleconference -- 10 Dec 2015

<MichaelC> meeting: ARIA

<jamesn> scribe: jamesn

<MichaelC> agenda order 1, 2, 3, 7

Publications update

MC: main thing is that make sure everyone knows

everythign published recently

FPWD of some

updated of all the rest...

talking timeline with Rich think would like to do an updated WD of 1.1 (pseudo last call)

early feb for that

close out remainder of issues between now and then

if we stick to timeline CR early april

just after CSUN

some of the other specs may advance to CR at same time. Core may but others may not

push to get 1.1 done is there. want to shift focus to 2.0

one of the issues is the piece of the puzzle for extended descriptions. nexts weeks call will focus on thaty

want to get this done b4 the holidays so people working over holidays can do stuff

Working Group transition update

MC: reminders. ARIA group exists. IEs can now join. nearly everyone here as member org has joined or is talking

PF mailing list will be for rest of this month. will be migrating tracker issues

there will be a tracker cutoff too

PF tracker needs to split issues and actions between apa and aria

if issues appear in new tracker b4 migration then may make it harder

try to avoid using them until january

Mailing list subscription policy update

<MichaelC> https://lists.w3.org/Archives/Public/public-aria-admin/2015Dec/0000.html

was a CFC on the aria and PF lists to change the policy for the public-aria list

already the case that members can post but if they want to subscribe they can do so and agree to the same process requirements that others accept

there was dicsussion on the PF list that looked like it was diverging from consensus

looks like that has moved so that we now have consensus

want to make it official so the chair needs to ratify the consensus on list

aria-primaryaction discussion

<MichaelC> scribe: MichaelC

JN: discussion on list, implementers seem to not support

want it to work differently

my concern is the proposed changes move the difficulty to frameworks and authors

rather than have the browser address

js: primary button on @@

jn: problem is it just points to @@

js: AXAPI might have problem

because have to search up the tree to find the button

the others might not have a problem

jn: worried this is hard for authors

js: I´m ok either way

others might object to non idref version

mb: agree shouldn´t be idref

developers don´t want to bother with idrefs

eventually this should be in browser

so shouldn´t bother developers now

jn: discussion moves too quickly

mk: some value for dialogs only

not sure as important for other contexts

jn: wizard isn´t always in a dialog

js: they don´t have roles...

mk: unsure of value of that for screen reader users anyway

I don´t trust default button, so I look around

jn: because it doesn´t tell you, but if it did you might trust it

mk: even so, not as high value for me

jn: mostly agree

it´s just a button attribute

mk: could be boolean on button, forget about context

then would need to drop language about author error

mc: there was some confusion, would a rephrasing on list help?

mk: need clearer statement of goals for screen readers

js: AX has a dialog, with a default button

though the button itself doesn´t have anything to make it the default

if you put aria-default on the button, the browser has to figure out what the context is that it´s the default for

that´s a pattern they have opposed elsewhere, for aria-current

mc: so now there are two things with similar problems

js: difference is that with default button you need more information about what´s gonna get dismissed

mk: @@

mc: for aria-current, we considered aria-currentfor, but rejected it

mk: is there a way of not having to go up the tree?

just say we care about the button, is a default, less important to know what it´s for

jn: sighted user doesn´t know, they infer

mc: they infer pretty strongly

jd: for end user, context isn´t that important

but for AT, on some platforms, there´s a ¨tell me the default button¨ command

having to go up the hierarchy isn´t so bad

but having to go *down* the hierarchy is much more expensive

user knows they´re in a dialog etc.

but the application has a much wider context

mb: there are only so many things the default button could be in

jn: maybe this can be solved in ARIA 2.0 with selectors

js: author has to know what to select

jn: they will know

<Zakim> MichaelC, you wanted to say take advantage of API

mc: even if we´re not as concerned about what AXAPI offers, is nice to support the greater richness where available

<Zakim> joanie, you wanted to say that Matt's suggestion works *if* we say the primary button MUST be in the context of a dialog

jd: Matt's suggestion works *if* we say the primary button MUST be in the context of a dialog

<Zakim> jamesn, you wanted to say wizards dont have role=dialog and cannot

jn: can´t have wizards have role=dialog

they have differences

you can leave them etc.

<joanie> +1 to getting it right in 2.0

mc: to summarize, sounds like support for doing well in ARIA 2.0 rather than badly in ARIA 1.1

so adjust issues accordingly

and inform the thread of this

Test planning (major topic for this call)

<scribe> scribe: jamesn

MC: Rich wanted in particular for MC to lead this discussion.

due to role in aria test harness

has been discussion in various other groups.

<MichaelC> Draft exit criteria

for MUSTs in 1.0 needed a testable statement

for most part was role, states and properties

what we care about is that is makes its way to the API so we depended on the UAIG - now the core and other mappings

says for the feature - how it should look in the apis

sometimes simple, sometimes more complex

so for ARIA testing means we need to test the mapping guides - then set some sontraints that need to find 2 implelmentations of each feature

for the mappings there may only be 1 implementation of a mapping - sometimes there is only 1 mapping for an API

yes - fred also test that

so find 2 instances of some mapping. so if find 1 on windows and 1 on mac then these are the 2 mappings

if cant find 2 on 2 platforms then look for 2 browsers on the same platform

that is a less preferred approach but did that for about 20 of the tests

so for a given role what you need is a testable statement which says here is a test of this role and says what you see in the AAPI

and then need to test it. created minimal html files which had just enough to be valid

then the aria feature, put an id of test on it so could find easily

then look for the mapping

for 1.0 was entirely manual. that is where the test harmess comes in

the test harness gives the statement, the result and the file. would open, then use your inspector to look for the result

we have talked about automating. we believe the majority are automateable, but dont have a platofrm for it

we do know the different vendors have platforms but hey are not public. there are some tools that can be scripted

Joanie has done some experimentation with automation. she is hopeful

want to come back to that too

mostly either roles states and properties table or MUST statements - acc name is part of that

"WAI-ARIA 1.1 builds upon WAI-ARIA 1.0, which met its implementation requirements in February 2014. Only features that are new or changed in WAI-ARIA 1.1 need be tested, as the WAI-ARIA 1.0 implementation report provides implementatibility and interoperability evidence for the remaining features. Features that will be tested for WAI-ARIA 1.1 include:"

Parts of the name computation algorithm may need to be tested but maybe not all of it

for if statements sometimes need a test file to test every one of them. also sometimes some negative test cases

if A+B then sometimes need failures

for key cases where a failure would represent an incorrect implementation then may need some failures too

FE: how do you test something that is not supposed to be in the tree

MC: you say the a11y tree does not have such and such

FE: in SVG have xyz does not get into the tree

MC: id they element with id test should not show in the tree then the expected result is that id test doesn't appear

would suggest putting something that should not appear in the tree with a certain idref

FE: is there an automarted way?

MC: I would think it could be automateable

MK: I missed which doc

MC: https://rawgit.com/w3c/aria/CR-pub/aria/aria.html#sotd

automation

JD: in terms of automating have talked in terms of webdriver and DOM. not going to happen.

have a POC hackaround by which i can run my tests

as an aside - the way the hack works is that it does a tree dive

<MichaelC> NVDA has a python-based tool to poke at IA2

<MichaelC> I might be able to hack that to test IA2

CS: so we are going to be automating our own implementation

I can share the results - but not certain about the code

if you can tell me what format can suck data in. In the ARIA 1.0 test harness could perhaps take data from elsewhere and put it into the harness

everything we are testing - i think the testable statements are the requirements

MC: if we had a way of mapping tests to statements couold suck in the results

we can work on our preferred format

would you be using our test files or are you doing your own set

CS: i suspect some of both

i suspect we will be doing a superset

MC: if your test files are different there is a potentially invalid situation
... my preference would be to use the same tests
... may mean among other things that joanie may want to look at other platforms than ui automation

in 1.0 decided needed to be HTML4 - likley need HTML5 and SVG files for 1.1

went to great pains than 1.1 tests were valid html4

still do need them to be well formed - minimize parsing artifacts. and that includes inline svg if needed

<Zakim> joanie, you wanted to ask the obvious, if it's UI *Automation*, there is surely public/open code we could use to create our own testing on top of the MS testing.

JD: could use for edge perhaps

CS: yes that is true
... wasnt designed for web content

we will probably do that work

JD: if there is public stuff there i could use to hack on your platform using open stuff
... the ID attribute - on most platforms that is the case

CS: it is exposed if there is an implicit id on the platform

<MichaelC> jn: in firefox you could build an extension that looks at the AAPI

<MichaelC> and should work on all its platforms

<MichaelC> and script that with selenium

<MichaelC> jd: we want to test what the AT sees

<MichaelC> not what the browser says it´s sending to the AAPI

<MichaelC> jn: it´s got an interface to the XPCOM information to show exactly what´s winding up in the AAPI

<MichaelC> jd: it´s talking to its own AAPI layer

<MichaelC> js: don´t know which it does

<MichaelC> jn: if I ask the role, it reports something different on different platforms

<MichaelC> jd: it says what it thinks it´s exposing, which varies by platform

<MichaelC> mc: should explore this separately...

<MichaelC> jg: if you have sample extension, we could explore making something work

<MichaelC> jn: looks like it comes from accessibility retrieval interface

<Zakim> fesch, you wanted to discuss asking about test file templates

FE: couple of questions about the template

can it be svg files or can it be html with svg inline

MC: want to make sure that we have consistent test files which could introduce artifcats
... best to pick 1 approach or another
... dont want to go into too much detail about structure of the test files

<MichaelC> ARIA 1.0 test files

the test files for 1.0 are in above

grouped into folders as 800 or so

lots are auto generated

testing a label when there is a menu role

the label should come out based on the combo box

in the test harness should see as correct

would suggest the same with the svg

JN: shouldn't there be something in the file for what the result should be?

MK: think it should all be in 1 file

MC: there are 2 reasons they are not in the file

even in comments could perhaps influence the results - could make it hard to debug

another reason is that there are 5 or 6 different results according to the platform

JD proposal does have them in the test file - we may be exploring that

i am inclined to say we might want versions without and then with results in them for the automation tool

MK: certainly wouldn't hurt to have the textual description of the result in the file

clown: in some cases use the same test file for different cases

MK: would rather duplicate the file

MC: link to test harness
... the test harness where there is a ref to what are testing
... there is a single expected result on all platforms

sometimes different for different platforms

i think probably need to organize a training on the test harness - if havent used it before then lets do a training

should schedule supplementary training

MC: will do a poll will send to aria mailing list i think

will need to test dpub roles and graphics roles too

FE: i think so - will discuss in the svg a11y group

MC: i know the a11y name computation has some host language stuff. some you might rely on existsing stuff - but others are svg specific

dont feel that have got enough on testing to add action items for over the holidays

FE: we plan on using these files and this format of test harness and test files

MC: so want test statements and expected results. if we dont build the test cases etc we wont be wasting time

when to use the exact test harness is an open question

no one loves the harness but it has the advantage that it exists ;)

will be using for reporting

<MichaelC> ARIA 1.0 test report

will be a group for svg and others for 1.1

etc.

if import tests from other locations need the report

<Zakim> clown, you wanted to ask if a difference is expected between svg file and svg-in-html file?

Protocols and Formats Working Group Teleconference

10 Dec 2015

Attendees

Contents

Publications update

Working Group transition update

Mailing list subscription policy update

aria-primaryaction discussion

Test planning (major topic for this call)

automation

Summary of Action Items

Summary of Resolutions

Scribe.perl diagnostic output