this is DPUB-ARIA

testing DPUB-AAM

DPUB-AAM published this morning at http://www.w3.org/TR/2015/WD-dpub-aam-1.0-20151203/

<MichaelC> PF test harness

michael: we have DPUB roles and AAM
... we care more about getting roles to CR, but they are dependant on AAM
... expected result of roles is that it is exposed in accessibilty mapping X as X
... when testing a specific role, using a browser with Accessibility API X
... and seeing the roles implemented in at least 2 APIs should get us to CR

tzviya: summarizes michael's comments: proof of implementation for DPUB roles is that any role is implemented in any 2 browser/API combo

ivan: the goal of CR is to prove implementability, but there is also goal to show usage
... are we also under obligation to show that the roles are in use?
... similar to semantic web. How do you check a vocab?

michael: we are first concerned with exiting CR, partly by pre-negotiating
... we propose our criteria to the director, and then we meet the goals
... We should care about implementation as well. For ARIA 1.0 we didn't have the resources, and let's look at the testing

tzviya: i know of organizations IRL that will use this vocab, so we can provide real examples in the wild

shane: we don't need to have examples to exit CR, but if we do, best method is to demo with schemas in validator.nu

michael: showing manual test procedure for Aria 1

<MichaelC> https://www.w3.org/WAI/PF/testharness/testcases/edit?testsuite_id=1&testcase_id=183

<scribe> scribenick: tzviya

michael: this is manual testing, but it will likely be in use for 1.1 as well (at least in part)
... must indicate exactly what API, browser, etc testing

<Zakim> ShaneM, you wanted to discuss other ways to test a vocabulary

michael: having 2 passes is enough to pass CR
... this can also be used for bug fixes

tzviya: who did the testing for ARIA 1.0?

michael: memebers of PF, including Jon Gunderson and others from U of Illinois
... Rich and Cynthia did a lot of testing as well
... there are 500 - 800

ivan: Other WGs had around 1000 tests as well, but the tests had reports, so the tests pushed information to the implementors
... this may be different though
... At the end of CR period, we need to put togethor a report

<MichaelC> ARIA 1.0 test report

michael: this is php pulled from mySQL db

agenda next

Test Automation

<ShaneM> It should be trivial to push results from automation into a database - set up a web service

michael: this stuff should be automatable
... looking for token, varies by API
... the a11y API inspector tools are scriptable, so can automated by platform, but it would be custom by platform
... ARIA WG will talk about this further next week

<MichaelC> Test the Web Forward

We don't know how much automation we'll be able to do in next few months bc this is ongoing

<Zakim> ShaneM, you wanted to suggest that we not make automation dpub's problem

shane: it's not DPUB's problem to figure out autotmation. We need to solve this for everyone.

tzviya: next steps - it looks like automation is a bit off
... how do we move this forward?
... it would be helpful to test all the AAMs together and tap into eager student bodies

<Zakim> ShaneM, you wanted to ask if there is a tool to extract the mappings automatically? (assuming the answer is yes)

shane: is there a way to extract mappings auto?

michael: we use xslt

shane: does respec produce xhtml? does respec spit out something reasonable?

<ShaneM> ACTION: Shane to script output of assertions from ReSpec into JSON [recorded in http://www.w3.org/2015/12/03-dpub-aria-minutes.html#action01]

<trackbot> Created ACTION-1756 - Script output of assertions from respec into json [on Shane McCarron - due 2015-12-10].

shane: testing of DPUB is much easier than core testing, because it's just role testing

tzviya: we have a note in current draft of DPUB that we can extend the vocab. At what point do we freeze the vocab?

<ivan> +1 to Michael

michael: freeze when you enter CR, but also goes to question of vocab extensibility
... do something of a virtual last call

<ShaneM> Also need to know when the AAM is stable...

<ShaneM> (in other words, do you trust the mappings?)

michael: we need to know when the DPUB vocab is set, when we are ready to test

ivan: for each of our statements, we must have a test file, using the role

<ShaneM> I envision a tool chain

ivan: shane's script should create HTML files from the code samples in the document

tzviya: we plan to provide more complicated, nested samples

<Zakim> MichaelC, you wanted to talk about test file format

michael: test files are the smallest granular chunks possible

tzviya: i'm thinking that we should leave existing examples, and add appendix to demo comprehensive examples

<ShaneM> <a>user agents</a>
...timeline: 2PWD just published - we should be able to comment on state of the vocab in January after feedback

<Zakim> ShaneM, you wanted to mention something about term references

This is scribe.perl Revision: 1.144  of Date: 2015/11/17 08:39:34  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/AAm/AAM/
Found ScribeNick: tzviya
Inferring Scribes: tzviya
Present: ShaneM Tzviya_Siegman janina Michael ivan rich Rich_Schwerdtfeger

WARNING: No meeting title found!
You should specify the meeting title like this:
<dbooth> Meeting: Weekly Baking Club Meeting

Agenda: https://lists.w3.org/Archives/Public/public-dpub-aria/2015Dec/0001.html

WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth

Got date from IRC log name: 03 Dec 2015
Guessing minutes URL: http://www.w3.org/2015/12/03-dpub-aria-minutes.html
People with action items: shane