See also: IRC log
<joanie> clear agenda
<joanie> agenda: this
<joanie> agenda: be done
<joanie> scribe: joanie
Link to documentation: https://www.w3.org/wiki/ARIA_1.1_Manual_Testing
https://www.w3.org/wiki/ARIA_1.1_Manual_Testing
ARIA 1.1 implementations in report: https://w3c.github.io/test-results/wai-aria/#index-of-implementations-in-reports
<tzviya> https://w3c.github.io/test-results/dpub-aam/#index-of-implementations-in-reports
<MichaelC> scribe: MichaelC
jd: above applies to ARIA and DPub
swapping dpub-aam for wai-aria in URIs
includes contact info of people working on platforms
I´ve done on some non-Windows systems
Jon working on IA2 testing, hopes to be ready by next week
but we still don´t have a plan for UIA testing
so at the moment, manual testing needed on that platform
MS has said they plan to do it; I´ve sent the instructions
if you get to the point you don´t have implementation data, may need to do manual testing, since the automated is happening
<joanie> https://w3c.github.io/test-results/wai-aria/all.html
I´m running automated tests for ATK and AXAPI, 2 UAs each
<joanie> https://w3c.github.io/test-results/dpub-aam/all.html
above links are test reports for ARIA and DPub
The headers in the table are explained in the index of implementations page
the two WebKit implementations show mostly passing results
the other two mostly failures
that doesn´t indicate manual testing needed - it´s valid data about not passing at the moment
it´s lack of data that needs manual testing
<joanie> https://w3c.github.io/test-results/wai-aria/all.html
in above link, fields with orangey-yellow lack data
there are a few reasons this can happen
we added tests but didn´t update report, or the test isn´t applicable to that platform, or we simply lack a test
mc: can we distinguish don´t need data from don´t have but need?
jd: documentation page gives pointers to figure that out
you can usually figure it out
often, when there´s an additional test file under a feature, it´s to test additional platform-specific requirements
<Zakim> MichaelC, you wanted to talk about what we tell Director, and if we can get additional info here
mc: Director will want to easily see difference
also would be good to add more clear ¨key¨ information
jd: requires test runner upgrade,
I can approach Shane about how to get that made
... for ARIA 1.1 we added some things to existing roles that
weren´t requirements in 1.1
for new roles we just test the whole role
for changes, we kinda need to test the new and old behavior
not sure if we´ll be able to do that automatically, so that could add a small set of new manual tests needed
though my goal is still to automated it
there are only 10 anyways
<Zakim> tzviya, you wanted to ask why I get permission denied messages when I click through
ts: when I follow links, i get ¨permission denied¨ errors
<joanie> https://www.w3.org/wiki/ARIA_1.1_Manual_Testing
jd: when you´re doing manual testing you shouldn´t encounter this
but it is something I plan to fix
<joanie> https://w3c-test.org/tools/runner/index.html
this is the test runner
put the shortname of the spec you want to test in the field (wai-aria or dpub-aam), with a starting slash
can take a while to load after that, then shows test case with a table
each test case is pretty minimalist
the table essentially says what you´re looking for, in the various APIs
you need to use an AAPI inspector for the platform
AccProbe for IA2?
look through the a11y tree to find the element called out in the test, and check its properties
see if it´s got the thing, or property has the value, the table says it should have
then go back to main window and choose ¨pass¨ or ¨fail¨
then it will show the next test
so you rinse and repeat
be sure not to close the test pop-up window, the test runner really wants it
<Zakim> MichaelC, you wanted to ask how you pause and resume a session
mc: can I pause a session and pick up where I left off later? e.g., after my browser has restarted so I didn´t just leave it open
jd: you can walk away from tests and come back, the harness will just sit there waiting on you
you can also select a subset of the tests at the start of the process
you can use regex, but more simply use prefix
you can get a JSON file and send it to me to incorporate into the test results repo
ts: for DPub-AAM, what´s still to test?
jd: Windows
ts: so there´s not a column for it yet?
jd: right, only the UAs listed are known to the system yet
so Windows isn´t included yet
mc: can we have placeholder columns?
jd: yes, after a code review
I haven´t because I don´t like fully empty info
mc: think an empty column is more clear for non-testing reviewers
jd: we´re still not fully sure of MS plans
tz: have access to various systems
jd: Firefox on Windows might be most useful place for you to start
suggest you test a nightly build, so you have the latest implementation
since features are landing as we speak
we are hoping MS will take care of Edge
we can more easily get nightlies of FF
also think you´ll get similar results to FF on Linux
Linux is the redheaded stepchild of the accessibility world
that´ll help you sanity check your process
tz: will look through this and come back with questions
jd: I don´t have Windows, except on a virtual machine in another country, so will appreciate additional guidance in the wiki about that platform if you have it
ts: will look at this, and ask dpub people to as well
jd: people can come to me with
questions if needed
... other comments, questions, concerns, volunteers?
jva: also mid-stream on figuring this all out
<joanie> https://www.w3.org/wiki/ARIA_1.1_Manual_Testing
jd: in https://lists.w3.org/Archives/Public/public-aria/2017May/0041.html JG says he hopes to have IA2 ATTA next week
in which case all the magic happens
otherwise manual testing
jd: progress on IA2 ATTA
note an ATTA is an ATTA, it works for any version of specs
jd: I´m focusing on the 2 right columns in the ARIA test report, turning any failures there to passes
mc: who´s doing the other two?
jd: bugs are filed with Mozilla
This is scribe.perl Revision: 1.152 of Date: 2017/02/06 11:04:15 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00) Found embedded ScribeOptions: -final *** RESTARTING DUE TO EMBEDDED OPTIONS *** Present: Joanmarie_Diggs tzviya MichaelC matt_king Job Regrets: Michiel_Bijl Rich_Schwerdtfeger Jon_Gunderson Found Scribe: joanie Inferring ScribeNick: joanie Found Scribe: MichaelC Inferring ScribeNick: MichaelC Scribes: joanie, MichaelC ScribeNicks: joanie, MichaelC Found Date: 18 May 2017 Guessing minutes URL: http://www.w3.org/2017/05/18-aria-minutes.html People with action items: WARNING: Input appears to use implicit continuation lines. You may need the "-implicitContinuations" option.[End of scribe.perl diagnostic output]