<plh_> http://www.w3.org/wiki/TestInfra/goals
plh: Page for requirements. We need to wrap this up. We're already late and need to move forward.
<MichaelC> Michael take on requirements
michael: too many things going on at the moment. I've added my comments in a separate page.
... in my email, I explain how to separate between use cases, general requirements, and test files. From most of the requirements I tried to detail things a bit. I know that I haven't captured everything from the original page.
... There's still room for some merging, I haven't gone into technical details.
plh: I'm skimming through it right now. It looks reasonable.
michael: I hope so. There's one requirement that there exist a layered approach between test files and test cases.
... One other core requirement is that you can have test files that are shared among working groups.
plh: already possible, yes.
michael: Too me, I thought this organization is better to grasp.
plh: things look good. We are working resources across working groups already. I just take those for granted.
michael: I don't think we should take things for granted when developing a product, we should be explicit.
judy: Even if the differences are not fully laid out here, we feel it's important to talk about that together.
plh: happy to do that, but we need to spell out the differences.
shadi: very briefly, in the requirements that we have so far, the question is how can we turn things that appear in the wiki into something that we can put in the development of a product?
... I think we need to converge and try to find consensus on specific requirements that we all share
judy: michael, could you lay out some of the differences?
michael: unclear to me what we have and don't have so far. For instance, testharness.js is good but I don't think that we're going to be able to reuse it for WAI-ARIA.
... There are test cases that cannot be expressed like that. I need separation between metadata and test files.
... We may be needing manual tests here.
plh: That's what we've been saying in the past two months.
... We have the exact same requirements in the HTML working group. Manual test cases will be needed. Metadata cannot be put in all test files. I don't hear any difference.
michael: then it goes back to the requirements page that is confusing.
<judy> http://www.w3.org/wiki/TestInfra/goals#Self_describing
plh: how is "self describing" test confusing?
shadi: where is the JavaScript one?
plh: one of our goals is to make tests as automated as possible. We know that it's not always possible. We'll have thousands of tests in HTML. We cannot afford to run these tests manually.
... There are examples there about what we mean by that.
... The framework that will be in charge of running the test cases and collecting reports will not preclude any specific type of tests.
shadi: I'm still confused. Maybe it's just because it's named "testharness.js"
... I see possibility to ask users for permissions. So this is not fully automated?
plh: right, the result can be reported automatically, but the test may involve manual user intervention.
shadi: so we might want to separate them
plh: maybe it's more that we should call them "automatic result reporting"
... whether the script can run the test automatically or not is independent on how it reports the result back to the framework.
shadi: another question. So we have this testharness.js which is pretty far already. It covers certain parts of testing that we'll want to reuse. What about the other types of testing?
... Are we going to develop another test harness for that?
plh: no, that will be part of the test framework, the thing that is on top of the rest.
... The framework will collect the result automatically if it's an automated test, perform the comparison if it's a ref test, or ask a human to report on the result of the test.
shadi: ok, thanks for the clarification, that's what I would call "harness", but not a big deal as long as we agree on definitions.
plh: whatever we're going to create in the end, I want to be as flexible as possible. The state of the art is going to change in the next two years.
... A new version of testharness.js is in the works.
... That's not a big deal, the important stuff is that they can be run in the same testing framework.
... Human won't change, so manual tests are easier. The test could be executed outside of the browser environments.
... The only requirement for manuel tests is running a browser environment to report on the result.
[Michael back to the call. plh going through same explanation]
judy: are there any reasons why the test harness being developed would not be a good fit for WAI-ARIA?
plh: The general harness for me is the framework.
... Let me share an example of such a framework.
<plh_> http://w3c-test.org/html/tests/harness/harness.htm
plh: That's not the best example that we have. Two parts when you load this page. The top part is the framework running. The bottom part is the test running.
... Looking at this, you should be able to tell whether the test has passed or failed easily.
... In case of automated tests, the framework would just collect the result and move on to next test.
... This harness has limitations: it does not know how to recognize automated and manual tests.
... Aesthetically, it's not the best one either.
... In some examples, the test itself cannot tell you whether it passed or failed.
michael: today is the first time that this is not the harness that we'll be developing.
judy: some kind of miscommunication. Given that plh is saying that i've heard that this is the kind of things that we'll be doing, does that sound like going in the right direction for WAI-ARIA if this thing is cleaned up?
shadi: Michael's requirements seem refinement of the test framework
plh: yes and no.
<judy> [judy suggests adding a glossary, for fear of additional work time lost in misunderstandings]
judy: it may be worthwhile to design a glossary not to lose additional time. It should be quick.
plh: Right. I tought I had settled down the terminology two weeks ago, but I don't seem to have been very successful at that.
... I probably need to create a diagram.
judy: Before we get to the top of the hour, can we focus on merging requirements?
... A quick assess on whether that's doable?
plh: It doesn't seem to be too complex to do the merging.
... I'm happy to take an action to try to put our requirements page into Michael's page to help clarify things.
<plh_> http://www.w3.org/2011/05/testing-ig-charter.html
plh: we've been talking about the draft IG charter for about month.
... Goal is to review it internally before call for review.
shadi: a couple of comments. Thanks for taking previous comments on the scope section.
... The paragraph before last paragraph in the scope section kind of narrows the scope and links to the Wiki which may not be appropriate in a charter.
plh: oh, we should be sending an advance notice to the AC as well.