Weekly Testing Call

Scope missing APIs relevant to accessibility in WG charter

shadi: comment 2 should be the one we should focus on
... related to accessibility, UI automation
... other ones are more editorial.
... or can be handled at your discretion.

mike: UI automation is something that is quite different from what we have in the current scope section. My understanding is that is a vendor specific platform and an OS specific API.
... Am I wrong about that?

Michael: yes and no. There's a Microsoft and an Apple API that are vendor-specific.
... A testing API needs to be aware of these APIs.

Mike: The WebDriver API should be at the core of the charter. The scope of it is well precised. It's something that runs in a Web browser.

<shadi> https://lists.w3.org/Archives/Team/w3mreq/2011Jul/0083.html

plh: I think Wilhelm said before that they take inspiration from accessibility APIs.

mike: right, what I do not understand here is how UI automation relates to this. I mean at the technical level.
... If I don't understand it, I can't write in the charter. If someone else can propose text, then I could include it in the charter, but I can't do that on my own.
... Another thing is that it seems kind of weird to me to mention UI automation in a charter focused on Web browsers.
... I do not have enough details to be able to write up a description to how this work relates to what we have in the charter.

wilhelm: I'm familiar with what we do here, at Opera. The different parts of the spec are at various maturity level. The core of the spec is only interested at what happens at the Web page level.
... In addition to the core API, there are some experimentations going on but no consensus yet among browser vendors.
... I'd prefer to postpone extensions until we have more feedback experience.

plh: ok, so maybe we're not clear enough in the charter. If we say we're not going to standardize the UI part, then we should make it clear in the charter.

<plh> http://www.w3.org/2011/08/browser-testing-charter.html

[reviewing draft charter text: "there is a specific need to simulate user actions such as clicking links, entering text, and submitting forms"]

plh: what did you have in mind, Mike, with "any other potential APIs useful in performing automated testing of Web applications"?

mike: I didn't have anything specific in mind.
... Happy to remove.

plh: that's the part you're most interested in, shadi, I believe. Is your comment still relevant now that we know the API is only intended to drive things in the HTML page and not in the browser chrome?

Michael: I would say yes as you can still drive the HTML content through the accessibility APIs.
... I think we need to acknowledge that.

plh: section 1.1 is about the scope. Trying to extend this section is a mistake, I think. It should actually be tighter.

mike: I agree we need review. But what does "dependency" mean?

Michael: It's more a point of coordination.

plh: ok, let's see where we can put it, then.

[discussion about "draft notes" section that is to go away]

Michael: I would add a new sub-section under Dependencies and Liaisons section, such as "relevant API".

<plh> - Accessibility Interoperability Alliance (AIA)

<plh> -- <http://www.atia.org/aia/>

<plh> - Open Ajax Alliance

Mike: that's easy enough to do, yes.

<plh> -- <http://www.openajax.org/>

Mike: Do they have some task force dedicated to accessibility I could point to?

Michael: there is an accessibility sub-group of the Open Ajax Alliance, but I would not restrict the coordination to it as the whole things is relevant to our work.

Mike: I have enough information to update the charter. I would still ask Michael to send more precise wording. "consider any other potential APIs" is too loose, I think.

Shadi: the idea is "consider" as looking at the appropriate relationship with existing solutions if, of course, organizations are willing to contribute their work anyway.

Michael: one reason to mention vendor-specific APIs is to have everyone be conscious of the overlap.

Mike: that won't happen. It is not duplicating anything that everyone else has done. There is no existing API that works across browsers right now.

plh: one way could be to expose these API in JavaScript.

Mike: right, but there is no standard mechanism that does what this API does.
... It would be good to get some direct review from people in these companies.

plh: Mike, to summarize, you need to run through Shadi's message.

Mike: ok, I can do that by tomorrow.

<MikeSmith> francois: what's the status on the IG charter:

<MikeSmith> plh: it's stable, no changes needed

Framework

<MikeSmith> http://w3c-test.org/framework/

mike: so, Francois deployed something not working and then cowardly left for vacation leaving everything on my plate
... I had a look, and ran into weird problems, apparently because PHP version was not 5.3, and also Apache version was too late.
... I upgraded everything on the server.
... The thing is working.
... I also replaced it with a clone of the CSS server Mercurial repository.
... Now we can pull and push updates directly.
... What we could do this week is to setup things for it to work with the Web Performance WG spec.

plh: Yes, good guinea pig. That's a good test.

Mike: everybody else who is interested in getting the framework populated now that it's set up, I'm happy to help with that.
... It's team only because of SSH access.
... I think it's bad practice to all act as root on that server.
... Better to create a user account on that machine.

plh: sounds good to me.

shadi: if you can write this up somewhere, that would help.

plh: you still need someone with root access to the machine.

shadi: just an email that explains what to request from systeam and what to setup on one's machine would be good not to forget steps later on.

plh: how to add a group in the framework would be a good thing to document

mike: pretty well documented, actually.
... The process is clear but requires a lot of manual work. We could provide a better interface later on.
... You work with text files, and have to work with a few mysql statements.

plh: ok, I don't think we need to focus so much on making things automated at this stage.

mike: there are two pieces: the "framework" which Peter calls "harness". The second is "shepherd" wich is the test suites management system.
... I'm not sure I see that as a requirement.

plh: well, that's why we picked up the framework in the first place.
... So would be good to deploy.

mike: once we get the framework setup for Web Performances WG, we'll be clearer as to what we need.
... That should be doable this week.

plh: we may need to update the tests, so could take longer.
... Another thing is multi-WG support.

<MikeSmith> here is Peter's how-to:

plh: Actually, I wonder if we want to tie this framework tied up to working groups.

<MikeSmith> http://hg.csswg.org/dev/harness/raw-file/tip/docs/index.html

plh: better to tie it to specifications to handle cases when a working group get splitt up for instance.

mike: here's the thing. That's more or less how it works today.
... It's per spec and test suite. We could simply add a column to link it to a working group.

<MikeSmith> francois: the harness is already set up to work with different specs

<MikeSmith> … and we could easily add a column for WG info if we want to

[francois thanks mike a lot for taking up the hard task of debugging the deployment while he was enjoying sun on vacation!]

<MikeSmith> francois, debugging wasn't so hard, and Tom helped some. I'm glad you got some vacation man

Informal summer meeting

shadi: I'm wondering what the process and expectations are.
... which kind of organizations are to attend the meeting. What's team attendance expectations, that sort of things.

wilhelm: the general idea is to get an overview of what browser vendors handle tests today. We need to know what vendors need to test. And what they'd be ready to contribute.

shadi: are we only focusing on browser vendors? Aren't we going to miss something?

wilhelm: to answer the two questions, that's good to narrow the scope. We will have to extend things afterwards. In one or two days, there are only so many stuff you can cover. I hope we'll kickstart a long discussion here.

shadi: The fix-up later approach is typically what makes accessibility added afterwards, which always creates problems. I'm concerned we'd be leaving a bunch of stakeholders.

plh: this is an informal meeting. Not a group coming with a set of requirements.

shadi: we want browser vendors to be more considerate about accessibility issues.

<Zakim> MikeSmith, you wanted to say we have the consideration already, and this is not a case of risking "fix it later"

Mike: we are already considering accessibility. Any group considering work in W3C already considers accessibility, even if not explicitly mentioned. I don't like the implication that we are ignoring accessibility here, and using a fix-it-up later approach.
... It's not a workshop, it's an informal meeting.

plh: What I heard Wilhelm say is that we have browser vendors which have an extensive set of test cases and do not know how to leverage those in W3C.

wilhelm: that's correct

plh: We could say that anyone who has vast amount of tests is welcome to attend to see how these can be contributed.
... It's not a meeting about coming up with requirements.

<gsnedders> I wonder how much of the infrastructure is going to be shared for a11y testing, because if you're testing what's passed to the OS APIs that's inevitable going to be a separate API to test.

Michael: what's missing here is that there are all ways to tackle testing things. For me, it's not optional to have other stakeholders here, it's important to have them.

<MikeSmith> which people

shadi: we are missing people who are doing accessibility testing.

<jeanne> there is a outside group doing accessibility testing - I will look up the name

<MikeSmith> bingo

shadi: The more we shortcut the deadlines, the more difficult it is to get people on board.

<MikeSmith> there is interest from browser vendors

plh: the goal is to see if there's interest from browser vendors in the first place.

<MikeSmith> the lack of a response from them so far does not indicate they are not interested

shadi: I do think we do need to take that we are trying to involve other players, and we should take care not to leave them aside unintentionally.

plh: what we want to learn is how people do testing rather than how we should be doing testing.
... That's why I'm more inclined to attract people that have good amounts of tests.

<Zakim> MikeSmith, you wanted to ask Michael to explain exactly what things and to say the idea is to keep this meeting small

plh: I need someone to take an action item to come up with a precise list.

Mike: in some comments you're making, Shadi, you're telling us that we may be forgetting accessibility but you're staying vary vague.

<jeanne> jeanne found the group, and it is the same group MichaelC has been discussing. Sorry for distraction.

Mike: Another point is that it is to remain an informal meeting, focused, with fewer people. The idea is to get these people into the same room.
... The goal is to save time, see how we can stop reinventing the wheel across groups. It would be a good idea to see people representing accessibility concerns but, as mentioned before, people involved in browser development are already addressing accessibility concerns in their everyday work.

shadi: we're really not having diverging points at all.
... Having accessibility as part of every developer day job is a good goal, but it's still a goal, not the situation today, even within W3C.
... There has to be a compromise between the number of people attending this meeting and the number of viewpoints we can have.
... I think we're all having the same vision for the future.

plh: What I'd like to have to move forward is to identify individuals, including accessibility.
... That's what Wilhelm did, targeting specific individuals, not companies.
... It would be good if you could come up with a list of people, Shadi.

<MikeSmith> while I agree that making specific mention of accessibility coordination in charters in activity statements and other places does not imply that accessibility concerns are being neglected or that there is not a high level of accessibility consideration among the people who produced the activity statements and charters

shadi: I'm asking Jeanne if she can take this up.
... We'll try to come up with a list.

wilhelm: goal is to have the meeting as soon as humanly possible.

mike: one thing about timing we should agree on is that when we have critical mass, we still need to announce it, about a month in advance for travel arrangements.

plh: we're going to have to announce it as well in some fashion.
... We're going to have to strike a balance between open and closed door.
... Not a proper workshop (no position paper).
... We should have a list of questions ready for people interested to show up to check whether we need them in the meeting.
... We need people with a large number of tests who would like to contribute them.

<shadi> http://www.w3.org/2005/10/Process-20051014/events.html

shadi: note there is no requirement for position papers to call some event a workshop.

Next call

plh: I'm on the road next call, so propose to cancel it.

<jeanne> jeanne is on vacation

plh: Next call on August, 1st.

[call adjourned]