IRC log of testing on 2011-02-28

Timestamps are in UTC.

14:03:54 [RRSAgent]

RRSAgent has joined #testing

14:03:54 [RRSAgent]

logging to http://www.w3.org/2011/02/28-testing-irc

14:03:56 [plh]

agenda+ annotations

14:04:01 [francois]

RRSAgent, make logs public

14:04:02 [plh]

agemda+ timeline

14:04:10 [plh]

agenda+ timeline

14:04:16 [francois]

scribe: francois

14:04:23 [plh]

zakim, move to next agendum

14:04:23 [Zakim]

agendum 1. "wiki" taken up [from plh]

14:04:26 [francois]

zakim, mute me

14:04:26 [Zakim]

francois should now be muted

14:05:01 [plh]

--> http://www.w3.org/wiki/TestInfra/goals Test Infrastructure

14:05:19 [francois]

mike: I took material written more than one year and half ago when we had some meeting and put together some requirements

14:05:35 [francois]

... linked from that page are some requirements from other groups

14:06:00 [francois]

... goal is to get all the requirements in one place so that we know what we want to do

14:06:45 [francois]

plh: I had a call on Thursday with accessibility people. We should break down the page into "what the harness should do", "how to submit the tests", etc.

14:06:56 [francois]

... Not all the WG will submit tests the same way

14:07:07 [francois]

... First priority: what the harness should do.

14:07:15 [MikeSmith]

http://www.w3.org/wiki/Testing

14:07:20 [MikeSmith]

http://www.w3.org/wiki/Testing/Requirements

14:07:22 [francois]

mike: doug set up another page

14:07:49 [francois]

... because he did not like the name. I don't care about the name

14:08:10 [francois]

ack me

14:08:36 [francois]

plh: I don't care either way. Let's take a decision. Do you care?

14:08:48 [francois]

[ok to flip a coin]

14:09:02 [francois]

plh: ok, please proceed under /Testing, then, Mike.

14:09:33 [francois]

plh: could you start classifying the topics in the page?

14:09:44 [francois]

mike: yes, easy enough to do, happy to do this this week.

14:10:02 [plh]

zakim, move to next agendum

14:10:02 [Zakim]

agendum 2. "annotations" taken up [from plh]

14:10:18 [plh]

ACTION: Mike to classify the requirements in the wiki

14:10:45 [plh]

Mike?

14:10:48 [plh]

can't hear you?

14:10:55 [Zakim]

-??P2

14:12:43 [plh]

agenda+ framework

14:12:51 [MikeSmith]

MikeSmith has joined #testing

14:14:43 [MikeSmith]

the fan on my laptop has quit, so my laptop is overheating now and then, to the point where it hits 85 degrees or something and then shuts itself down

14:15:09 [MikeSmith]

I have a new laptop on the way but I won't get until for at least one more week

14:15:23 [plh]

can you rejoin the call or should we continue on irc?

14:15:32 [Zakim]

+??P2

14:16:04 [MikeSmith]

Zakim, ??P2 is me

14:16:04 [Zakim]

+MikeSmith; got it

14:16:12 [francois]

mike: let me paste in the URL for annotations

14:16:16 [MikeSmith]

http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/

14:16:36 [francois]

mike: here's the system that Philipp Taylor's set up for canvas.

14:16:58 [francois]

... not scalable to the whole spec as it does regular expression checks specific to this section.

14:17:39 [francois]

... what we could do is to scope the tests with the IDs of a specific section and then we can have it scale for finer-grained addressing.

14:18:01 [francois]

... If it works out for HTML5, it should work for everything else, since there's hardly anything bigger than the HTML5 spec.

14:18:09 [MikeSmith]

http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html

14:18:29 [MikeSmith]

http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html#testrefs.initial.colour

14:18:46 [MikeSmith]

http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/initial.colour.html

14:18:55 [francois]

mike: you'll see annotations in the result specifications, attached to testable assertions.

14:19:21 [francois]

plh: I thought this was hard to replicate on a larger specification?

14:19:56 [francois]

mike: not really. The problem is if we don't have any way to scope a test case to a specific section of the spec. If we do, then this thing can scale.

14:21:28 [francois]

mike: the point is that you have at the beginning of the test case the id of the section in the spec, and then the regular expression would apply to this section only.

14:21:36 [francois]

... If we do it that way, it would scale.

14:22:02 [francois]

plh: but the specification is a moving target!

14:22:41 [francois]

mike: when the prose changes, we need to come up with an error message that says "I cannot find this assertion anymore".

14:22:56 [francois]

plh: so that's what you're working on?

14:22:58 [francois]

mike: yes.

14:27:02 [MikeSmith]

http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/tests2dtext.yaml

14:27:36 [francois]

francois: in short, I'd introduce a level of indirection with a mapping table between testable assertions, extracted by a regular expression, and an ID, that tests can then reference.

14:28:01 [francois]

mike: it seems doable and is not far away from what I've been contemplating do.

14:28:12 [francois]

... I need to take a closer look at that.

14:28:23 [francois]

... I should be able to work on that later this week.

14:28:28 [plh]

zakim, move to next agendum

14:28:28 [Zakim]

agendum 3. "timeline" taken up [from plh]

14:28:29 [francois]

plh: anything else on annotations?

14:28:51 [francois]

plh: in terms of timeline, I'd like us to be done with requirements by the end of March.

14:29:13 [francois]

... I'm hoping that Shadi and Michael are going to be able to show us what they need in the coming week.

14:29:34 [francois]

... Once done with requirements, April and June should be spent implementing the bouzin.

14:29:47 [francois]

s/April and June/April to June/

14:30:04 [francois]

... We may need some things running in May, for instance for Michael.

14:30:31 [francois]

... At the minimum, we should try not to change the format of the tests.

14:31:50 [francois]

... Because of the parsing tests, we'll need some way to have tests completely separated from metadata about the tests.

14:32:37 [francois]

francois: will we need to setup some hybrid system? I'm all for keeping things separated, but some people may think otherwise.

14:33:08 [francois]

plh: Right. About parsing tests, we may need tests that are exactly 1 byte long, and we cannot include metadata in there.

14:33:32 [francois]

mike: yes, I think we already discussed and agreed about the need to have a manifest file that describes the tests.

14:33:58 [francois]

plh: what I would suggest for this metadata information is to look at what the CSS working group did, because they are pretty advanced there.

14:34:01 [francois]

mike: ok.

14:34:38 [francois]

plh: After June, once we are done with setting up the framework, we should focus on integrating tests in there, such as HTML parsing tests.

14:34:48 [francois]

... tests done outside of W3C, for instance.

14:35:25 [francois]

mike: getting the parsing tests in is definitely an important goal.

14:35:47 [francois]

plh: yes, if we manage to get parsing tests in, the rest should be relatively easy to integrate.

14:36:11 [francois]

... Some tests may apply to places where JavaScript is not available and may be trick, but the rest should be fine.

14:36:18 [francois]

s/trick/tricky/

14:36:48 [francois]

... On the framework front, we need to identify all the frameworks that are out there. I believe that the Wiki page already has such a list.

14:37:11 [francois]

... Someone needs to look at what Mozilla is doing and understand how Mozilla is testing their implementations.

14:37:22 [francois]

... They are probably the most advanced ones out there.

14:37:35 [francois]

francois: happy to have a look

14:38:05 [MikeSmith]

Ms2ger

14:38:18 [francois]

ACTION: francois to see how Mozilla is testing Firefox

14:38:51 [francois]

mike: you may get in touch with Ms2ger, as he is a key contributor to testing efforts.

14:39:34 [francois]

plh: I think we should still have a direct look at it. Francois, would be good to spend time and see e.g. the test format, how they test their mobile implementation, etc.

14:39:45 [MikeSmith]

Mark Finkle

14:40:07 [francois]

mike: the guy to get in touch with to have pointers to the right things is Mark Finkle.

14:40:16 [francois]

plh: is there an IRC channel for testing?

14:40:26 [MikeSmith]

irc.mozilla.org, ask on #developers

14:40:42 [francois]

mike: don't know, but you should be able to find the info on Mozilla's main developer irc channel.

14:41:30 [francois]

plh: don't know how long it would take, probably two weeks. I think that's valuable. We need it before end of March.

14:42:00 [francois]

francois: ok, will do asap.

14:42:14 [francois]

plh: anything else on the testing framework?

14:42:35 [francois]

... I'll ping Michael and Shadi to make sure they get their requirements right.

14:42:41 [francois]

mike: next meeting?

14:42:53 [francois]

plh: how about this day and time?

14:43:34 [Zakim]

-francois

14:43:37 [francois]

[should work]

14:43:44 [francois]

RRSAgent, draft minutes

14:43:44 [RRSAgent]

I have made the request to generate http://www.w3.org/2011/02/28-testing-minutes.html francois

14:48:38 [francois]

Chair: plh

14:48:46 [francois]

Present: plh, francois, mike

14:49:40 [francois]

Meeting: Testing team call

14:49:51 [francois]

s/plh: anything else on annotations?//

14:49:55 [francois]

RRSAgent, draft minutes

14:49:55 [RRSAgent]

I have made the request to generate http://www.w3.org/2011/02/28-testing-minutes.html francois

14:50:38 [MichaelC]

MichaelC has joined #testing

14:56:09 [Zakim]

-Plh

14:56:10 [Zakim]

Team_(testing)14:00Z has ended

14:56:12 [Zakim]

Attendees were Plh, francois, MikeSmith

15:02:51 [francois]

RRSAgent, bye

15:02:51 [RRSAgent]

I see 2 open action items saved in http://www.w3.org/2011/02/28-testing-actions.rdf :

15:02:51 [RRSAgent]

ACTION: Mike to classify the requirements in the wiki [1]

15:02:51 [RRSAgent]

recorded in http://www.w3.org/2011/02/28-testing-irc#T14-10-18

15:02:51 [RRSAgent]

ACTION: francois to see how Mozilla is testing Firefox [2]

15:02:51 [RRSAgent]

recorded in http://www.w3.org/2011/02/28-testing-irc#T14-38-18