SV_MEETING_TITLE -- 12 Feb 2013

<krisk> Looks like hober is attending!

<krisk> We'll waiting a few minutes to see if others join

<krisk> Welcome yosuke

<yosuke> Hi krisk!

<krisk> yosuke are you representing Tomo-Digi?

<yosuke> Yes. I'm also a researcher of Keio university.

<krisk> Here is the Agenda http://lists.w3.org/Archives/Public/public-html-testsuite/2013Feb/0019.html

<krisk> yosuke if you have something you would like to discuss not on the agenda we can discuss at the end of the meeting

<jgraham> yosuke: Not really, it isn't usually busy enough to be needed

<krisk> We stopped using Zakim and an actual conf call...for well over a year

<krisk> No one really dialed in and IRC seems like enough

<jmdyck> are there minutes from jan 29 meeting?

<yosuke> Oh, I see. Thanks.

<krisk> yes, hang on for the link

<krisk> See http://www.w3.org/2013/01/29-htmlt-irc

<jmdyck> tx

<krisk> jmdyck (you are mikesmith right?)

<jmdyck> nope, Michael Dyck

<krisk> OK

<jgraham> MikeSmith: is MikeSmith I expect :)

<krisk> The last meeting conflicted with a testing meeting, so attendance was light

<krisk> Though the sync between git -> www.w3c-test.org is now up and running which was the main Q for the meeting

<krisk> Let's start the meeting now

<krisk> Agenda Item #1: sync between git -> http://www.w3c-test.org/

<krisk> This looks like it is up and running for the 'master' branch - nice!

<krisk> Is it possible to get a sync setup for a 'branch'?

<krisk> Mike or Robin?

<jgraham> It is possible, but afaik it is not done yet

krisk: it's up and running for both branches

see http://www.w3c-test.org/html-testsuite/CR/

it's just that nothing has been folded to CR yet

<jgraham> Ah, I was thinking "for pull requests"

so it's actually all operational

<jgraham> Which is also possible

<jgraham> Dunno if you plan to do that

ah, no, we don't have that for PRs

well, the problem is that PRs are not reviewed, so we couldn't accept PHP code there

<jgraham> I guess that could be an issue for PRs containing PHP

<jgraham> Yeah, exactly

exactly

also, I'm not sure it has value for PRs

<jgraham> Hah, won that one :p

<krisk> * need to add seconds to IRC time stamp?

what we want is for PRs to be reviewed quickly

<jgraham> Well, it has some value, being able to see the tests

<MikeSmith> i am not jmdyck :-)

<jgraham> But the PHP thing could be a showstopped

yeah, it's a problem

<jgraham> (although, long term, PHP must die)

also, if you want to check out a PR on your own local copy, it's easy enough

<jgraham> Well, sure

I propose that we don't do PRs now, and if it turns out to be a problem we revisit the issue

<jgraham> That sounds Good Enough (TM) to me

<krisk> jgraham - what do you want to use instead of PHP?

<jmdyck> (ah, PR = "pull request", not "Proposed Recommendation")

<jgraham> jmdyck: Yeah, darobin likes confusion ;)

<jgraham> krisk: Long term we need something that everyone can use in production

<jgraham> AIUI Mozilla, Google, etc. won't be happy with PHP in production

jmdyck: yeah sorry, we really don't care about Proposed Recs — those are for the AC :)

<krisk> Microsoft also doesn't use PHP

is someone working on an alternative?

<jgraham> Because they run the server on individual test machines and apache + PHP is heavyweight

<yosuke> How about node?

<jgraham> So the long term plan is to write a simple server in python that allows tests to plug into the request lifecycle in more places

<krisk> We picked PHP because it worked on a large number of platforms and has alot of example

<jgraham> Sure

<jgraham> PHP works well for what we do at the moment

part of the problem in having Apache+PHP is that there's a lot of magic going on that interferes with testing

so long term removing that combo as a requirement is certainly a plus

<krisk> IMHO Microsoft is open to using something else

<jgraham> But going forward tests that people aren't running on every internal test cycle are a waste of time

for instance, we have a long standing bug where the server is intercepting an OPTIONS request we'd like the code to handle, and figuring out why is painful

<krisk> The websocket tests ran into a simalar issue - much more complex 'backend'

<jgraham> So the big challenge, as I see it, for the next year or two is going from "we have some tests, but no one runs them" to "we have tests that are run by all the major players dozens of times every day"

<jmdyck> (sorry, where does PHP come into it?)

<jgraham> jmdyck: Some tests currently use PHP on the backend to get special behaviour for the response.

jmdyck: for some tests we need some control over what the server returns

<jgraham> jmdyck: But PHP isn't acceptable for all the scenarios in which we wich to see the tests deployed

<jgraham> *wish

<jmdyck> can you link to such a test?

<tobie> hey

<krisk> For Microsoft we end up having to convert tests into PHP when we submit the to the W3C so that others can use them

tobie: I invited you here because it's the weekly chat of the HTML Testing TF

I reckon it should be on your radar

<jgraham> biweekly

krisk: what do you use internally?

<tobie> ty

<tobie> indeed.

<tobie> Is this a call?

<tobie> or chat?

<krisk> just IRC

no, it's IRC only

but we all agree to be here at the same time

which is... almost like the rest of the day except that we sort of focus a bit more ;)

<krisk> Windows+ASP.NET of course!

lol: )

well, I suspect that won't fly elsewhere :)

<jgraham> jmdyck: http://w3c-test.org/html-testsuite/master/old-tests/submission/Opera/preload/resources/ has some PHP for example

<krisk> That is why Microsoft was open to using something else...especially when we added support for 'async' in IE10

<krisk> Which needed something on the backend when we were testing and submitting tests for this part of the spec

well, I'd rather we didn't get bogged down on picking a winner here — I'm happy to go with anything sensible (NodeJS, Python, Ruby) that someone puts the time into building

<jmdyck> (tx for link, though it looks like server doesn't want to show .php as text)

<yosuke> (JFYI: I used PHP to create some special responses from servers when we were developing test suites for DTV-sets UAs 5 years ago in Japan.)

jmdyck: no, it's running it — you can see the same file in GitHub though

<krisk> The wiki still has all the info to get these files from Hg

<krisk> see -> http://www.w3.org/html/wg/wiki/Testing

<jgraham> jmdyck: https://github.com/w3c/html-testsuite/blob/master/old-tests/submission/Opera/preload/resources/range-request.php

<jgraham> https://github.com/w3c/html-testsuite/blob/master/old-tests/submission/Opera/script_scheduling/scripts/delay.php is another classic example

<jgraham> (wanting control over timing is pretty normal when testing)

<tobie> Are you talking about the required server-side components?

<krisk> yes and possibly other items like specific HTTP HEADERS

<jgraham> tobie: I accidentially sidetracked the discusssion into requirements for the server side

<tobie> when I built that into the prototype test suite a while back, we had a JSON API to request stuff from the server.

<tobie> prob not JSON actually, now that I think about it.

<jgraham> Hmm? That sounds liek something different.

<tobie> but there was a bunch of stuff we could do, like delay. echo headers. echo content. assign headers, etc.

<jgraham> This is about needing to finely control the exact response sent by the server

<tobie> exactly.

<jgraham> Which we currently do by allowing PHP scripts

<tobie> right.

<tobie> Basically, we had a couple of endpoints which allowed you to do different things like:

<krisk> The model that we have used was adding stuff when we had a specific need for a test

<jgraham> But that isn't a good solution going forward because the situations where we want the ts deployed (browser vendor automated regression systems) don't all support it

jgraham: what tobie is describing is a server endpoint that you can configure to behave specifically with the request

<tobie> darobin: I name you official translator of my walled brain.

<jgraham> I see. Perhaps

tobie: btw, please add the HTMLT wiki to the list you have of wikis to consolidate into a single testing documentation set :)
... anytime buddy

<tobie> the list?

<krisk> tobie: feel free to propose something else than PHP

(BTW in case some people don't know, tobie is the new W3C Testing Czar)

tobie: I'm sure you have a list of those :)

<tobie> krisk: will do. Need to understand the requirements precisely, though.

<tobie> glad I found the right place to ask. :)

<tobie> This is on my radar.

<jgraham> It doesn't seem like it will be possible to abstract everything out into a clean set of configurable options, so I imagine our solution will need to allow test-specific scripts of some sort

<krisk> really when I started working on the HTML TFT 3 years ago all we had was CVS and static html content

crazy

<jgraham> tobie: I'm sure I had an email talking about requirements somewhere

I mean what's next? flying cars?

<tobie> jgraham: we'll see. You're prob right, but I'd rather avoid the problem of allowing any language scripts altogether.

<krisk> X-Domain testing is one needs, same with SSL (valid cert) comes up as well...

<jgraham> tobie: Avoid what problem?

<jgraham> Oh

<jgraham> Well I'm not advocating *PHP*

krisk: x-domain can't be done by some script alone (you need some DNS) but I agree it should also do SSL

<jgraham> I'm advocating the ability to write test-specific responses

would be good if it could also do invalid cert

I think there's value in both

<jgraham> Whihc might be like "return 2 bytes, pause 5 seconds, return 2 bytes, close the connection"

having a generic thing that can be configured just with the request is great

and we should leave the door open to weird stuff if needed

<tobie> yeah, my feeling is it gets us 95% there.

<jgraham> Sure, some scripts might be reusable

<krisk> darobin: This is why we have http://www2.w3c-test.org/ and http://www.w3c-test.org/

krisk: indeed

<jgraham> But experience suggests that every new set of tests gives a slightly different set of requirements

<krisk> ...as long as stuff syncs then a contributor doesn't have to figure out how to setup two web servers and getting PHP going.

<jgraham> So a 95% solution can quickly become a 5% solution

<tobie> jgraham: "return 2 bytes, pause 5 seconds, return 2 bytes, close the connection" is the kind of stuff for which it is easy to have a protocol

<jgraham> tobie: Sure, you can have a single script and stuff that kind of thing in the URL or something

<tobie> it also kind of falls into the QoI aspect of testing which we should probably not target immediately.

<tobie> jgraham: yup.

<jgraham> It really isn't QoI

<jgraham> It is essential

<krisk> tobie/jgraham do you mind if we get back on track with the agenda?

<jgraham> e.g. for testing the script scheduler, unless you can reliably set up the order in which responses will some back

<tobie> oh. you guys have agenda?

<jgraham> you can't test *anything*

<krisk> Love the chat but I'd expect we get more progress with sending an email to the list with the problem being solved and a proposal?

<tobie> krisk: apologies.

<krisk> NP

<tobie> krisk: had no idea you had an agenda.

<krisk> Getting back on track a bit :)

<krisk> Ok so we have CR 'branch' that people can working and once they are all set the can ask for a pull request, correct?

<jgraham> tobie: (OT: http://www.w3.org/mid/5076ECF7.7050306@mit.edu from bz has some of the requirements)

<jgraham> krisk: Correctish. You should work on master unless you are specifically testing something that differs between CR and master

<krisk> robin/mikesmith: do you agree?

<krisk> ..I'm looking at the stuff on github...

<krisk> The way to contribute is just as usual:

<krisk> ork this repository (and make sure you're still relatively in sync with it if you forked a while ago);

<krisk> a branch for your changes, git checkout -b submission/your-name;

krisk: sorry, was afk for a second

<krisk> ake your changes;

<krisk> NP

the idea is that people make changes against master

someone who cares about CR (probably me) gets to merge stuff to CR

right, and after you've made your changes, push them, and make a PR

your local branch should be against master

it'll always be more up to date

and CR only contains stuff picked from master

<krisk> I get that part...

(which is almost everything right now)

<krisk> What I don't get is that once a contributor creates a branch

<krisk> ..where on www.w3c-test.org do the tests live so that they can be tested?

no, that branch does not go live (as discussed all the way up at the beginning)

because it requires review

*but*

the idea is that we now have *much* faster turnaround on reviewing

we've been running a few weeks, and of 30 pull requests 22 have already been reviewed and accepted

(a lot of that being the "submissions" backlog)

<krisk> The problem now is that a contributor actually needs to create a web server running and all the backend as well.

<tobie> \o/

well you need that anyway frankly

there are two situations here:

<tobie> krisk: yeah, we'd want this to be a one click install, really.

Situation A: your code does not require any server side functionality. In that case, you can trivially point a local web server to the root of your checked out repo

<krisk> This problem didn't exist with Hg and watching people at the test the web forward event this was a real issue

<jgraham> Huh?

Situation B: you need server-side stuff. In that case the local setup is harder, but we would need to review it anyway.

<krisk> Since at the event the tests we not in Hg rather some random cloud location

<jgraham> This exact situation existed with hg

errr

at TestTWF no one could use hg

<jgraham> (if you used PHP and/or didn't want to develop directly against the real repo)

<krisk> People could they just didn't get setup :)

well, most of them didn't know what hg was, and then they couldn't just get an account

<tobie> krisk: not sure if trolling or…

those people need to know how to point a local web server at an arbitrary directory... otherwise I'm not sure I want their tests :)

<tobie> darobin: you'd be surprised.

tobie: I'm not saying that we can't offer an out of the box solution

<tobie> sure.

but running a local server is not a high bar to pass really

<tobie> why is file: an issue?

it's like, npm install -g httpster && httpster . :)

<tobie> the links to the JS resources?

<jgraham> file:// is a huge problem

file will screw you over seven ways to sunday

<jgraham> It has different security setups in every browser

<tobie> not for 80% of the tests

but yeah, even in cases in which it works the tests point to /resources/testharness.js

<tobie> but yeah, its a real pain.

<jgraham> Right, sadly a local webserver is really a requirement

everyone should note with a big happy that I've included testharness.js using a git submodule

<jgraham> even if it's "python -m SimpleHttpServer"

so checking out now gives you a 100% ready-to-go server (for stuff that doesn't use PHP)

<jgraham> darobin: "happy" and "git submodule"?

<jgraham> I disapprove of this

hey, it's better than anything else we've got

besides, I'm the one maintaining the submodule link so you should not worry about the pain :)

<jgraham> Well apart from the fact that submodule breaks all normal git commands

tobie: because people are surprised that they are snapshotted to a given commit

<tobie> that's a feature.

<tobie> anyway.

I know, but it's one that surprises people

WARNING: RELIGIOUS WAR

<tobie> I'm doing this again. back to your topic, folks. sorry.

+1 to back to topic

<krisk> tobie: can't wait to see your proposals on the list

<krisk> getting back on track...

<krisk> The other agenda item was the whole meta-data questions.

<krisk> Unlike CSS we didn't place meta-data as a requirement in the actual tests for a few reasons

<krisk> #1 you can do this for 'parser' tests since the markup is super specific

<tobie> krisk: cried when I realized meta-data couldn't be gathered using static analysis.

<jmdyck> s/can/can't/ ?

<krisk> jmdyck - correct :)

metadata in file names is nice for our needs

I'm happy with the proposal to use .manual. in names for tests that have to be manual

we also need a few others, e.g. things that are resources to other tests

I reckon .resource. (or whatever colour you like this bikeshed) WFM

<krisk> I'm OK with that as well...

<jgraham> I don't love that, but I suppose it is OK

<jgraham> I should say

<krisk> Though we might be better having it be a bit more specific, e.g....

<jgraham> There is still a major problem here

<krisk> .testharness.

<krisk> .manual.

<krisk> .reftest.

<jgraham> Tests are defined by URIs, not by filenames

<krisk> Yes indeed!

<krisk> -testharness-

<krisk> -manual-

<krisk> ...etc..

<jgraham> So simply walking directory tree and looking for filenames *does not* tell you what tests exist

<jgraham> For example

<jgraham> the html5lib tests can behave in 3 different ways depending on the query string

well

how do you propose to surface that?

<krisk> In the past we had a text file that enumerated all actual tests

<krisk> Someone was going to extend this file

yeah but I'd be very worried that that would go out of sync very fast

<krisk> move to JSON and add other data to it as well

I mean we have 12k tests

moving to JSON is unlikely to help :(

<krisk> or xml..

<jgraham> One file per directory listing the tests doesn't seem that bad

<jgraham> Also

<krisk> or just a delimited txt file

<jgraham> Reftests are defined by two URIs and a comparator

<jgraham> So you can't fully define reftests using filenames alone

<jgraham> (sorry, all this stuff is just paging back in now)

<jgraham> (I think I should have said some of this earlier)

<krisk> Microsoft is open as long as it's clear we have the exact URIs to run all the tests

<krisk> ..knowing one is manual or a ref test is a bonus

so, there are several things there

I don't personally care about fully defining reftests using file names alone

<jgraham> Well I strongly object to it

<jgraham> :)

I'm mostly interested in being able to detect which files are not supposed to be containing tests

(by which I meant automated tests)

<krisk> It's after the schedule meeting time...I propse we hash this out on the list more?

<jgraham> If you have a single file saying "PASS" in it that is the ref for 200 tests, you shouldn't need 200 copies of the file

identifying manual tests is important for the runner, and for coverage

yeah, I agree

why would we want that?

<jgraham> We wouldn't

it can just be called PASS.resource.html

then I know it's not something that ever needs to show up in any runner interface, any bug report about non-automated tests, or any coverage analysis

<jgraham> But saying "the ref for foo.html has to be foo-ref.html" is an attractive nusiance

<jgraham> Or at least has been suggested before

<krisk> jgraham/darobin: This is important but I don't believe we'll get consensus in the next few minutes

sure

we can take it to the list

<krisk> sounds good, if the list goes wacky then we can meet again on IRC

<tobie> I'm actually interest in that topic but I'm missing background here. Any pointers?

<jgraham> I can't think of anything written down

<krisk> They are in the list archieves if you dig/read

<krisk> Let's adjourn!

<tobie> Moz is doing something rather similar internally with refs tests. And I'm trying to find a common ground with webkit which apprantely handles ref test meta data yet another way.

<krisk> Thanks for all the new participants and lively conversation

<tobie> yeah, sorry for derailing. I'll get my shit more together next time.

<krisk> Note moz has simple files in each directory called MANIFEST

<tobie> also my language. oh well.

<krisk> see http://www.w3c-test.org/html/tests/submission/Ms2ger/elements-in-the-dom/

<krisk> Yes no need to use vulgar language

<jgraham> Yeah, I think the Moz. system is fine fwiw

<jgraham> The WebKit system (as it was two years ago, at least) less so

- DRAFT -

SV_MEETING_TITLE

12 Feb 2013

Attendees

Contents

Summary of Action Items

Scribe.perl diagnostic output