These are Chaals' rough notes - libby's rough notes are also available. EARL discussion begins at [15:06:38]
staff contact for ERT group. Taking Sean's examples, (0.95) where there are tools generating, and thinking about queries. Those might be difficult with the current spec.
Jim's tool will accept EARL from another tool, and use it to make an annotation using annotea. It can be queried for "all the annotations".
I have been thinking about how we can simplify EARL, based on looking at the graphs of existing material. I have a graph I want, but don't know how to make it work in RDF yet.
Making a script that produces EARL for use with test suites. Working on making an object interface for creating EARL. Not sure I understand why the things in EARL are the way they are.
GB what is the input?
NH There is a test file that specifies a list of tests. It presents a form, so a human can evaluate a page and the results are generated as EARL.
trying to collectg people who do EARL things and get user scenarions. Chasing Annotea folks and trying to work out how to get annotea working.
WC Jim has two ways to query EARL - one of them is an IRC bot. We have a list of user scenarios - want to develop them and make sure we can do queries that we need.
My understanding of EARL in our tool is a format to use in the result - we could export EARL instead of HTML/XML. That should be able to be implemented easily. Being able to import EARL and use it requires more work - I need to be able to parse EARL, I need to have something like a fuzzy Xpointer, rather than character offset.
Pointer is crucial for two things - referring to a mnual test result, and when someone makes a correction it changes the character offset so I need to be able to keep that up to date.
WC Nick's tool is using fuzzy pointers for that. It needs more testing, but seems to interoperate with what Jim is doing so far.
GB back to our problem: being able to export EARL doesn't affect UI so doesn't affect assumptions we make about target population. Using EARL as an underlying representation for new functionalities impacts the task that the tool would support. To specify this part of the tool I would need to know what we can do for that part of the interaction. A lesson I learned is that adding an interface feature may or may not be helpful, even if the function is important.
CMN Importing the EARL means that you can add it on internally, but may or may not have an interface component to help fix it.
GB In LFD we have a set of rules (e.g. WCAG, 508, ...) and for each one we have a package of information with it - describe the problem, what code applies, how to fix, etc. For some problems we have repair wizards. If I have to integrate tests described in earl coming from another tool, I need to have this kind of information, or the interaction will be significantly different. For example now you get the one-line description, editor positions to the problem in source and we open a description. If I don' have all of those, I won' be able to make all these things happen.
CMN each test has a URI, so if you have the test ID, you can see if you happen to have a wizard, even if you don't have a test for it.
GB It would be good to have a package of the ninformation instead of just having the test ID.
WC We have talked about a test description language
NH that would be really helpful
Karl Dubost We have discussed the interest of such a language, but don't have one yet. You should talk with Mary Brady (NIST) who has the beginnings. Ther are also people from the Batik project - they have a language to describe the tests that they are doing.
GB is that something on REC track?
KB Don't know - not sure it will be useful - because a general test framework will be difficult.
CMN It is possible to annotate the test ID with the test info package, and query it with the same way we are querying EARL. Might be best to start like that, then see if the test info is close enough to try and standardise, or whether each tool can publish its own package, and make or ask people to make annotations to extend the tool.
KB ISO have been working on a test description language - available for money.
GB In a couple of onths from now we will release the SDK for LFD so people can write their own tests. At that point, at least the structure we are using will be public if people pay, so that could be a starting point.
I haven't done much about EARL recently. I do RDF and querying stuff. So I ahve done demos of using and making RDF queries to find out about EARL.
Dan Brickley came up with a simpler way of dealing with some of the hard syntax bits, and I think that is interesting - I am intersted in annotation in general, and so it would be a good test case from that aspect.
WC We have talked about it and there is a feeling that it might be difficult to get pieces together. Developers were keen to get rid of reification, but don't want to go back and forth to the server.
LM Jim stores RDF in a database keyed from one field. I am using an RDF-based db, and query RDF directly.
Discussion of the annotea model of an annotation.
LM EricP probably stores it in a triple store, which is a lot more flexible - it makes it easy to query.
WC Jim' is really an XML store, optimised for particular queries
Libby has a query engine for asking for anything in RDF
Jim has a tool to store and return RDF
GB Why don't we proceed bottom up - focus on problems, see how they proceed, and work out where to go next? So looking at a wrapper for EARL, my burden is to produce EARL. The wrapper tool can provide some queries or comparisons and decide what kind can be done. If I have to implement queries, I have to see what is done and how to optimise it. In the wrapper tool someone can experiment with processes for specifying and querying.
WC Right, one of our goals is to show an end-to-end use case. We have started with implementations to get kinks out of the language
GB It isn' necessarily true that you can define different phases in isolation. I would consider this a prototype, and then we build a prototype application, then see where we need to go.
JL isn't sure algae is powerful enough to query assertors...
GB We might be able to give you a platform (this is an idea, not a proposal...). Say we give you LFD development, and tell you how to work on it, so it can be used to develop a wrapper. For example the interface for the wizard can be extended. I don't know yet what license conditions and so on are, but this might be worth exploring.
CMN so for example if it is easy for me to add one new wizard, and I use dreamweaver, and have a different tool to test one particular thing, then that seems helpful - LFD gets extended, I can use the other tool for the test, and I haven' had to do a whole lot of work for it all.
GB So I can make LFD output EARL, but I need a lot of examples so I know how to make it work.
WC I think that we should sort out the simplification stuff first. Can we talk about the language now?
Have a graph that we get now, want a graph that looks like @@
GB there are several tests for a given checkpoint - how do we deal with that
CMN you can define what the requirement for a checkpoint is in terms of the subtests. That schema can be published on the web as RDF.
LM It might be a good idea to have an XML schema as well as an RDF one. Then some processing can be done with XML processors, which are widespread.
GB so why RDF?
LM There are specific tools in RDF for merging information, and there is an explicit description of the model.
If we have the model wrong, when we merge reports they can lose things.
discussion of modelling
Action CMN: Write up how to state that a collection of earl results implies an earl result.
Issue: how do we deal with representing parts of pages and dealing with them as a whole page. hashing to identify chunks is messy in fitting it into earl.
$Date: 2002/02/28 14:17:30 $ Charles McCathieNevile