See also: IRC log
<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130611
<MartijnHoutepen> diff version: http://services.w3.org/htmldiff?doc1=http%3A%2F%2Fwww.w3.org%2FWAI%2FER%2Fconformance%2FED-methodology-20130219&doc2=http%3A%2F%2Fwww.w3.org%2FWAI%2FER%2Fconformance%2FED-methodology-20130611
eric: 'common functionality' to 'core functionality' discussion
<Vivienne> Eric, do you have a link to the disposition of comments?
http://www.w3.org/WAI/ER/conformance/comments-20130226 disposition of commetns
detlev: thinks that Step 3 does not include enough dynamic states
vivienne: additional comments in email from Ryladog(?) should be added to disposition of comments
<Vivienne> Sarah, that was comments from Giorgio Brajnik
eric: will add to comments doc
<Detlev> fine!
<MartijnHoutepen> +1
<Liz> +1
<Vivienne> +1
<ericvelleman> +1
<Mike_Elledge> +1
eric: proposing to add to comments even though public comment period is over
+1
<MoeKraft> +1
<Detlev> yes!
<richard> +1
<ericvelleman> http://www.w3.org/WAI/ER/conformance/comments-20130226
eric: Disposition of Comments id#3 replace 'common functionality' with 'core functionality' - also mentioned a couple of other places in comments; there will be a survey to decide this
peter: appreciates example of going from common to core. is the expectation that all comments will have a proposed resolution in the next survey?
eric: yes, proposed resolutions will be in the survey linked to the editor draft. Eric will group more of the comments for the survey.
<ericvelleman> : <http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0046.html>
eric: agenda pt 3 Design support evaluation versus conformance evaluation
<Vivienne> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0046.html
detlev: three different items on goals (basic, detailed, and in-depth report); initial vs final evaluation for finished content. maybe reorder and rephrase report names.
peter: agrees with primary two uses, but having 3
choices are the wrong reports to have. Having use cases dictate the report type
is useful.
... still concerned with the word, 'conformance' since we aren't looking at
every page
... talk about sampling, problems found, and confidence in results found.
shadi: it would be an accurate representation of
how well the site conforms overall.
... agrees with use case driven reports. how about when product owner wants to
do an interim review to find out how the site is progressing.
<korn> +1 to that use case
shadi: +1 use case
<MartijnHoutepen> +1
<Detlev> "Define the Goal of the Evaluation" -> 1. Development support evaluation 2. Conformance evaluation?
vivienne: agrees with Detlev. might want to do a quick pass during development (basic report), but also could have an assessment of functionality after it's completed. wants to emphasize that this protocol is useful during development too.
katie: conformance discussion - it's ok that this protocol is associated with performance, but want to say that it's for the pages tested.
peter: concern is the language of WCAG conformance is that you can only make a claim if it's perfect. we need to have language that is specific to the parameters of the sample.
<Ryladog> I agree with this be associated with conformance for the pages tested. WCAG Conformance is per page
<ericvelleman> :-)
detlev: agrees with peter, not to focus so much
on conformance. the value is showing that the site is pretty darn good. define
the goal of evaluation and report types is a bit of a mismatch.
... development support evaluations and full/formal evaluations
... evaluation of a legacy site is another variation. they know the site is
bad, but still want to do the test
vivienne: can't the product owner still say that the site was evaluated according to WCAG-EM with the specific sample?
<Detlev> Conformance clains according to WCAG 2.0 apply to individual pages anyway...
peter: asks for clarification of Vivienne's comment
vivienne: if i evaluate the website according to WCAG-EM, and we produce a report with the problems identified, then the site is retested a new sample later after the problems were fixed, can't they make a claim that the website for that sample was compliant with WCAG A, AA or whatever?
peter: can't claim conformance for the site from the collection of pages
<Ryladog> You can say for the pages tested on that date
<shadi> +1 to Eric
eric: can't ever say everything is perfect, but you can say that the pages tested are good.
peter: have to look at every page before conformance for the site can be claimed.
vivienne: when a very indicative sample of pages is chosen, then the testing/retesting is done, they should be able to make a claim about the site based on the sampling.
eric: we can say something like this, but it is all about the confidence in the sampling. maybe we need to adjust/clarify this in the draft
richard: agrees with parking this discussion since it's not on the agenda
shadi: need to provide a template where evaluators can indicate the number of pages, sampling x amount of pages, etc., that can be filled out. 50 pages vs 1 page review will give a strong indication of the confidence in the sampling.
eric: 'confidence' needs to be expanded in the document
detlev: basic and detailed evaluation report types; support evaluation could be used for legacy sites
<Zakim> shadi, you wanted to talk about when an evaluation is done, and how rigorous it is
shadi: use cases for evaluation - evaluations
done at different types in time. is there another dimension regarding the
'depth' of the evaluation?
... agrees with detlev and peter that having use cases is good, but what about
other variations other than just having a basic report. depth vs point in time
when eval is taking place
detlev: true for different levels of depth or to show that the site is terrible, but normally the aim is to cover the main templates and functionality. want to avoid making this too complex or complicated. likes the idea of a score for level of conformance.
peter: likes shadi's third use case related to regression, i.e., this site has improved a little, a lot, or has gone backwards. likes the idea of showing progress.
eric: agenda #4. To sample or not to sample - <http://lists.w3.org/Archives/Public/public-wai-evaltf/2013May/0028.html>
vivienne: we cover it fairly well by saying that
sometimes the sample is the whole website, but when it's not feasible, then a
sample is appropriate.
... recommends researching the effect of sample size and it's agreement with
results from evaluating every page.
sarah: i meant 'its' rather than 'it's'
martijn: this is covered well already in the document. likes the way it is in the doc right now.
richard: just came through a series of
evaluations where it was cheaper to do the whole site, but there are formulae
for determining the proper sampling size. evaluator has to do a cost-benefit
analysis of conducting the eval on the whole site vs coming up with a sampling
strategy.
... when doing a random selection, you'll end up with some pages that you had
already pre-selected, which is okay.
... evaluator needs to decide on the sampling/whole site eval strategy
sarah: agrees with leaving the sampling up to the evaluator - and leaving the doc alone.
eric: will make a editor draft, survey, etc. for the next meeting, which will be on June 27 (not June 20). meeting in two weeks.