13:58:06 <RRSAgent> RRSAgent has joined #eval
13:58:06 <RRSAgent> logging to http://www.w3.org/2013/06/06-eval-irc
13:58:08 <trackbot> RRSAgent, make logs world
13:58:10 <trackbot> Zakim, this will be 3825
13:58:10 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 2 minutes
13:58:11 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:58:11 <trackbot> Date: 06 June 2013
13:59:21 <ericvelleman> ericvelleman has joined #eval
13:59:31 <Kathy> Kathy has joined #eval
14:00:20 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
14:00:29 <Zakim> + +31.30.239.aaaa
14:00:36 <MartijnHoutepen> Zakim, aaaa is me
14:00:36 <Zakim> +MartijnHoutepen; got it
14:00:44 <Zakim> + +31.30.239.aabb
14:00:46 <Zakim> +Kathy
14:00:58 <ericvelleman> Zakim, aabb is me
14:00:58 <Zakim> +ericvelleman; got it
14:01:51 <Zakim> +Peter_Korn
14:03:10 <MartijnHoutepen> Zakim, mute me
14:03:10 <Zakim> MartijnHoutepen should now be muted
14:05:10 <MartijnHoutepen> scribe: MartijnHoutepen
14:05:17 <MartijnHoutepen> yes
14:05:34 <MoeKraft> MoeKraft has joined #eval
14:05:41 <MartijnHoutepen> topic: Disposition of comments and editor draft
14:06:01 <MartijnHoutepen> EV: editor draft is in progress, probably before next meeting
14:06:03 <Zakim> +MoeKraft
14:06:39 <MartijnHoutepen> EV: we will discuss this in the next telcon
14:06:49 <MartijnHoutepen> topic: testrun 2
14:07:12 <MartijnHoutepen> EV: 4 people responded
14:07:13 <ericvelleman> <http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jun/0000.html>
14:07:14 <MartijnHoutepen> ack me
14:07:18 <korn> korn has joined #eval
14:07:32 <richard> richard has joined #eval
14:07:57 <MartijnHoutepen> http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Jun/0000.html
14:08:02 <MartijnHoutepen> https://www.w3.org/2002/09/wbs/48225/testrun2/results
14:08:07 <ericvelleman> Martijn presents the summary of website#2
14:08:53 <Zakim> +matthias_samwald
14:10:09 <richard> richard has joined #eval
14:10:37 <Detlev> Detlev has joined #eval
14:10:45 <Zakim> -matthias_samwald
14:11:02 <Zakim> +Peter_Korn.a
14:11:05 <Zakim> +[IPcaller]
14:11:07 <Zakim> -Peter_Korn
14:11:28 <korn> q+
14:12:00 <Detlev> Just peeking a bit over IRC today
14:12:10 <richard> zakim, IPcaller is Richard
14:12:10 <Zakim> +Richard; got it
14:12:31 <MoeKraft> q+
14:13:05 <MartijnHoutepen> PK: regular expressions maybe not needed, too formal
14:13:38 <MartijnHoutepen> MH: agrees
14:13:43 <MoeKraft> Korn: It is confusing to talk about regular expressions. Using this as an example of formalizations.
14:14:09 <ericvelleman> q?
14:14:10 <korn> We don't need the full glory of regular expressions
14:14:14 <korn> q-
14:14:22 <MoeKraft> Korn: Wouldn't say regular expressions aren't needed. Just don't need a whole lot of them.
14:14:51 <Zakim> +Peter_Korn.aa
14:14:53 <MartijnHoutepen> MK: we don't require them, but suggest re
14:15:00 <Zakim> -Peter_Korn.a
14:15:24 <MoeKraft> Moe: Regular expressions are not required but formalizations are recommended to identify the web pages in scope
14:16:12 <MoeKraft> Martijn: One thing to mention about Step 1, when we have a big website such as sub-domains you would need more contact with evaluation commissioner to define scope of evaluation.
14:16:31 <MoeKraft> Martijn: Since webiste is so big it is hard to know when to stop. Need input from commissioner.
14:17:49 <MoeKraft> MoeKraft: Martijn: Step 2, one evaluator listed common pages, others listed templates and how used. One evaluator looked at functionality, another looked a visual appearance, other evaluation looked at structure and sections of web pages.
14:18:13 <MoeKraft> MoeKraft: Martijn: not sure if guidance is clear since we have different results
14:18:37 <MartijnHoutepen> q+
14:18:49 <MoeKraft> Kathy: The comment regarding that the website is too big and need more help from commissioner, maybe we need to have guidance to break site into smaller pieces.
14:18:59 <MoeKraft> Kathy: Maybe have guidance on how to define scope of review.
14:19:07 <ericvelleman> q?
14:21:58 <MoeKraft> Korn: I think the problem is that exactly the problem we are tackling in the first place, We indicate that you have to sample.
14:23:29 <MoeKraft> MoeKraft: We did test this site and worked closely with the commissioner on defining the scope
14:23:53 <MoeKraft> Kathy: Maybe we need more guidance on communicating with the commissioner for example providing a set of questions to ask.
14:24:05 <ericvelleman> q?
14:24:09 <MartijnHoutepen> ack me q- moe
14:24:13 <MartijnHoutepen> q- moe
14:24:15 <MartijnHoutepen> ack me
14:25:05 <MoeKraft> Martijn: Sometimes the scope of the website is simple and you do not need to interact with commissioner but sometimes the website is so big that we do need to involve the commissioner and I think providing more guidance on this would be helpful.
14:25:17 <MoeKraft> Martijn: For instance asking questions of the commissioner.
14:25:37 <ericvelleman> Note: Involvement of the website owner and/or website developer can be useful to help identify common web pages, functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough evaluation.
14:26:28 <ericvelleman> help
14:27:00 <MoeKraft> Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.
14:27:30 <MoeKraft> Eric: This is very general. Is this enough?
14:27:56 <MoeKraft> Eric: Do we need more information here?
14:27:57 <MartijnHoutepen> scribe: MoeKraft
14:28:20 <richard> Q+
14:28:41 <MartijnHoutepen> +1
14:28:56 <MoeKraft> Should we add a statement that indicates based upon the size and complexity of the website that it would be good to ask questions of the commissioner to define the scope.
14:29:31 <Detlev> How do you meet the requirement for 'objective assessment', given that 4 evaluators arrived at different results?
14:29:38 <MoeKraft> Richard: No good to have components that are accessible and others that are not. It is important that the commissioner define what the elements are that belong in the scope of the test.
14:29:56 <Kathy> q+
14:30:16 <MartijnHoutepen> q- rich
14:30:26 <MoeKraft> q+
14:30:30 <ericvelleman> q?
14:30:55 <MoeKraft> Richard: Concerned that we are getting away from our guidance to look at all components.
14:31:02 <MartijnHoutepen> q- kath
14:31:13 <MoeKraft> Kathy: Not clear when to start and stop. Good idea to have questions to come up with representative example of all page.s
14:32:10 <MoeKraft> Kathy: There is a lot of information we can get from the commissioner before evaluating a site
14:32:28 <MoeKraft> Eric: This is used regularly in our testing in the Netherlands when to start and stop
14:32:40 <ericvelleman> q?
14:32:53 <MartijnHoutepen> q+ korn
14:33:33 <MartijnHoutepen> q- moe
14:34:20 <MoeKraft> Eric: Need help from the commissioner
14:34:28 <MartijnHoutepen> q- korn
14:35:00 <MoeKraft> Just to respond to Richard is that we can get pulled into many different websites both owned by IBM and not and need guidance from commissioner as to what belongs to support website.
14:35:24 <MoeKraft> Korn: It is entirely appropriate for team to scope what pages it will look at in a given moment in time.
14:35:32 <Tim> Tim has joined #eval
14:36:09 <MoeKraft> Martijn: Common functionality. Section 2b. Responses are pretty much the same from all respondents.
14:36:20 <MoeKraft> Martijn: Step 2b
14:37:17 <MoeKraft> Martijn: Step 2c Variety of web pages. One evaluator referred back to Step 2a where pages were already mentioned. Not sure if this means we don't really need 2c. Other responders described templates used.
14:38:06 <MoeKraft> Martijn: Step 2d is found to be confusing
14:38:15 <MoeKraft> Martijn: Technologies Used
14:39:12 <Detlev> Richard and I seem to agree that it would be the *custom* techniques that need to be listed
14:39:43 <richard> q+
14:40:13 <MoeKraft> Moe: When looking at common functionality want to be careful to not test functionality but to test the access to the functionality
14:40:36 <MoeKraft> Richard: Be careful not to call Common Functionality but instead Important Functionality
14:40:56 <MoeKraft> Richard: Receiving many comments on this section from the public comments.
14:41:20 <MartijnHoutepen> in the disposition of comments
14:41:25 <MoeKraft> Eric: Is there an alternative proposal in the results page?
14:41:55 <Zakim> +Tim_Boland
14:41:57 <MoeKraft> Richard: No. In the disposition of comments.
14:42:23 <ericvelleman> http://www.w3.org/WAI/ER/conformance/comments-20130226
14:42:42 <MartijnHoutepen> http://www.w3.org/WAI/ER/conformance/comments-20130226#comment3
14:43:08 <MoeKraft> http://www.w3.org/WAI/ER/conformance/comments-20130226#section1cont
14:43:11 <MoeKraft> Look at ID 3
14:43:29 <MoeKraft> Eric: Concerned that people don't see Section 2 as the Exploration of the website.
14:43:43 <MoeKraft> Eric: The third step should product the sample.
14:43:50 <MoeKraft> Eric: Maybe not clear for everyone.
14:43:59 <MoeKraft> q+
14:44:06 <ericvelleman> q?
14:44:15 <MartijnHoutepen> q- rich
14:44:25 <MartijnHoutepen> q - moe
14:44:28 <MartijnHoutepen> q- moe
14:45:18 <MartijnHoutepen> q+
14:46:11 <Zakim> +Peter_Korn.aaa
14:46:16 <MoeKraft> Eric: Let's just flag this one.
14:46:17 <Zakim> -Peter_Korn.aa
14:46:18 <ericvelleman> q?
14:46:24 <MartijnHoutepen> ack me
14:46:41 <richard> +1
14:47:09 <ericvelleman> Eric: Look into results of step 2. Do we have to put down everything there? flag this for review if we want to put everything there. Should it not be basis for sample?
14:47:11 <MoeKraft> Martijn: Maybe partly because in the results of the questionnaire we were asked what everyone did maybe we didn't have to note the list but this was asked of the questionnaire.
14:48:00 <MoeKraft> Moe: Step 2a states "During this step the common web pages of the website are identified and documented."
14:48:37 <korn> q+
14:48:49 <MoeKraft> Martijn: Section 3: Selecting a sample. Two people gave URIs another person selected all pages. Last evaluator selected a small sample and guidance how to get to pages.
14:49:28 <MoeKraft> Martijn: People who gave URIs gave them to pages identified in sample 2. Comment regarding small sample, describes pages but no URIs.
14:50:12 <MoeKraft> Eric: Do we have an idea that these samples would miss things? Or would they cover the same things?
14:50:27 <MoeKraft> Martijn: Different scope produced different samples.
14:50:53 <MoeKraft> Martijn: Detlev had a much more in depthy scope and produced different results.
14:51:12 <Detlev> My hunch is that we would uncover many of the non-conformant things regardless of the exact shape of the sample...
14:51:20 <ericvelleman> q?
14:51:37 <MoeKraft> Martijn: May need to be more explicit.
14:52:33 <MoeKraft> Korn: Peter is looking at answers. A couple answers give a lot of pause. 1 evaluator is expecting to test all pages of web site.
14:52:51 <MoeKraft> Richard: Self contained element should be tested.
14:53:18 <MoeKraft> Korn: "All pages in the application will be evaluated. "
14:53:39 <MoeKraft> Richard: Scope was already defined in Step 1. So all pages in application were already defined.
14:54:59 <MoeKraft> Richard: Portal gave access to all support website. Did not identify all of ibm web site. Went by scope that indicated ibm.com/support.
14:55:18 <MoeKraft> Richard: Only checked the portal.
14:55:23 <MoeKraft> Richard: 9 pages
14:55:43 <MoeKraft> Eric: Home, Support, Downloads, etc. Tabs at top of pages
14:55:58 <MoeKraft> s /ibm/website2/
14:56:41 <MartijnHoutepen> q+
14:57:57 <MartijnHoutepen> q- korn
14:58:08 <MoeKraft> Korn: Find the product and other links on this page. Were these in scope or out of scope of the test?
14:59:03 <MoeKraft>  Richard: Looked at horizontal menu, 7 tabs, and opened each tab, and followed links that went to portal. Downloads and Troubleshoot went to external areas.
14:59:14 <MoeKraft> Richard: Essentially looked at application as a portal.
15:00:09 <MoeKraft> Martijn: I agree with Richard. It is a portal. I did the same as Richard but also included some more static pages and inclued some functionality from main page, signing in, look for products.
15:00:18 <MoeKraft> q+
15:00:23 <MartijnHoutepen> q- mart
15:00:44 <MoeKraft> Eric: Interesting to see the differences in sample.
15:01:44 <MoeKraft> Moe: This brings us back to our first discussion regarding communicating to the evaluation commissioner to define the scope.
15:01:58 <MoeKraft> Eric: Looking for volunteer to write a summary of results for website 3.
15:02:31 <MoeKraft> Eric: Will search for a volunteer outside this telco.
15:02:41 <Zakim> -Kathy
15:02:42 <Zakim> -MoeKraft
15:02:42 <Zakim> -Peter_Korn.aaa
15:02:43 <Zakim> -Richard
15:02:51 <MartijnHoutepen> trackbot, end meeting
15:02:51 <trackbot> Zakim, list attendees
15:02:51 <Zakim> As of this point the attendees have been +31.30.239.aaaa, MartijnHoutepen, +31.30.239.aabb, Kathy, ericvelleman, Peter_Korn, MoeKraft, matthias_samwald, Richard, Tim_Boland
15:02:52 <korn> korn has left #eval
15:02:59 <trackbot> RRSAgent, please draft minutes
15:02:59 <RRSAgent> I have made the request to generate http://www.w3.org/2013/06/06-eval-minutes.html trackbot
15:03:00 <trackbot> RRSAgent, bye
15:03:00 <RRSAgent> I see no action items