inspired by a political campaign where feedback from a blog created real-time changes in how the campaign was being run.
a) Each spec. independently testable;
b) Identify dependencies on other specs/groups at the testing level;
c) Create a process for coordination.
1 | ||
2 | "regrets" | |
3 | w3c should have a higher-level management that tracks test suites
and implementation reports. should not be working group owned, it
should be centrally located and searchable. tests will live beyond
the WG.
WG should not cease existence until they complete well-documented test suite. Issues with test suites at different levels of maturity - versioning or a completeness issue? does the test suite properly reflect the requirements in the specification? is there a formal process for validating test suites? |
|
4 | CSS tests are organized by metadata, try to avoid to use too
complex format support so as to as to test only the CSS aspects
[issue of testing only your spec and n ot the other technologies. css
- html or "host" tech overlap)
big efforts for VoiceXML were put into creating an abstract metalanguage that could be re-purposed for testing VoiceXML independently of the context it is used for identified who is likely to need joint test suites, who not. differences between Implementation Testing and Conformance Testing, but the former tends to get close to the latter when it is maintained and integrates e.g. bugs reports |
|
5 | The group basically exchanged ideas on good principles for test
suites
making the test suite open for contributions, and with that in mind make it as easy to contribute as possible, and not as easy to automate as possible. Joint testing perhaps not as important as doing good test suites and making it easy for others to contribute and use. But have to be careful with centralization. W3C style may be to define a common reporting scheme for groups that want to. |
|
6 | ||
7 | w3c develop test harness that can we used by all. w3c create test
suite management system.
w3c can help by: developing tools, educate WGs, "enforcement" to create test suites, some coordinates [QA] |
|
8 | resource needs: take advantage of cvs, bugzilla. finding
knowledgeable people - ppl who write specs often have different
skills that ppl who create tests.
managing dependencies is a resource demand. takes time to coordinate. to get the right people looking at it and determining what needs to be tested. if making a test suite, makes your spec better. you can refernce in the spec what to do. some groups only seem to do minimum needed for CR. Evidence that if create test suite during spec creation is it actually going to be a better spec? Needs to be central place [needs to be more awareness about QA and matrix?] |
|
9 | need a test suite coordination group.
WG should document a procedure for submitting new tests, so public can contribute |
|
10 | ||
11 | ||
12 | issue: what are we testing?
WBS-like tool for generating test cases. XML Binary - some be open source, some by proprietary some tests may mean different things to different ppl. may not always be the same output. test suites may be hierarchical - some global, some not if testing a protocol, not always possible to test behavior of an implmentation w/out a "standard" api |
|
13 | test suites can be used to learn a new technology. YEAH tests!!! | |
14 | test suites need to be in the culture of the WG, not just the
responsibility of a couple of people.
DAWG chair is insistent that WG members provide test cases on disputed issues. |
|
15 | ||
16 | See a need for a framework or environment into which you can drop
test suites to get
interoperability and composability. Need to think about how to integrate with technologies from other organizations. For example, Choreography needs to be able to generate BPEL endpoints, and uses reliable messaging. XML depends on Unicode. In SMiL, needed two tests for every feature, since the data format wasn't universal. |
|
17 | using the test suites to pressure browser makers to support
rendering + technology
need to make the test suite while you make the spec can't test formal semantics document. other expect ~250K test cases??? voicexml, ssml both develoepd by VBWG but aren't joint. |
|
18 | discussion about wcag testing, desire for xmlspec-like framework. difficulty of creating. perhaps limit scope tests? discussion of binary xml taking over the world. | |
19 | specs need to be testable. testable assertions should be included
in spec intself.
not all specs expressible as a set of features each of which may be implented or not. [are some specs not testable?] how do test suites incorporate optional features? |
|
20 | http://www.w3.org/XML/Group/2005/03/test-suite-lunch-20.txt | |
21 | concern about ppl outside of group misinterpreting arbitrary limits?
Not all requirements are machine-testable. versioning an issue concern about testing complex interactions. when putting together a variety of technologies, how do you test all of that? concern about decoupling test suites (and then merging back together?) |
|
22 | what are the features needed for a test suite? normative parts of
spec
if there is a dependency between test suites and there is a change, how manage? development of test suites should start early. person who writes the spec is best person is to write the tests. |
|
23 | questions about the question.who owns it? what constitutes a test?
suggest a mailing list for discussion of join test suites. cases where a test can pass but result isn't useful. [issue of spec or test?} In Schema before a change is made a test has to be created. Neil/Dave/Judy/Bill... we pretty much all agreed on nap and showering areas available in offices. Matt: -- Personally -- I think we need more tools for producing specs, cross linking, etc. Having spent a while hand tweaking the DPF spec, only to realize the XML Spec schema was out there was kind of a shock -- Ideally I could link my test suites easily into another specs test suites, etc. |
|
24 | proprietary issues. | |
25 | Lots of really good questions about what test suites are and how
they are generated.
Requirements for Joint Test Suites a) Each spec. independently testable; b) Identify dependencies on other specs/groups at the testing level; c) Create a process for coordination. |
|
26 | need for central repository to help manage test suite creationg.
xforms (embedded tech) creating tests that can be reused (where it will be embedded) Subsets... Is it OK to pass some tests and not all? It may be useful to separate the spec for a language (eg data formats) from the spec for a processor of that language (otherwise I can create a processor that does useful stuff but with a subset of the language). |
|
27 | how does a WG know what a future group will ask of their test
suite? [versioning an issue]
how find test suites? [unawareness of testing. is the QA page hard to find?] Reusability an issue. Ability to extend existing test suites. Online web service for test suites. |
|
28 | did it, but didn't get to the notes in time (before presentation) | |
29 | they talked about how the different groups they participated in could intergrate, e.g., SMIL, SVG, I18N | |
30 | ||
31 | test suites are good.
test suites as communication tool |
|
32 | it would be good to have a process document about how to write test
suites and how to link the tests to the concepts in the spec.
oftentimes it's hard to generate the test suite early before you know what the spec will look like. |
|
33 | Issue: IPR (ppl contributing), open source [learn more about the
test suite submission process]
Issue: could the w3c license test suites to peple? [emphasize Patent Policy] When publish a WD, should publish a test suite. Transforms spec into test suites. PPL don't have enough resources in own group, joint test suites be a lot more work. Some tests are boolean, some need a human interpretation. Is everyone aware of that? some tests are very hardware/software specific and difficult to replicate. encryption tests - some may be proprietary relationship of w3c standards and working with other oganizaations technologies and tests. interoperability not just w/our own specs, but with other people's. |