14:31:32 RRSAgent has joined #vcwg-special 14:31:37 logging to https://www.w3.org/2023/05/16-vcwg-special-irc 14:31:37 RRSAgent, make logs Public 14:32:08 please title this meeting ("meeting: ..."), ivan_ 14:36:52 Meeting: Verifiable Credentials Working Group Special Topic Call on Test Suites and CR Exit Criteria 14:36:52 Date: 2023-05-16 14:36:52 chair: kristina 14:36:52 Agenda: https://www.w3.org/events/meetings/f6342df0-f7b5-4fc9-babd-61e55dc5fc2f/20230516T110000 14:36:52 ivan_ has changed the topic to: Meeting Agenda 2023-05-16: https://www.w3.org/events/meetings/f6342df0-f7b5-4fc9-babd-61e55dc5fc2f/20230516T110000 14:59:20 hsano_ has joined #vcwg-special 14:59:37 andres has joined #vcwg-special 15:00:07 mprorock has joined #vcwg-special 15:00:57 present+ 15:01:31 present+ andres, mprorock, hsano, manu, tallted 15:01:38 present+ PaulDietrich 15:01:40 manu has joined #vcwg-special 15:01:43 present+ dlehn 15:01:46 present+ 15:02:00 present+ 15:02:30 present+ 15:02:35 decentralgabe has joined #vcwg-special 15:02:36 present+ 15:02:37 cabernet has joined #vcwg-special 15:02:39 present+ oliver, kristina, cabernet, will 15:02:39 present+ 15:02:44 present+ 15:02:58 oliver has joined #vcwg-special 15:03:00 present+ 15:03:23 pdl_ASU has joined #vcwg-special 15:03:31 present+ 15:03:32 present+ orie 15:03:44 kristina has joined #vcwg-special 15:03:45 dmitriz has joined #vcwg-special 15:03:49 present+ 15:03:49 Will has joined #vcwg-special 15:04:39 Aha - found it... 15:05:00 present+ 15:05:54 present+ selfissued 15:06:11 present+ JoeAndrieu, dmitriz 15:06:44 JoeAndrieu has joined #vcwg-special 15:07:26 present+ 15:07:38 Orie has joined #vcwg-special 15:07:45 present+ 15:07:46 guest+ Li_Wang 15:08:12 scribe+ 15:08:20 s/Li_Wang/Li_Wang_Hong/ 15:08:29 selfissued has joined #vcwg-special 15:08:35 Aisp_ has joined #vcwg-special 15:08:48 present+ 15:08:52 liwanghong has joined #vcwg-special 15:09:45 present+ nistor 15:09:57 mircea_nistor has joined #vcwg-special 15:10:31 Topic: CR Exit Criteria 15:10:59 kristina: for the agenda today we would like to make a decision whether the documents that we have are ready to go to CR, or whether we want to keep informative features in the draft. What are the criteria that we want to decide as a group? 15:11:28 q+ to propose something, generally at first. 15:11:33 q+ 15:11:45 kristina: We haven't focused the conversation on this topic. We don't have a goal to pass a resolution, instead to get people to share what they have in mind... 15:12:11 We want to strike a fine balance between loose requirements and rigid requirements... 15:12:12 ivan has joined #vcwg-special 15:12:24 s/We want/kristina: We want/ 15:12:43 Whatever we end up with, we want to have the document be a living document ... 15:13:06 s/Whatever/... Whatever/ 15:13:24 ack manu 15:13:24 manu, you wanted to propose something, generally at first. 15:13:54 q+ 15:14:52 validFrom is the starting date for which the claims the issuer has made are to be considered valid for the subject... it is not a security feature, it is information / data model. 15:15:06 present+ identitywoman 15:15:07 IdentityWoman has joined #vcwg-special 15:15:13 present+ 15:15:14 manu: for those who are going through this for the first time. The minimum bar is two demosntrate at least 2 interoperable implementations. For example `validFrom` has some normative statements in the specification. The min bar is for two implementation from different people that did not collaborate together, and both implementation pass the tests. If both implementers do so correctly, then it passes the minimum bar. 15:15:31 ... My suggestion is that we start there as a base line. 15:15:48 ack ivan 15:16:19 ivan: Maybe before the details, there is worth emphasizing some things for newer people. 15:16:42 kgriffin has joined #vcwg-special 15:16:58 present+ 15:17:03 ... The goal of CR is to prove to the outside world that this spec can be implemented in an interoperable way. 15:17:18 ... What Manu mentioned are tools in this direction. 15:17:55 ack orie 15:17:56 kristina: Yes. It's how do we prove to the outside world that this spec can be implemented. 15:18:14 Orie: 15:18:49 ... How you prove that normative statements are correctly implemented gets... tricky. 15:19:03 ... For example, you write tests, but the tests aren't correct. 15:19:29 ... Maybe we can align right now. Is it enough to read normative statements to align on the core data models? 15:19:35 q+ 15:19:50 ... What do we actually mean as interoperable? How do I understand evidence? 15:20:35 ... We would be well served spending time on defining what is required for a normative statements, specially for JSON-LD data models. 15:20:39 ack ivan 15:21:00 q+ 15:21:41 Sounds like matrix testing for each feature that has normative statements.... 15:21:43 Ivan: Another starting point beyond what Orie said, which I know we'll get. In our case, we can prove that VCs can be interchanged. Meaning, implementation A produces a VC using X, Y, Z features using Omega serialization. Then it's received by B, and they can understand it. 15:22:10 ... If we have a test that does this in practice, we are likely to cover all the pieces that are needed to prove interoperability. 15:22:26 Great point regarding "test vectors"... being the basis for testing normative statements. 15:22:40 ... That's the kind of interoperability that reviewer will look for. This was what they looked for during the DID CR phase. 15:22:51 ... Maybe we should concentrate on that from the start. 15:23:10 q+ 15:23:11 q+ 15:23:17 It is in scope. 15:23:20 for vc-jwt. 15:23:20 ack mprorock 15:23:29 kristina: I agree in practice the world view described. I do not think it's in the scope for the working group to try. 15:23:56 https://w3c-ccg.github.io/traceability-interop/reports/conformance/ 15:24:05 https://w3c-ccg.github.io/traceability-interop/reports/interoperability/ 15:24:24 mprorock: Appreciate the separation between conformance and interoperability. The scope will be the question. Will share a couple of links on conformance and interoperability tests. They do what Ivan just described. 15:24:36 to be clear, we don't need protocols, to do multi implementation matrix tests for test vectors. 15:25:03 q- 15:25:03 ack ivan 15:25:04 ... The downside is that protocols are needed and it's run using postman. Like Orie mentioned, you could run matrix tests using some tech like docker. 15:25:21 +1 Ivan - this can be done independent of protocols 15:26:12 ivan: I didn't mention protocols. It's true that we don't standardize them. There has to be some agreement on how the test suite is built to carry pieces of data from one place to another. 15:27:06 ... Put another way, if we cannot prove that there is an interplay between the various types of implementations, then I'm quite sure our spec will be rejected. 15:27:19 q? 15:27:21 q+ 15:27:23 ... We should not forget that. I'll stop there. 15:27:26 ack manu 15:27:48 How do we test a JSON-LD data model? 15:28:00 q+ 15:28:01 q+ 15:28:11 manu: We got into implementations very quickly. It's a good discussion, but trying to get the the exit criteria for the CR phase, are there other suggestions beyond the 2 independent implementations per feature? 15:28:31 ... If we agree, we could get more into the weeds. 15:28:32 ack mprorock 15:29:03 mprorock: At this point, 3 seems like a good number. Enough implementors and implementations. 15:29:13 q+ 15:29:21 ... Acknowledged that 2+ can get hairy very quickly. 15:29:40 q+ to warn that 'change later' is procedurally risky 15:30:01 ... We can improve on what the definition of independence is (like using the same lib or something). 15:30:22 there are few components "2" "independent" "implementations". is it 2? what is the definition of independent? to be called an implementation, is a PoC enough or does it have to be in production? 15:30:24 -1 to going beyond the usual standard of 2, but we can highlight where there are more than 2 15:30:25 ... There is a lot of shared libs and reuse in the space. 15:30:37 ack Orie 15:31:31 There's a distinction between the number of implementations and the quality of the tests that demonstrate their conformance. I'm not sure that increasing the number raises the bar. 15:31:38 -1 to trying to satisfy others by raising our own bar beyond what is typically required 15:32:02 +1 to trying to show more implementations anyway! 15:32:09 orie: We've felt the pain of doing tests in the past. It would be good to not get formal objections because of poorly written test or implementations that are not provably interoperable. 15:33:11 +1 to avoiding DIDcore pattern 15:33:31 ... Talking about the core data model. It has a single serialization. When we say tests, we are thinking about a concrete serialization. I propose we test vc+ld+json only. We test "what does it mean for two independent implementations to agree?" 15:34:33 note: we can just test that the context is set to the right value and test the specific rules for fields in the spec 15:34:38 ack manu 15:34:40 ... What does testing "The field has to be an IRI"? How do we give higher confidence to the fields? How do we add additional safety checks to the fields related to JSON-LD processing? 15:34:57 +1 manu 15:35:03 changes prior to CR 15:35:10 i meant change after today 15:35:30 manu: To Mike's statement "We can change it later". We cannot change the exit criteria after we enter CR. Before then, we can decide on the number of implentations. 15:35:54 How do we test JSON-LD Compact form? 15:36:03 i think ivan said interop with v1.1. do we want that to be in scope? that is a lot and our charter allows for breaking changes 15:36:06 ... To orie: "how do we test the spec?" -> We have added language to say you have to use json ld compact form, which means you can use jsonschema. 15:36:08 that's a very good point, that we have JSON Schema on our side this time 15:36:13 Can we resolve to use JSON Schema to test JSON-LD Compact form? 15:36:15 +1 to manu, IMO, testing should be even easier than it was in 1.0 for what Orie is concerned about 15:36:24 @Orie - I think we should 15:36:34 I do too. 15:36:42 +1 to using JSON schema 15:36:55 I would also like to drop 1.1 for this WG 15:37:08 but to be clear, that means nobody will every need to understand the exapnded term IRIs, that we are trying to make normative. 15:37:15 ... While I understand the need for better discussion, I would avoid that discussion. I would try to keep it more pragmatic. For example, run regexes against the time based fields. Something like check the JSON, check the context, check the schema, and as a result the json-ld is valid. 15:37:18 that seems like not a good thing. 15:37:34 drop 1.1 means what? 15:37:54 ... I think it's fairly straightforward in Data Integrity because you know the result if the signature is correct or not. 15:37:58 there are 1.1 tests already, focus tests for CR on 2.0 only 15:38:08 q? 15:38:11 ack ivan 15:38:11 ivan, you wanted to warn that 'change later' is procedurally risky 15:38:19 ivan: I have two statements. 15:38:45 I don't think testing anything related to JSON-LD is straight forward sadly.... especially document loaders, IRIS and how the context shape effects the graph, and how that graph matches the normative statements. 15:38:50 q+ to talk about "multiple independent implementations" 15:39:02 +1 to 2 minimum, even if our internal bar is higher 15:39:08 .... going back to the magic number. When we enter CR, we have to document the criteria we have agreed on. It cannot be changed. My advise is to keep that to a minimum. Let's say 2. If we have a test suite that covers more, then it's a safe bet. 15:39:10 +1 to Ivan, fully agree 15:39:19 +1 to 2 minimum, even if our internal bar is higher 15:40:00 Ivan, that is not correct... see the v1 test suite. 15:40:23 ... Going back to what Orie said. I would say the important part is not the testing of the Data Model. If we concentrate *only* in the interop, then we are automatically testing the VCDM. 15:40:27 the problem is that security suites don't cover understanding the VCDM. 15:40:40 that may true for data integrity ... not for JWTs, unfortunately. 15:40:41 only data integrity covers the term IRIs. 15:40:43 ... There might be some formal testing related to what Manu mentioned and work we could reuse. 15:40:45 nah, we're going to throw the v1.0 and v1.1 tests away :P (really) 15:40:53 ^ yes. 15:41:06 ... The interesting part is that implementations are interoperable between generating and receiving VCs. 15:41:31 IIRC we got to CR with each document... they don't need to all go together... if we plan correctly. 15:41:36 it might be testing just DI and JWT will cover the data model tests, i think is the point. 15:41:50 yes, that ^ 15:41:51 ... Over the 7-8 docs that we have, we'll go through the same things. For example, if we test BBS then we'll have automatically tested things in the VCDM 15:41:53 (DI being the thing that covers it) 15:42:13 ... It's up to the working group how we organize the testing suites. 15:42:17 q? 15:42:17 What's being said, is on the whole, we end up testing everything at an appropriate level. 15:42:17 yes, data integrity covers it, but then it does not need to be understood in the core data model, with normative statements about term IRIs. 15:42:21 ack manu 15:42:21 manu, you wanted to talk about "multiple independent implementations" 15:42:25 might need something for the JWT translation if we're still supporting that vs. using the base media type. 15:42:53 basically, don't try cheating, by putting core data model tests in data integrity. 15:43:45 testing core data model should not be bound to implementing DI 15:43:45 -1 to what manu just said... we are not going to say "we don't need to test vc-jwt, because we have data integrity". 15:43:51 manu: Re mike's statement. Ivan is saying that in total we will be testing that appropriate processing is done. We didn't have that in previous versions. By testing DI you will have to have done the transforms. Maybe that's not true for vc-jwt, but we've done all the testing around DI and the signatures will end up being tested. 15:43:59 +1 orie 15:44:05 Orie: he didn't say that ... he said we don't need to do data model tests twice 15:44:33 VC-JWT must be tested and show it can translate to the data model (or just secures it directly) ... and the data model tests are checked elsewhere. 15:44:37 q? 15:44:37 ... Mike's statement related to independent implementations and shared libraries. There is language in there that mentions that the libraries should be different. 15:44:43 I agree we don't need to do data model tests twice... but does that mean that the tests proposed in data integrity belong in the core spec? 15:44:54 q+ 15:44:57 s/translate to the data model/translate to the base representation/ 15:45:13 present+ 15:45:14 q- 15:45:32 sounds like we should run a poll regarding using JSON Schema to test the core data model. 15:45:44 Orie, to be clear, I didn't say "we don't need to test vc-jwt"... we do need to test it 15:46:09 +1 orie 15:46:15 I'm saying "If someone asserts that we didn't test the "JSON-LD bits", we can say -- we did in the DI test suites... you have to do those transforms there. 15:46:25 kristina: Not yet clear to me how we related different test suites in different documents. 15:46:29 min bar is 2, have to use diff libs, testing only vc+ld+jwt 15:46:35 +1 15:46:35 manu right, but the JSON-LD comprehnsion part, you asserted would not be covered in vc-jwt, because it was already covered elsewhere... that is what we should discuss... 15:46:44 +1 15:47:05 q+ 15:47:06 q+ 15:47:10 ack manu 15:47:19 yeah, I think we've got pretty strong agreement re JSON Schema re JSON LD 15:47:21 manu: Using something equivalent to json schema seems fine. 15:47:37 ... e.g. issuanceDate might just be a regex test. 15:47:40 "testing only vc+ld+jwt" is fine if that's all we're doing in the spec these days (if we're doing some mapping thing, that needs to be tested too) 15:47:57 ... I'd rather let implementers choose how best to test. 15:48:08 I think we should constrain the test suite authors, or prepare for lisp / scheme test suite. 15:48:10 "something like" JSON schema != json schema? 15:48:13 ... I'm fine to saying "using something like json schema do x and y" 15:48:35 +1 to Manu's point regarding JSON Schema not working, but why would that be the case? 15:48:39 I think Manu means "start with JSON Schema as base, and then sprinkle additional tests as necessary" 15:48:51 ... There may be some tests in this spec that cannot be tested with json schema. 15:48:59 +1 to use JSON schema for everything it can test 15:49:23 I would like to understand what JSON Schema CANNOT test... in a JSON-LD TR... 15:49:23 +1 to using JSON schema 15:49:29 ... What I mean if that you can do an equivalent test to JSON schema regex validation, without using JSON schema. 15:49:54 +1 to not writing tests for v0 or v1.1 15:49:58 kristina: There is a comment about versions. My suggestion is that interop between versions is not a goal. 15:50:00 We are writing tests for v2.0 15:50:08 (not for v1.0 or v1.1) :) 15:50:10 +1 to only writing tests for v2.0 TRs 15:50:31 everyone: (wide agreement to only testing v2.0) 15:50:40 q+ 15:50:45 ack orie 15:51:10 +1 orie 15:51:13 orie: I suspect talking about testing transformations would take the rest of the call. 15:51:26 double +1 15:51:38 if that's the only one that's normative in the spec (vc+ld+jwt), that's fine 15:51:48 we just can't have something that's normative and not test it. 15:52:03 ... I am strongly opposed to having normative tests until we have gotten to consensus on the mapping subject. 15:52:09 q+ 15:52:20 +1 orie 15:52:30 ack dlongley 15:52:32 I /think/ I agree with Orie? 15:52:36 ... People are asserting that mappings should be tested normatively. I would rather not do it. 15:52:54 This would mean only testing vc+ld+jwt, and not definining or testing vc+jwt. 15:52:58 dlongley: If we are saying that something is normative, we need tests. If normative, then no. 15:53:19 kristina: I encourage people thinking learnings during v1.1 15:53:24 s/If normative, then no."/If not normative, then no." 15:53:31 s/If normative, then no./If not normative, then no. 15:53:32 q+ 15:53:36 q+ to say without testing of normative mapping how is VC-JWT in scope 15:53:40 ack manu 15:54:03 We still have vague extension points. 15:54:09 JoeAndrieu: it's just a securing mechanism (a signature over vc+ld+json) 15:54:23 manu: What went wrong: we didn't have securing mechanisms. A JSON doc with extension points and fields. At the end of v1.0 and v1.1, we had a JSON object with well defined properties and extensions points. 15:54:41 my question is about the implementation/testing procedure. 1) do we actually want to use CLI (stdin + stdout) as the test mechanism, OR do we go the Universal DID Resolver route and require Docker containers which we're talking to via HTTP. and 2) Are we ok with "honor system" (implementers running the test suite themselves), OR are we having a central testing server (where ppl upload library, and it runs the suite) 15:54:41 dlongley is correct, current vc+ld+jwt is a JWS over cty vc+ld+json. 15:54:53 +1 to Orie -- that makes it simple. 15:55:09 ... What we're doing now is securing the credentials. It's part of the charter now. 15:55:36 If we are not testing v1 and v1.1, can I remove them from the current vc-jwt draft? 15:55:38 please? 15:55:43 ... There is a lot of processing to ensure this. And the charter now allows us to test a lot of the things that we wanted to test. Still a lot of work. 15:55:48 +1 to manu, but the charter _requires_ us to test not only _allows_ us :-) 15:55:57 true ^ :) 15:56:02 q+ to ask about testing procedure 15:56:02 q? 15:56:07 ack JoeAndrieu 15:56:07 JoeAndrieu, you wanted to say without testing of normative mapping how is VC-JWT in scope 15:56:12 Orie: might want to add a statement to the vc-jwt draft saying not to use them in new applications as well and to instead use vc+ld+jwt 15:56:21 (them = 1.0 and 1.1) 15:56:30 I prefer to just remove them. 15:56:46 ack dmitriz 15:56:46 dmitriz, you wanted to ask about testing procedure 15:56:46 joeandrieu: if we are not testing a mapping, then I'm not sured how we'd be able to secure it. 15:56:46 I'd be supportive of that ^ 15:57:29 dmitriz: Implementation details and testing procedure: Do we want to use the CLI (stdin and stdout)? Or do we ask implementers to contribute docker containers? 15:57:32 q+ 15:58:03 ... What's our general testing philosophy? Should implementers run things locally? Is there a testing server and people upload their library? 15:58:05 ack ivan 15:58:21 +1 Ivan 15:59:51 +1 Ivan, makes sense. 15:59:56 -1 :) 15:59:59 Ivan: The goal is to test whether the specification is ok or not. All implementers are working towards the same goal. It's ok for them to do it locally and tell us the result. 16:00:03 so then we should agree re CLI vs Docker/HTTP 16:00:03 I don't think that is good enough for v2.0 16:00:12 We made that mistake w/ v1.0 :) 16:00:20 "min 2 implementations using different libraries, test only vc+ld+jwt serialization, using JSON-schema-like to test json-ld, not testing the mapping to other serialization in vc-jwt, not testing anything v1.1" 16:00:36 +1 ^ 16:00:44 +1 ^ 16:01:01 +1^ 16:01:18 rrsagent, draft minutes 16:01:20 I have made the request to generate https://www.w3.org/2023/05/16-vcwg-special-minutes.html ivan 16:01:29 hsano_ has left #vcwg-special 16:05:01 zakim, end meeting 16:05:01 As of this point the attendees have been ivan_, andres, mprorock, hsano, manu, tallted, PaulDietrich, dlehn, dlongley, oliver, kristina, cabernet, will, decentralgabe, pdl_ASU, 16:05:04 ... orie, dmitriz, selfissued, JoeAndrieu, Aisp_, nistor, identitywoman, kgriffin 16:05:04 RRSAgent, please draft minutes 16:05:06 I have made the request to generate https://www.w3.org/2023/05/16-vcwg-special-minutes.html Zakim 16:05:13 I am happy to have been of service, ivan; please remember to excuse RRSAgent. Goodbye 16:05:13 Zakim has left #vcwg-special 16:05:15 rrsagent, bye 16:05:15 I see no action items