14:48:57 <RRSAgent> RRSAgent has joined #vcwg-special
14:49:01 <RRSAgent> logging to https://www.w3.org/2023/08/08-vcwg-special-irc
14:51:10 <ivan> Meeting: Verifiable Credentials Working Group Special Topic Call on PR Discussions
14:51:13 <ivan> Date: 2023-08-08
14:51:31 <ivan> Agenda: https://www.w3.org/events/meetings/f6342df0-f7b5-4fc9-babd-61e55dc5fc2f/20230808T110000/
14:58:17 <hsano> hsano has joined #vcwg-special
14:58:21 <ivan> present+
14:58:25 <ivan> present+ brent
15:00:02 <hsano> present+
15:01:41 <JoeAndrieu> JoeAndrieu has joined #vcwg-special
15:01:43 <seabass> present+
15:02:01 <andres> andres has joined #vcwg-special
15:02:05 <seabass> need a scribe?
15:02:05 <andres> present+
15:02:08 <JoeAndrieu> present+
15:02:16 <ivan> present+ andres, manu, JoeAndrieu, phil
15:02:20 <pl-ASU> pl-ASU has joined #vcwg-special
15:02:24 <pl-ASU> Present+
15:02:27 <andres> Am I the only one not hearing anything?
15:02:30 <ivan> present+ dlongley
15:02:52 <pl-ASU> +1 to Ivan's comment ;-)
15:02:53 <manu> present+
15:04:37 <ivan> present+ dmitri
15:05:38 <cabernet> cabernet has joined #vcwg-special
15:05:42 <cabernet> present+
15:05:47 <ivan> present+ oliver
15:05:56 <ivan> present+ chris
15:06:06 <andres> scribe+
15:06:36 <andres> brent: welcome to special topic call: test suites
15:06:39 <ivan> present+ benjamin
15:06:57 <andres> ... we last decided what the exit criteria was
15:07:29 <ivan> Topic: Test suite
15:07:30 <andres> ... that conversation was a precursor to what we'll be talking about today: test suite.
15:07:52 <andres> ... In order to exit CR we need to demonstrate that data models we produce are interoperable by having a test suite.
15:08:04 <ivan> present+ orie
15:08:11 <andres> ... Implementations can then show they are compliant with the data model.
15:08:24 <manu> q+ to go over one of the test suite proposals sent to the mailing list.
15:08:31 <andres> ... Any rec track document we produce will have testing associated to it.
15:08:52 <DavidC> DavidC has joined #vcwg-special
15:08:57 <DavidC> present+
15:09:07 <seabass> q+
15:09:10 <andres> ... Do we want one repo for all test suites? One suite to rule them all? Let's decide all the details. Can be docker, http, chapi, etc... everything is on the table.
15:09:14 <manu> q- later
15:09:26 <seabass> q+ to bring up criteria for choosing
15:09:28 <ivan> present+ gabe
15:09:33 <andres> ... manu sent an email to the group explaining what digital bazaar did for DI.
15:09:35 <decentralgabe> decentralgabe has joined #vcwg-special
15:09:38 <decentralgabe> present+
15:09:52 <andres> ... The goal for today: get as close as we can to what we want this test suite to look like.
15:09:53 <Orie> Orie has joined #vcwg-special
15:10:06 <andres> ... Implementation trumps opinion in most cases.
15:10:26 <manu> scribe+
15:10:30 <andres> ... That is, we appreciate opinions, but we do need the test suite.
15:10:57 <andres> seabass: You mentioned what are the criteria. I would say reproducibility is one of the goals
15:11:01 <bigbluehat> bigbluehat has joined #vcwg-special
15:11:19 <andres> ... HTTP could be, but that relies on external systems that the w3c doesn't control.
15:11:21 <Orie> q+ to talk about vc-jose-cose test suite planning
15:11:32 <andres> ... Docker would actually make it more reproducible.
15:11:33 <brentz> ack seabass
15:11:33 <Zakim> seabass, you wanted to bring up criteria for choosing
15:11:34 <brentz> ack manu
15:11:34 <Zakim> manu, you wanted to go over one of the test suite proposals sent to the mailing list.
15:11:46 <andres> manu: I'm going to share my screen so I can show folks where we are with our proposal.
15:11:58 <andres> ... email went out about a week ago.
15:12:47 <andres> ... we propose one test suite per specification so authors can work independently.
15:13:14 <andres> ... E.g. if you work on vc-di-eddsa then you have a suite for it.
15:14:01 <andres> ... Putting all the suites together would cause people stepping on each other.
15:14:24 <andres> ... Splitting it would help parallelization of the work. And would leave the test suite responsibility to the authors of the spec.
15:14:30 <seabass> +1 to manu: the suites can be merged after we've finished them completely
15:14:44 <andres> ... Currently, the exit criteria is multiple independent implementations.
15:15:09 <andres> ... <shows the status of 6 different implementers>
15:16:07 <andres> ... For each normative statement we create a row in a table. The columns correspond to each implementer. The value is a pass/fail checkmark/X.
15:16:44 <andres> ... One of the problems we had in the vc1.x is asking people to run and give us a report. So implementers never came back after running the report.
15:17:09 <andres> ... This suite is different because we're the ones deciding when to run the suite. And how often.
15:17:42 <andres> ... <shows a similar table for vc-di-eddsa>
15:18:01 <andres> ... <and again for vcdm2.0>
15:18:36 <andres> ... To Sebastian's comment regarding Docker containers, that's in the roadmap.
15:18:56 <andres> ... We believe the docker images are easy to integrate with the existing tests.
15:19:02 <seabass> q+
15:19:13 <andres> ... The test runner would spin up the container and execute the HTTP calls.
15:19:47 <andres> ... The proposal is for this to be one among many test suites. One test suite per specification for max flexibility.
15:19:58 <brentz> ack Orie
15:19:58 <Zakim> Orie, you wanted to talk about vc-jose-cose test suite planning
15:20:04 <andres> brent: <some german words saying thank you>
15:20:18 <andres> orie: I want to focus on the vc-jose-cose item only for now.
15:20:37 <andres> ... I think it would great to test all the media types related to the spec.
15:20:56 <andres> ... Also important to have support for cryptographic agility (i.e. all the signing formats).
15:21:42 <andres> ... There are issues with cred schema and cred status from a testability perspective. I'd like to see something covering interop for both.
15:22:09 <Orie> https://github.com/danielfett/sd-jwt/blob/main/tests/testcases/array_recursive_sd_some_disclosed/specification.yml
15:22:09 <andres> ... We're required to test normative statements. But sometimes that's not as valuable as covering the important test cases.
15:22:36 <andres> ... Sharing a link to what sd-jwt did. It shows what the input is, and output after specification.
15:22:44 <manu> q+ to speak to status list, testing of normative statements, and going beyond normative statements.
15:23:04 <andres> ... I'd like to see something similar for normative and important cases.
15:23:23 <andres> ... The test suite should be helpful, not only covering normative statements.
15:23:53 <andres> ... Another point of concern is source of randomness. We should plan for implementations to not be able to backdate.
15:23:58 <dmitriz> dmitriz has joined #vcwg-special
15:24:03 <Orie> https://identity.foundation/JWS-Test-Suite/
15:24:07 <andres> ... It should be easy to take an implementation and show conformance.
15:24:27 <andres> ... I also wanted to share two links
15:24:39 <Orie> https://github.com/transmute-industries/vc-jwt-test-suite
15:24:46 <andres> ... The test suite at DIF does secure JWT, so a similar approach could be done here.
15:25:50 <brentz> +1 to closed-source implementations being supported
15:25:54 <andres> ... The second link shows accepting reports. Some security folks will not share the docker image. While that's not reproducible, they aren't willing to participate if they need to share.
15:26:15 <andres> ... I think we're pretty far behind, but really excited to work on this. Certainly for vc-jose-cose.
15:26:16 <brentz> ack seabass
15:26:30 <andres> seabass: Responding to 2 thing.
15:27:23 <andres> ... Re: docker images, there are additional advantages. As implementer, you don't need to run a web service. Also, the WG can look back as long as we want to.
15:28:01 <andres> ... Re: orie's points. Thanks for sharing the yaml file for sd-jwt. We have to be careful that we aren't testing details about an implementation.
15:28:32 <brentz> ack manu
15:28:32 <Zakim> manu, you wanted to speak to status list, testing of normative statements, and going beyond normative statements.
15:28:40 <andres> ... Finally want to note that being closed source should not stop reproducibility. We can have NDAs in place to address, or confidentiality agreements.
15:28:43 <Orie> I suspect you won't get the code, but you might get a vendor report run
15:29:20 <andres> manu: We did have a test suite for status list ready to go. But the many changes haven't been incorporated (e.g. multi-status). We need to update it.
15:29:26 <Orie> q+
15:30:16 <andres> ... If status list needs to be secured, then we need to be able to write tests that ensure they are interoperable.
15:30:37 <ivan> +1 to Manu about what this group needs to do
15:30:50 <andres> ... This group only needs to test normative statements. While I'm supportive of integration and unit tests, it's probably not this WG's responsibility.
15:31:17 <andres> ... We can have another suite that goes above and beyond.
15:31:24 <brentz> q+
15:31:27 <Orie> YEs, split the suites up... cover the mandatory W3C stuff first.
15:31:32 <andres> ... It can include edge cases, fuzzing, etc...
15:32:11 <andres> ... In our examples, we also do n-by-n testing across implementations.
15:32:26 <andres> ... It has helped identify gaps in our test suite.
15:32:55 <brentz> ack Orie
15:33:04 <andres> orie: Agree with everything manu said
15:33:10 <ivan> q+ identitywoman
15:33:33 <andres> ... Clarifying a point. We have normative statements in different specs. Some specs are extensions.
15:34:12 <andres> ... In the context of a securing specification, unless there provable interoperable implementations, then I wouldn't be comfortable with the implementation.
15:34:48 <andres> ... There is some assurance in the securing specification that there will be interop with the VCDM with extensions.
15:35:04 <manu> q+ to note "uncomfortable normative statements in the VC v2.0 specification" and "we want to get better, but might not make perfect"
15:35:10 <ivan> q- identifywoman
15:35:10 <ivan> present+ identitywoman
15:35:21 <andres> ... That's an area I'd like to see better execution. You want to believe that an extension point exists, and that can be secured interoperably.
15:35:24 <manu> q- identitywoman
15:36:03 <andres> ... You can do it by looking at the securing specification under the lens of doing validation and signing of all the VCDM with extensions.
15:36:07 <brentz> ack brentz
15:36:28 <andres> brent: Thanks for everything that has been shared.
15:37:14 <andres> ... The primary goal is to test suite is to test the specification. We do that by testing each implementation. Everything else beyond that is gravy.
15:37:14 <brentz> ack manu
15:37:14 <Zakim> manu, you wanted to note "uncomfortable normative statements in the VC v2.0 specification" and "we want to get better, but might not make perfect"
15:37:36 <andres> manu: +1 to brent. However, we're at a point where I think we should do better.
15:37:40 <ivan> q+
15:38:22 <Orie> +1 manu, lets cover the basic requirements, and then lets address the issues that implementers have, and enable them to contribute to what they want to see tested.
15:38:28 <andres> ... We should allow us to do better. We don't need to as a group, but shouldn't block it.
15:38:43 <andres> ... Where do we want to go with interop testing as a group?
15:38:43 <seabass> +1 from me
15:39:14 <andres> ... In implementing the VC 2.0 model tests. The engineers were uncomfortable with the extension point tests.
15:39:29 <Orie> +1 to the extension point tests being dangerous... and possibly not useful, unless addressed better.
15:39:30 <ivan> present+ kristina
15:39:36 <ivan> chair: brent
15:40:10 <andres> ... We need to talk more about this as a group. There is no clear solution that would achieve consensus. We've tested the type of the object and that's about it.
15:40:27 <Orie> To be direct, the problem with the extension points is having no "output format" to compare to... for tests.
15:40:43 <andres> ... The only extension points we could do better is credentialSchema and status list.
15:40:50 <brentz> ack ivan
15:40:55 <andres> ivan: Something a bit different.
15:41:19 <identitywoman> identitywoman has joined #vcwg-special
15:41:27 <andres> ... Among other things, we define vocabularies. The way they're handled is to prove that the terms defined in the vocabulary are used by actual implementations.
15:41:32 <Orie> q+ to ask about testing normative context and vocab
15:41:45 <manu> q+ to note "systematically cover all terms defined" and how we're indirectly testing that now.
15:41:47 <andres> ... I don't know if the tests systematically cover all the terms that we've defined. Even those we've described as optional.
15:42:10 <andres> ... Maybe we need a separate report that for each term it makes a statement specifying which terms are used by which implementation.
15:42:13 <brentz> ack Orie
15:42:13 <Zakim> Orie, you wanted to ask about testing normative context and vocab
15:42:29 <andres> orie: So glad you raised that. There's a series of issues open related to this.
15:42:52 <ivan> q+
15:42:56 <andres> ... One category is the value of the json model value after processing.
15:43:12 <andres> ... I can imagine us testing the normative context with framing and processing.
15:43:24 <brentz> q+ to ask for a summary of the plan
15:43:30 <andres> ... I would want us to do it if the context ends up being normative.
15:43:52 <andres> ... Re: vocabulary. That's an area I believe we should invest on.
15:44:31 <andres> ... Folks should see the terms that are being tested very clearly.
15:44:38 <brentz> ack manu
15:44:38 <Zakim> manu, you wanted to note "systematically cover all terms defined" and how we're indirectly testing that now.
15:45:13 <Orie> -1 to citing data integrity tests as a way to "ensure the normative context is useful".
15:45:25 <Orie> but its a good start / or in addition too.
15:45:26 <andres> manu: Reacting to ivan and orie. We do some level of testing already for the context because we are doing RDF expansion.
15:45:54 <andres> ... We are testing that today in DI.
15:46:07 <Orie> +1 to testing that the vocab can be loaded / processed.
15:47:10 <Orie> q+ does that mean that JSON-LD processing is always mandatory for expanded term defintions?
15:47:16 <Orie> q+ to ask does that mean that JSON-LD processing is always mandatory for expanded term defintions?
15:47:20 <andres> ... More ideas are welcome on how we can better test (and contributions)
15:47:21 <brentz> ack ivan
15:47:30 <andres> ivan: I think I wasn't understood.
15:47:51 <andres> ... I'm not talking about testing. I'm talking about the CR exit criteria. These two aren't the same.
15:48:09 <andres> ... We have to prove that every term in the vocabulary has been designed for a reason.
15:48:31 <andres> ... We need a property table that justifies which implementations uses what.
15:49:03 <manu> q+ to respond to brent and summarize "the plan" (as it stands today).
15:49:16 <andres> ... In other groups we have relatively large sets of metadata terms that are used for a11y. We had to show which implementations used which terms. That was the reason why we had each term in the vocab.
15:49:19 <Orie> I think "JSON" processing will NOT use ANY of the vocabulary terms...
15:49:36 <Orie> at least the URLs for the term definitions... is that ok?
15:49:39 <brentz> ack brentz
15:49:39 <Zakim> brentz, you wanted to ask for a summary of the plan
15:50:02 <andres> brent: Good convo. From what I've heard, there is a common direction where we're heading. Can someone summarize?
15:50:04 <brentz> ack Orie
15:50:04 <Zakim> Orie, you wanted to ask does that mean that JSON-LD processing is always mandatory for expanded term defintions?
15:50:27 <andres> orie: My question is related to Ivan's clarification.
15:51:04 <andres> ... Some implementations will test some terms directly.
15:51:17 <ivan> q+
15:51:30 <brentz> ack ivan
15:52:02 <andres> ... Do we have to prove that implementation use URLs that JSON ld processing is producing?
15:52:08 <andres> ivan: No, we don't.
15:52:12 <dlongley> the existence of syntactic sugar to make things simpler for users does not mean users must unwind that sugar.
15:52:20 <dlongley> the meaning is the same.
15:52:20 <andres> ... That isn't our WG's problem.
15:52:22 <oliver> oliver has joined #vcwg-special
15:52:23 <oliver> present+
15:52:34 <andres> ... The really important point is to see whether those terms are used in practice.
15:52:37 <brentz> ack manu
15:52:37 <Zakim> manu, you wanted to respond to brent and summarize "the plan" (as it stands today).
15:52:40 <andres> manu: Summary attempt:
15:52:53 <Orie> Ahh, so just because a data integrity proof would use a term, does not actually mean that people use it.
15:52:56 <andres> ... 1. Will do one test suite per specification for parallelization.
15:53:27 <seabass> q+ about test suites lying
15:53:29 <andres> ... 2. Each test suite will do their best to support live APIs, docker images, and proprietary implementations that submit a file saying they pass.
15:53:32 <dlongley> Orie: "the term" is the same whether it's syntactically shortened from a URL or not.
15:53:38 <seabass> q+ to refer to test suites lying
15:54:09 <andres> ... 3. Decide whether the WG wants to pull in the work that has been done.
15:54:29 <brentz> ack seabass
15:54:29 <Zakim> seabass, you wanted to refer to test suites lying
15:54:40 <Orie> Can we please have a repository for testing vc-jose-cose ?
15:54:54 <andres> seabass: I wanted to clarify a bit. closed source is not secret.
15:54:54 <manu> and can we please transfer the repos mentioned during the call to VCWG?
15:55:00 <manu> Can we get a resolution for both today, brent?
15:55:03 <andres> ... We've supported closed source in the past.
15:55:45 <andres> ... Supporting secret is something we've never done, and relying on trust wouldn't help us as a working group.
15:56:03 <andres> ... We can have them submit docker with obfuscated source, for instance.
15:56:19 <andres> brent: <drafting a proposal (in english, not german)>
15:56:47 <ivan> q+
15:56:52 <kristina> kristina has joined #vcwg-special
15:56:53 <kristina> present+
15:56:54 <brentz> ack ivan
15:57:04 <andres> ivan: Does this mean that we'll create one repo per test-suite?
15:57:12 <manu> q+
15:57:32 <brentz> ack manu
15:57:32 <andres> ... May be an overkill. We are already managing so many repos. Do we want to double? It may become unmanageable.
15:58:03 <andres> manu: Ivan, that's what was discussed. If we find it's unmanageable, we can discuss.
15:58:36 <brentz> PROPOSAL: We will move forward with the testing plan as outlined and create the necessary repositories and import the necessary files as mentioned
15:58:39 <brentz> +1
15:58:41 <manu> +1
15:58:41 <dlongley> +1
15:58:41 <Orie> +1
15:58:43 <dmitriz> +1
15:58:44 <ivan> +1
15:58:48 <pl-ASU> +1
15:58:54 <bigbluehat> +1
15:58:55 <cabernet> +1
15:58:58 <JoeAndrieu> +1
15:58:59 <DavidC> +1
15:59:00 <decentralgabe> +1
15:59:08 <hsano> +1
15:59:12 <andres> +0
15:59:15 <oliver> 0
15:59:17 <seabass> +0
15:59:18 <kristina> 0
15:59:36 <brentz> RESOLVED: We will move forward with the testing plan as outlined and create the necessary repositories and import the necessary files as mentioned
16:00:03 <ivan> rrsagent, draft minutes
16:00:04 <RRSAgent> I have made the request to generate https://www.w3.org/2023/08/08-vcwg-special-minutes.html ivan
16:00:16 <cabernet> "detailed"
16:00:20 <brentz> ausfuelich
16:00:43 <ivan> zakim, end meeting
16:00:43 <Zakim> As of this point the attendees have been ivan, brent, hsano, seabass, andres, JoeAndrieu, manu, phil, pl-ASU, dlongley, dmitri, cabernet, oliver, chris, benjamin, orie, DavidC,
16:00:46 <Zakim> ... gabe, decentralgabe, identitywoman, kristina
16:00:46 <Zakim> RRSAgent, please draft minutes
16:00:47 <RRSAgent> I have made the request to generate https://www.w3.org/2023/08/08-vcwg-special-minutes.html Zakim
16:00:49 <Zakim> I am happy to have been of service, ivan; please remember to excuse RRSAgent.  Goodbye
16:00:53 <Zakim> Zakim has left #vcwg-special
16:01:27 <ivan> rrsagent, bye
16:01:42 <ivan> rrsagent, set log public
16:01:50 <ivan> rrsagent, bye
16:01:50 <RRSAgent> I see no action items