16:38:21 RRSAgent has joined #did-topic 16:38:21 logging to https://www.w3.org/2021/02/25-did-topic-irc 16:38:23 RRSAgent, make logs Public 16:38:24 please title this meeting ("meeting: ..."), ivan 16:38:40 Meeting: DID WG Topic Call on Issue Processing Working Session and Test Suites 16:38:40 Chair: brent 16:38:40 Date: 2021-02-25 16:38:40 Agenda: https://lists.w3.org/Archives/Public/public-did-wg/2021Feb/0016.html 16:50:03 TallTed has joined #did-topic 16:58:29 justin_r has joined #did-topic 17:00:46 present+ 17:01:07 present+ 17:01:11 scribe+ 17:01:14 present+ 17:01:16 present+ 17:01:33 present+ 17:02:01 markus_sabadello has joined #did-topic 17:02:08 present+ 17:02:08 present+ orie 17:02:11 present+ 17:02:29 q+ 17:02:32 q+ 17:02:38 Orie has joined #did-topic 17:02:43 ack manu 17:02:51 present+ 17:03:01 manu: the topic today was to figure out how we're going to test all the sections in excrutiating detail 17:03:06 ... everyone writing tests has their marching orders 17:03:11 ... some sections are easier than others 17:03:16 https://github.com/w3c/did-test-suite/issues?q=is%3Aissue+is%3Aopen+tests+for 17:03:29 ... reminder - there's a link of the tests that need to be written, for volunteers 17:03:32 ... about 127 of them 17:03:51 ... I would like us to decide on exactly what kind of tests we're writing 17:04:04 ... whether they're spec only tests, static tests vector tests, or any live integrations 17:04:13 ... that's meant to be a gradient of possibilities not just three possibilities 17:04:34 ... once we get that settled then we can talk about how each section should be tested, highlighting the ones where we're very concerned about testability 17:04:39 ... depending on the approach we take 17:04:41 q+ to ask about vendor methods vs generic 17:05:00 ack markus_sabadello 17:05:08 markus_sabadello: I thought the topic of this call was to work on the open PRs on did core 17:05:13 ... in which case we could look at the remaining diagrams 17:05:17 ... and some ways of finishing those 17:05:25 ... more than happy to first discuss the test suite, also important 17:05:35 ... when we say there are some sections that are hard to test we probably mean resolution and dereferencing sections, which I signed up for 17:05:43 ... it would be valuable to discuss how to approach those 17:05:45 q+ to note that I think we're close for DID Core diagrams... further away for test suite. 17:05:58 ... and also the sections where we may be able to do something that manu said about dynamic and live testing rather than just static files 17:06:08 brent: we are hoping to have time today for both 17:06:18 ... if you would like we can timebox and spend a certain amount of time and reasess 17:06:23 ack Orie 17:06:23 Orie, you wanted to ask about vendor methods vs generic 17:06:36 Orie: I'd be happy to do the open PRs first, I feel like thats going to be more productive 17:06:44 ... and they're going to be related to knowing whether or not anything is testable 17:06:49 ... those PRs should be addressed first 17:06:58 manu: I'm confused, I thought markus was talk about the diagram PRs, not the test suite PRs 17:07:12 Orie: I mean the diagrams in DID core make it untestable in its current form, and markus' PR addresses that issue 17:07:43 manu: I'd rather not do that, but I'm hearing people wanting to do that... I think we need to come to ground on the test suite. I dont think talking about the diagrams is going to get us there. I would like to timebox to 20 past the our 17:07:45 ack manu 17:07:45 manu, you wanted to note that I think we're close for DID Core diagrams... further away for test suite. 17:07:46 s/our/hour 17:07:48 q+ 17:07:54 ack markus_sabadello 17:08:18 markus_sabadello: there are two open PRs with diagrams that I was working on 17:08:22 ... one is 596 17:08:25 ... on production and consumption 17:08:32 ... one on resolve and resolveRepresentation 17:08:37 ... 597, I have not worked on updating that yet 17:08:43 ... I have tried to update the production and consumption one 17:08:46 ... I came up with this version 17:08:59 ... latest attempt maybe not perfect, but how close is it to what people want to see? 17:09:07 q+ to suggest changes to the diagram 17:09:10 https://github.com/w3c/did-core/pull/596 17:09:16 ... tries to implement this separation of the maps that we're saying we're passing into the production function and getting out of the consumption function 17:09:24 ... and the did document data model and representation specific entries as a separate map 17:09:27 ack manu 17:09:27 manu, you wanted to suggest changes to the diagram 17:09:42 manu: this is not necessarily aligned with where I thought we were going 17:09:53 ... its' confusing because the bottom one looks like i'ts just using the representation specific entries 17:09:58 ... some changes 17:10:07 ... if you take diddocument data model at the top and figure out how to move it into that dashed box 17:10:17 ... and representation specific entries, strike jsonld, and move into the dashed box below 17:10:22 ... and do a larger dashed box around both 17:10:29 ... and have produce and consume go from that would solve the issue for me 17:11:08 markus_sabadello: *live edits diagram* 17:11:19 manu: it does dodge some of the questions, but addresses my concern, if everyone agrees 17:11:23 markus_sabadello: I can live with it 17:11:26 q+ 17:11:32 ack Orie 17:11:33 +1 with these changes 17:11:41 q+ 17:11:43 Orie: what was proposed is consistent with the current diagram that has been merged into did core 17:11:51 ... the upper left hand box with dashed lines is already represented in did core under the data model section 17:12:00 ... i'd be fine if we used the existing diagram and didn't add another diagram that in some ways conflicts 17:12:06 ... the existing diagram we have in did core is acceptable for me 17:12:07 q+ 17:12:09 ... as a result of consumption 17:12:15 q+ to use same diagram if we can 17:12:19 ... i'm struggling why we should show different pictures of the same data model 17:12:27 ... almost there, stop representing the data model differently in different diagrams 17:12:30 While I almost agree to the diagram... +1 to Orie's opinion 17:12:30 ... use the same diagram for each of them 17:12:32 q+ 17:12:33 ack TallTed 17:12:49 TallTed: i'm concerned that surrounding dashed box suggests that those representation specific entries are part of all of the productions 17:12:55 (reuse same diagram on same concept) 17:12:56 ack ivan 17:13:11 ivan: orie, when you talk about the diagram in the spec right now, are you talkin about that one or the change that is part of another PR? 17:13:28 manu: merged few minutes ago 17:13:34 ivan: oh come on. 17:13:38 ack markus_sabadello 17:13:46 manu: I like where orie is going with this, we do have something already, can we reuse that 17:13:50 ... that would be better I think 17:13:52 I mean use this diagram: https://w3c.github.io/did-core/#data-model 17:13:54 :) 17:14:08 ivan: there is a difference, what markus is doing is with code extract which here is not the case 17:14:13 ... reusing the same style and colours would make sense 17:14:16 ... but it's not the same diagram 17:14:20 ... here we list some of the core properties, not all 17:14:21 +1 ot Ivan 17:14:27 s/ot/to 17:14:33 q? 17:14:33 +1 this is not the same diagram, this is how we get TO that diagram 17:14:36 (and from it) 17:14:37 +1 these are different diagrams for different purposes 17:14:38 ack manu 17:14:38 manu, you wanted to use same diagram if we can 17:14:43 q+ 17:14:58 +1 17:15:01 ack markus_sabadello 17:15:08 drummond has joined #did-topic 17:15:09 markus_sabadello: also some similarities and differences 17:15:15 present+ 17:15:22 ... both show the data model, but this one here that ivan made is intended to be more generic and higher level, whereas the other has a concrete example 17:15:28 q+ to ask Orie if he's fine w/ the other diagram (hold your nose) 17:15:40 ... the other difference is that this is showing the concepts of core things and registered extensions and unregistered extensions, which are irrelevant in the other diagram which is about production and consumption 17:15:49 +1 to the diagram we are seeing now on "production and consumption" 17:15:54 ... using similar colours and style, I'd be happy to make an attempt to do that 17:15:58 ack manu 17:15:58 manu, you wanted to ask Orie if he's fine w/ the other diagram (hold your nose) 17:16:04 q+ 17:16:09 present+ drummond 17:16:16 ack Orie 17:16:16 manu: orie, would you be okay with that? I buy the arguement that they're slightly different.. would you be okay that kind of set up? 17:16:25 Orie: we have a resolution that hasn't made it into the spec yet around ignoring and consuming properties 17:16:30 ... somewhat related to what we're talking about here 17:16:47 ... this diagram that's showing concrete productiona nd consumption for three representations 17:16:51 ... aligning the colours is a good idea 17:16:59 ... dangerously to represent the adm abstractly and concretely in slightly different ways 17:17:04 ... an issue of colouring and constency 17:17:19 ... and boxes coloured one way in the abstract sense, should show how properties are preserved during consumption should show up in those boxes 17:17:29 agropper has joined #did-topic 17:17:41 ... when you consume your properties are going into one of many different coloured boxes, and when you rpoduce they're coming out of the coloured boxes and going into one representation 17:17:49 ... I like the image in did core right now, and this one doesn't look like that 17:17:50 present+ 17:17:53 ... the subtle differences are harmful 17:17:57 ... we should use the same labels 17:18:07 q+ to agree with Orie and answer TallTed, maybe 17:18:08 ... representation specific entries, core representations specific entries 17:18:14 ... all of the term labels should be the same 17:18:14 ack manu 17:18:14 manu, you wanted to agree with Orie and answer TallTed, maybe 17:18:18 manu: I agree with what orie is saying in general 17:18:23 ... is markus willing ot give that a shot? 17:18:33 ... Ted raised that it make sit look like you use all those things to produce and consume 17:18:39 ... that's why I was saying there's a bit of a handwave that is okay 17:18:40 ... you can 17:18:48 For the record, I am fine with the "content" of the diagram, just concerned about colors / labels 17:18:51 ... nothing to say that an application can't go into these boxes and modify them between production and consumption 17:18:58 q+ 17:19:06 ... I might consume jsonld, and if I know i'm going to produce json and my application feels strongly about removing @context it has the choice of doing that 17:19:10 ... we don't have anything in the spec that says you can't 17:19:15 ... we suggest that you should preserve everything 17:19:20 ... but apps can process these boxes for their own reasons 17:19:24 ... in consumption we say you have to preserve it 17:19:31 ... but we don't say you ahve to all the way through to production 17:19:35 ... you can modify them between those two 17:19:39 ... thats why we're handwaving a bit over this 17:19:51 ... I think that's okay because we need to get something that we're more or less okay with, otherwise this is going to keep getting blocked 17:19:56 For example, you are not required to preserve prototype pollution or sql injection attacks :) 17:19:58 ack markus_sabadello 17:20:13 markus_sabadello: agree that what we have now would allow both 17:20:19 q+ to time box and move on 17:20:35 ... it would allow having @context int he representations specific entries then producing plain json, and including @context there though personally I don't like it I think it's allowed and possible with the spec language 17:20:49 ... what I could also do is what we discussed before to split this up into multiple diagrams where we show different flows or different ways of using this 17:20:53 ... first consumption, then production 17:20:59 ... sometimes @context is there, sometimes it's not 17:21:03 ... I'd rather have one diagram 17:21:27 brent: feels like we have consensus on direction 17:21:30 ... I'd also rather have one diagram 17:21:42 ack manu 17:21:42 manu, you wanted to time box and move on 17:21:42 Also for the record, when you advocate for producing application/did+json without an @context, you are advocating for incompatiblitty / non-interoperability with JSON-LD verifiable credentials. 17:21:43 q+ to ask the big testing questions 17:21:43 ... opposition to moving on to testing? 17:21:49 ack manu 17:21:49 manu, you wanted to ask the big testing questions 17:22:01 manu: on tuesday I heard people stretching further than I was suggesting 17:22:07 TOPIC: Test suite 17:22:19 manu: as brent mentioned, there are three general classes of things we can shoot for 17:22:23 ... bare minimum make sure spec is testable 17:22:28 ... human testable comments throughout the spec 17:22:37 ... my suggestion is that's the weakest form of making sure the ecosystem si following a spec 17:22:41 ... not a very useful place to get to 17:22:48 ... most wgs shoot higher than that 17:22:52 ... orie has already done that kind of work 17:22:55 ... it's good work and we should pull it in 17:22:58 ... that's the baseline 17:23:03 ... we could say we're done..... 17:23:27 q+ to note that my solution does not actually allow methods to demonstrate conformance 17:23:30 ... but I have heard a number of people complain about the VC test suite and how we didn't do enough testing, and all these people saying theyr'e conformant with the VC test suite when they're not really 17:23:33 +1 17:23:46 ... the vaccination credentials initiatives are an example of that - they're doing things wholly outside of what the WG would cnsider as conforminig implementations 17:23:48 ... we should avoid that 17:23:52 q+ 17:23:58 ... what orie proposed for the test suite, the second higher class of testing,w as good 17:24:04 ... the static test fixtures 17:24:08 ... if we did that would be good 17:24:21 ... and allow implementers to submit their test fixtures and for us to run tests against them to make sure they're within the ballpark 17:24:31 ... machine testability, trying to get all the statements in the spec machine testable, that's what I was targeting 17:24:38 ... the suggestion we have static tests fixtures for everything 17:24:43 ... I believe the spec is in that shape today and we can do that 17:24:56 ... some disagreements between markus and orie about committing certain types of implementation things and we need to get that resolved now 17:25:03 ... if that's all we do I'm happy with stopping there 17:25:14 ... what I heard on tuesday was that a subset of people are like why shouldn't have live testing, or a subset of live testing 17:25:21 ... or that you have to have a live version for it to be testable 17:25:33 ... I think we should have that discussion, but after we decide whether we're trying for test fixtures 17:25:49 q+ to say that stopping now sounds quite nice 17:25:51 ... that's the first decision we need to make, are we going to stop at spec only or try for test fixtures with implementations submitted 17:25:56 ack Orie 17:25:56 Orie, you wanted to note that my solution does not actually allow methods to demonstrate conformance 17:26:04 Orie: it's accurate what manu said 17:26:21 ... markus and i were working on the test suite in the w3c repo, looking at how can you provide evidence ot show your method is conformant with the spec, disagreement on how to do that 17:26:28 ... so I proved that the spec was testable in a separate repo 17:26:34 ... the spec being testable is not helpful to you as avendor 17:26:40 ... it doesn't give you any ability to show up as a green checkmark 17:26:58 ... when you want a list of vendors tha thave implementations that are conformant, they're going ot need to provide their implementations in some format, and the test suite is going to have to process the implementaitons 17:27:05 ... disagreement around how are we going to do that in the did test suite repo 17:27:09 ... we need to have that conversation 17:27:13 ... how do methods show they conform to the spec 17:27:19 ... I heard markus and justin saying it's not a thing we care about 17:27:23 ack markus_sabadello 17:27:36 markus_sabadello: the disagreement we had on some of the work orie contributed 17:27:44 ... we all owe orie a lot for doing that work and taking leadership 17:27:50 ... we wouldn't have anything that looks like a test suite without him 17:27:54 s/avendor/a vendor/ 17:28:04 ... I opposed some of the PRs because my understanding of them is that they would claim to be testing something different to what they're actually testing 17:28:25 ... if what I did was to block a PR that simply added data form one specific vendor, if that's what was going on in the PR, then I misunderstood, and I apologise for that 17:28:28 ... that was not my intention 17:28:37 ... I did thinkt here was a problem with how production and consumption and resolution was designed 17:28:44 ... that was my concern, not the vendor specific choices 17:28:54 ... having said that, going back to resolutiont ests, I think it would be very exciting if we did dynamic tests 17:28:58 ... if we had resolver endpoints for example 17:29:08 ... that can be given a did and can return a did document plus the other result values 17:29:11 ... that would be nice ot do 17:29:20 q+ to proposed a layered approach 17:29:23 ... there may have been some concerns that it may not be possible with the way the resolution spec is currently written but maybe we can disucss that 17:29:36 ... can we do that and do we want to to test the reoslution secitons with live dids and live endpoints? 17:29:36 ack brent 17:29:36 brent, you wanted to say that stopping now sounds quite nice 17:29:38 q+ to speak to "live endpoint testing". 17:30:03 brent: stopping now sounds nice.... with my chair hat on, as long as we have tests that sufficiently allow us to exit CR, anythign beyond that is admirable but we're not going ot let it delay us 17:30:08 ... let's do the things we need to to exit CR first 17:30:15 ... if the test suite folks want to continue expanding, two thumbs up 17:30:15 q? 17:30:18 ack Orie 17:30:18 Orie, you wanted to proposed a layered approach 17:30:21 Orie: +1 to what brent said 17:30:21 +1 to Brent 17:30:24 live endpoint testing can require an adapter to talk to the test suite 17:30:31 ... we should aim for a layerered approach where if we fail to meet the upper layers we're okay 17:30:39 ... a weekend ago I tested the normative statements in the spec without testing any specific method 17:30:49 ... th enext layer up is show that specific methods can conform to the normative statements in the spec 17:30:53 ... we were trying to figure out how to do that 17:31:04 ... I don't hold any grudges about markus blockign the way I was trying to test that... it is hard ot figure out how to test that 17:31:18 ... we don't have perfect answhers to show how an implementation of a method is conformat 17:31:20 ... but we need to get to that 17:31:26 ... to show what is conformant, which methods are equivalent 17:31:27 q+ to ask whether that is in our scope? 17:31:29 ... we need a chart to show it 17:31:35 q- later 17:31:42 ... the next layer on top of that is the dynamic testing, maybe wiring into universal resolvers or vendor specific resolvers, I'd love to get there 17:31:52 ... I think we can get there, but not if we get through the intermediate step 17:31:56 ... suggest a three phased approach 17:32:01 ... phase 1 is that normative statements are testable 17:32:13 ... phase 2 is that vendors can implement and show conformance ot the spec - we've been working on that, no great solution 17:32:21 ... phase 3 is live realtime testing with a resolver or something like that 17:32:31 ... markus and I have worked on various versions of that in other work prevously, I believe we can get to that 17:32:36 ... encourage us to shoot for all 3 phases in tha torder 17:32:41 ack drummond 17:32:41 drummond, you wanted to ask whether that is in our scope? 17:32:51 drummond: I love the idea of a full interop test suite for did methods 17:32:58 ... but is that in our scope? is that realistic? 17:33:08 ... vs something that's just a test suite that tests the spec? they seem like two different objectives 17:33:13 ... seems like more work than we signe dup to take on 17:33:17 ack manu 17:33:17 manu, you wanted to speak to "live endpoint testing". 17:33:24 manu: I think we're in agreement 17:33:28 ... I liek the 3 phase approach 17:33:40 ... my assumption was to say to exit CR all we had to do was phase 1 17:34:05 manu: I don't think we need to resolve the first one, it's to do the minimum necessary to get to CR 17:34:11 ... the second is the static test suite 17:34:23 ... the third phase is the live testing 17:34:46 manu: I don't want to put the live testing between us and getting to PR 17:34:59 ... the harder these things are, the less likely it is that things stay in the spec 17:35:06 ... there are combinations here that I'm concerned about 17:35:29 ... eg. if we hit phase 2 on everything except resolution, and resolution is this phase 1 thing I personally have a problem with that because it's going to result in a week part 17:35:41 ... but the group is going to vote for the minimum bar, which is fairly useless from a vendor perspective 17:35:59 ... the other issue i've seen is if we have guaranteed success the second we go to cr there's no motivation to work on anything 17:36:11 ... there's a motivation issue created by making the bar low 17:36:23 ... all that said.. skip the first proposal and pick up the second? 17:36:31 drummond: first one is redundant 17:37:05 manu: anyone objecting to static tests to exit CR? 17:37:09 proposal two is "submit serializations of did documents in conformant representations" not just "did documents" 17:37:31 brent: I'd prefer to have a more minimal bar set, but I believe we'll achieve it 17:37:39 ... minimal bar with stretch goals 17:37:49 q+ 17:37:59 manu: it evaporates the motiation to do more... is my concern 17:38:04 ack Orie 17:38:08 Orie: I would see the stretch goal as live testing with resolvers 17:38:27 ... I don't see static test fixtures a stretch goal, that's basica bility to demonstrate your method conforms 17:38:31 ... it's important to meet that bar 17:38:33 +1 Orie 17:38:40 +1 to Orie 17:38:52 manu: no objections to making the minimum bar testing serializations? 17:39:00 q+ 17:39:02 +1 to Orie 17:39:04 ... we will enable implementers to submit did docs to test against the applicable normative statements? 17:39:09 ... that's the minimum bar to exit CR 17:39:14 ack Orie 17:39:19 Orie: we can't use the term did document any more. It has to be all of the data models in the spec 17:39:35 ... it's hard to write tests around an adm and part of the problem of the resolution sections is its hard to imagine how you'd test them statically 17:39:48 +1 17:39:48 ... the short answer is we can write tests that will work for one way, and note that you could do it another way 17:39:51 q+ 17:40:03 ... it can't just be the did doc data model, it has to be 'the data model' and cover metadata and all the other data structures 17:40:13 ... the fixtures might not be the only way that stuff gets tested or used 17:40:26 ... if we're to make those sections testable we're goign to have to cover the data model and production and consumption, concretely, to meet that phase 2 bar 17:40:36 manu: +1 to that 17:40:36 ack markus_sabadello 17:41:02 q+ to speak to testability of resolution. 17:41:13 markus_sabadello: I thought we made the abstract parts of the resolutn sections testable by adding statements saying when one of thes eabstract data structures is returned here is a way to serialize it 17:41:24 ... we added a sentence that says you can turn it into a concrete representaton and test that 17:41:33 ... for metadata you can represent the as json using the standard infra section that defines that 17:41:52 ack manu 17:41:52 manu, you wanted to speak to testability of resolution. 17:41:54 ... is it solved? we can't test the abstract hings directly, but we can turn them into concrete things and thest those, therefore implicitly testing the abstract part 17:42:03 manu: exactly, that's what we were shooting for 17:42:06 ... I believe this is achievable 17:42:12 ... we should talk about it because the specifics are important 17:42:21 present+ 17:42:23 ... Yes. We have a path to testing all of the data structures in DID core 17:42:27 ... by serializing them to something 17:42:36 ... that goes for all the consumption/productin tests, resolution, core properties 17:42:39 ... we can do all of those 17:42:44 ... the phase 2 thing is achievable, just details 17:42:49 ... rewrote the proposal 17:43:09 ... any changes? 17:43:23 brent: I'd get rid of "all of" and say "for the data structures" because not every submission needs all 17:43:37 PROPOSAL: The DID Core Test Suite will, at a minimum, enable implementers to submit at least one set of serializations for the data structures defined in DID Core to be tested against the applicable normative statements in the specification. 17:43:38 +1 17:43:40 +1 17:43:43 +1 17:43:44 +1 17:43:44 +1 17:43:48 +1 17:43:49 +1 17:43:50 +1 17:44:11 +1 17:44:18 RESOLVED: The DID Core Test Suite will, at a minimum, enable implementers to submit at least one set of serializations for the data structures defined in DID Core to be tested against the applicable normative statements in the specification. 17:44:22 q+ 17:44:28 ack manu 17:44:48 manu: should we talk about how every section and how we test it with this proposal, and treat phase 3 as a stretch goal 17:44:54 ... everyone agrees it'd be awesome to hook it up to live systems 17:45:04 ... let's prove that what we resolved is testable and how 17:45:24 https://github.com/w3c/did-test-suite/issues?q=is%3Aissue+is%3Aopen+tests+for 17:45:50 manu: I'm going to assert some are easy 17:46:05 q+ to describe the fixtures 17:46:15 ... core rpoperties is easy. the implementer presents a did doc and the test suite runs a bunchof tests. does the id conform to the abnf? is a verification method right? we can use json schema to test a lot of that stuff 17:46:21 ... production into json, jsonld and cbor is easy 17:46:30 ... they're going to provide a document and tell us what the media type is 17:46:34 q+ 17:46:43 ... the test suite is going to pull that in, use a standard read rules to put it into a data structure and do all the tests 17:46:47 ... resolution metadata, easy 17:46:50 ... just a json structure 17:46:57 ... take it as a fixture input and check it 17:47:01 ... did parameters, easy, just a string 17:47:06 ... same thing for did syntax and url syntax 17:47:19 ... most difficult is consumption portion, and some resolution and dereferencing 17:47:30 ... for consumption the only thing we need to do is two people need to write separate implementations on consumers 17:47:35 ... eg. things that say a consumer must throw an error 17:47:36 q+ 17:47:41 ... i don't know how we're going to do that 17:47:59 ... for resolution and dereferencing I'm suggesting we provide inputs to the function and outputs from the function and we test those for conformance 17:48:12 ack Orie 17:48:12 Orie, you wanted to describe the fixtures 17:48:13 Orie: good approach 17:48:24 ... as markus mentioned, a difference between serializations of the adm and serializations of a representation 17:48:29 ... what's confusing is that the fixtures are json 17:48:44 ... There's going to be json that is used to represent the ADM that isn't a JSON DID Doc 17:48:48 ... and we'll have to make that very clear 17:48:52 +1 to that 17:49:09 ... I don't know how we're going to test those consumption rules outside of providing a standard way of reporting that a consumer would throw an error 17:49:19 ... I think we have some of those kinds of errors in the spec, invalid did and did not found 17:49:40 ... testing those from static fixtures, you have some concrete function that is processing expected inputs and outsputs, and just checking for string equivalents 17:49:45 ... which is why it's better to have live tests 17:49:49 ack markus_sabadello 17:49:52 markus_sabadello: agree with Orie 17:50:03 ... about confusion that we will probably have since we are going to deal with the adm but will use json to express that 17:50:09 ... at the same time json is one of the supported representaitons 17:50:16 ... a consequences of the fact wer'e doing the test suite in javascript 17:50:31 ... like Orie said as long s we name that in some smart way confusion can be minimised 17:50:48 ... agree with everything manu said on how to test different sections wrt resolution and dereferencing I have some idea 17:50:54 ... having inputs and outputs and running tests against those 17:51:14 ... I'd add to what manu said, about testing production we'd want to test the inputs and outputs since we're now saying production rules take two maps plus options 17:51:20 ... in the test sweet we watn to model that 17:51:31 ... here are the contents of those inputs of the production function and here's the expected output representation 17:51:32 +1 to what markus is saying 17:51:32 +1 to markus_sabadello 17:51:35 ack justin_r 17:51:49 justin_r: we need to be really careful with being clear to ourselves and to the community about what we are testing 17:51:52 q+ to ask who is not here 17:52:01 ... when we're testing production what we're saying is I will give you an ADM and I want to see how you serialize that to make sure it's okay 17:52:05 ... to test your output I'm going to parse that 17:52:12 ... to consume it and makes sure it maps to the model I suspect 17:52:27 ... I cannot test you're not doing some sort of weird non conformant magic trickery in order to generate that serialization up to and including just spitting out a static file 17:52:38 ... but I can set a test harness such that itt sets expectations 17:52:46 ... I have a description of the model, you tell me the serialization and I'll handle it 17:52:50 ... the inverse applies to consumption 17:53:10 ... I give you a serialized document to consume and I expect you to come up with this model. Error cases are harder to detect because the end result is an internal state of the system under test 17:53:18 ... in order to test those a lot of the results are going to be asserted by the person running the test 17:53:23 ... we ran into this with openid testing 17:53:32 yep, +1 to justin_r... its hard to test error conditions that are not externalized. 17:53:33 ... al oto f error conditions in that protocol don't end up as messages within the protocol, but screens shown to a user 17:53:47 ... so in that case we had a way to insert a screenshot or a webpage capture as part of your test results to say i can prove this is what happened 17:53:49 ... not saying we do that here 17:53:50 q+ to ask Orie if we have enough to make progress on the test suite... to start getting vendor implementations in. 17:54:05 ... but there is precedent for having a useful set of tests to allow developers to say this is the stuff I need to deal with 17:54:20 brent: who is not here that this information is vital for, besides wayne? 17:54:22 q+ to not required changes needed to the test suite 17:54:29 manu: dan buchner 17:54:34 brent: I will ping each of them with the minutes 17:54:35 ack brent 17:54:35 brent, you wanted to ask who is not here 17:54:42 ack manu 17:54:42 manu, you wanted to ask Orie if we have enough to make progress on the test suite... to start getting vendor implementations in. 17:54:48 manu: I'm hearing a lot of alignment which is great 17:54:55 ... hearing a fairly clear implementation path forward 17:55:01 ... want to see if anyone disagrees? 17:55:28 ... if we submit a vendor implementation of did:key and I specifically write these core property tests against that implmementation I would expect there nt to be a lot of disagreement on those things existing in the test suite 17:55:31 ... anyone disagree? 17:55:44 Agree with Manu seems like we're all aligned 17:55:45 ack Orie 17:55:45 Orie, you wanted to not required changes needed to the test suite 17:55:53 https://github.com/w3c/did-test-suite/issues/32 17:55:53 Orie: I opened issue 32 17:56:21 present+ justin_r 17:56:21 ... the main problem that we will have without some guidence from someoen willing to do some initial work is for somebody to a PR similar to oen markus objected to previosly that has support for both concrete and abstract data model fixtures 17:56:31 ... we need an exampe of a vendor submitting both abstract and concrete 17:56:35 q+ to note "implementer" not "vendor" :P 17:56:37 ... the tests are written over expected data structures 17:56:43 ... you can't write tests unless you have some expected data structures 17:56:57 .. that's the way the test suite is written today, it'll be hard tow rite tests untl we have that contribution 17:56:57 ack manu 17:56:57 manu, you wanted to note "implementer" not "vendor" :P 17:57:13 manu: I don't think we had... I think we start with testing concrete things, if we get around to the abstract things great 17:57:19 ... all of thenormative statements in the spec are around concrete things 17:57:25 ... nothing abstract, modulu did method stuff 17:57:36 ... if we focus on concrete representations and testing those first, we shouldn't block on abstract stuff 17:57:45 ... Also: implementor not vendor. Vendors may hav emultiple implementations 17:57:53 ... you can include the vendor's name in the implementation 17:58:08 ... but using vendor language causes people to get upset about it being vendor focussed vs implementaton focussed 17:58:23 brent: thanks everyone for coming 17:58:28 ... incredibly productive 17:58:41 ... I'll reach out to wayne and dbuc with the minutes so they can see what they signed up for 17:59:06 rrsagent, draft minutes 17:59:06 I have made the request to generate https://www.w3.org/2021/02/25-did-topic-minutes.html ivan 18:00:09 zakim, end meeting 18:00:09 As of this point the attendees have been brent, rhiaro, TallTed, shigeya, manu, ivan, orie, markus_sabadello, drummond, agropper, justin_r 18:00:11 RRSAgent, please draft minutes 18:00:11 I have made the request to generate https://www.w3.org/2021/02/25-did-topic-minutes.html Zakim 18:00:14 I am happy to have been of service, ivan; please remember to excuse RRSAgent. Goodbye 18:00:17 rrsagent, bye 18:00:17 I see no action items 18:00:18 Zakim has left #did-topic