DID WG Topic Call on the Test Suite — Minutes
Date: 2021-04-29
See also the Agenda and the IRC Log
Attendees
Present: Manu Sporny, Brent Zundel, Orie Steele, Shigeya Suzuki, Markus Sabadello, Ted Thibodeau Jr.
Regrets:
Guests:
Chair: Brent Zundel
Scribe(s): Brent Zundel, Manu Sporny
Content:
Brent Zundel: Let’s dive in…
1. DID test suite
Manu Sporny: https://github.com/w3c/did-test-suite/pulls
Manu Sporny: https://github.com/w3c/did-test-suite/pull/70
Manu Sporny: PRs to review, Shigeya’s looks good. I think we should pull it in.
… I still don’t know what the docker image does.
Orie Steele: for folks who want to run the test harness as a web server, then you can send JSON to it.
… if someone wanted to host it, that would allow real-time DID testing
Manu Sporny: is there an example of how to do it?
Orie Steele: there was. I expect it should still work.
Manu Sporny: I don’t expect it to work. I am skeptical that it will work after the restructuring. Is there anything in the readme?
Orie Steele: there was, at one point we hosted a version of it. possible the restructuring broke it.
Manu Sporny: shigeya have you been able to test that this works?
Shigeya Suzuki: it worked as expected
… I agree with Orie
Manu Sporny: https://github.com/w3c/did-test-suite/pull/71
Manu Sporny: other PR is initial JSON test, is buchner here?
… hopefully he got his answers. Can’t merge until tests are done.
Manu Sporny: we don’t have any boolean or null properties, not sure if there are sequence values where order is important.
… it may be that we have some tests for which there aren’t test values
Orie Steele: https://did.actor/mike/
Orie Steele: https://did.actor/mike/did.json
Orie Steele: ^ a did document that uses features nobody else uses.
Manu Sporny: it is possible that two implementations could submit a test with unregistered properties that have these value types to test them
… would there be objections to that?
… Orie has one, we would need one more
… I don’t think this really needs to be tested, even if we leave the normative statements in, but the easiest thing is to get two examples in.
Orie Steele: these features need to be tested or removed.
Manu Sporny: if Ories submits did mike, then daniel buchner can submit something as well. We can point him to this conversation and ask him to submit an implementation that tests them.
… Amy has created a checklist for all normative statements
Manu Sporny: https://github.com/w3c/did-test-suite/issues/72
Manu Sporny: https://github.com/w3c/did-test-suite/issues/73
Manu Sporny: https://github.com/w3c/did-test-suite/issues/74
Manu Sporny: https://github.com/w3c/did-test-suite/issues/75
Manu Sporny: https://github.com/w3c/did-test-suite/issues/76
Manu Sporny: https://github.com/w3c/did-test-suite/issues/77
Manu Sporny: wee need to assign these to people, this will let us be sure all the normative statements are tested.
… who did syntax and parameters?
Markus Sabadello: I did some
Manu Sporny: I will take data model
… production and consumption are all together, so this will be daniel (JSON), me (JSON-LD), and Jonathan (CBOR)
… markus is resolution
Markus Sabadello: yes
Manu Sporny: did methods are all human testable, I’ll assign Drummond
… once we go through the checklists we will know if we have all the tests we need.
… need to be done before our next tuesday call
… any concerns or objections?
Brent Zundel: not hearing any
Manu Sporny: https://github.com/w3c/did-test-suite/issues/51
Manu Sporny: last item, @issue51
… we have 5? implementations
Orie Steele: There’s not enough data to justify investing in visualization
Manu Sporny: I disagree, we have 9 implementations, when is it enough?
Orie Steele: if we have 9 actual separate ones, that would be enough. Last I checked it wasn’t enough.
Manu Sporny: there are 5 methods and 3 resolvers and 1 dereferencer
Orie Steele: at a minimum we need two of each
Manu Sporny: but we still need a report
… we need at least one of each thing. I suggest we have enough. If you disagree, I want to know how much we need.
Orie Steele: there was automated reporting set up. It should be possible, but I am hesitant to do it with only 1 of a thing.
Manu Sporny: the only thing that could change at this point is adding or removing tests, I hope the test generator doesn’t care.
… I don’t think it’s wasted effort to do a report.
Orie Steele: generating the report it trivial, anyone can do it.
… the question is if we should do it now
Manu Sporny: I would prefer that
Orie Steele: at what point is the report visualized enough. we have the automated one, I did an implementation that was better, but it is possibly broken since the refactor
Manu Sporny: that’s the one I mean, I’m fairly sure its broken, could you fix it?
Orie Steele: if tests change, that report breaks
Manu Sporny: that means we’ll have to wait. I was hoping we could get it to a stage where we can see how far off we are.
… without having to manually do it.
… that’s not ideal
Orie Steele: the code is there, anyone can update it.
Manu Sporny: it was hard for me to understand it. Not clear to me how the tallying was happening
… looking at the package.json
Orie Steele: follow the link at the top level readme will lead you to throw to generate the test report.
… it would do well to have another developer look at it
Manu Sporny: then we will need someone to step forward.
Orie Steele: I at least need someone else to work with
Manu Sporny: we’ll put that out to the group. We need that for visibility in how we’re doing
… any other issue to discuss?
Orie Steele: https://github.com/w3c/did-test-suite/tree/main/packages/did-core-test-server
Markus Sabadello: which part of the code is about the reports?
Orie Steele: https://github.com/w3c/did-test-suite/blob/main/packages/did-core-test-server/package.json#L11
Manu Sporny: https://github.com/w3c/did-test-suite/tree/main/packages/did-core-test-server#usage
Orie Steele: https://github.com/w3c/did-test-suite/blob/main/packages/did-core-test-server/generate-latest-respec-data.js
Markus Sabadello: thanks
Orie Steele: https://github.com/w3c/did-test-suite/blob/main/index.html#L686
Orie Steele: ^ rally this is the problem
Orie Steele: report is generated client side from json in respec thats highly brittle
Brent Zundel: I will send an email to the mailing list that points folks to github
Brent Zundel: action brent will send an email to the list