<scribe> scribenick: kaz
McCool: my OCF device is still
down
... but want to talk about speech device over google
hangout
... would work on OCF devices as well
... merged Oracle's update
... now 6 vendors
draft implementation report (local file on McCool's PC is newer than this online version)
McCool: consistent terminology
... generator or producer
... this (Node-WoT with Oracle Binding) is interesting
<McCool> mlagally, are you on mute? we can't hear you
<McCool> https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/report.html
McCool: wanted characterize all the
implementations
... consumer/producer
... this (Device Model Converter) is both consumer/producer
Lagally: yes
... Oracle Digital Twin Simulator exposes TDs
McCool: goes through Oracle's
implementations
... btw, this "Oracle Asset Monitoring" doesn't produce or
consume TDs, do it?
... a couple of things to restructure the interoperability
table
Lagally: it's not a simulator
... integration via node-wot
... connecting TDs to devices
McCool: we should be clear about whether double counting or not
Lagally: count based on code-base?
McCool: we count systems with mostly independent code bases
Lagally: to be clear, code base of the
asset management is significantly bigger
... so would say it's a separate implementation
McCool: also we should be clear which
each implementation is, producer or consumer
... moving on
Lagally: McCool, please create a PR with the proposed changes to the Oracle input
McCool: unfortunately, Ege is not here
... but used my time for playground as well
<McCool> https://github.com/w3c/wot-thing-description/issues/321
McCool: issue 321
... problem with assertion stated
... TD arrays
... required and enum
... created an issue for TD
... also
... another problem
... issue 322
<McCool> https://github.com/w3c/wot-thing-description/issues/322
McCool: readOnly and writeOnly
... are optional
... but for JSON Schema, they're required
... if you look at the testfest-1 TD version
... writeOnly is mandatory
... this version has writable and writeOnly
... so there are mistakes
... we have a decision to make
... modify the schema to be up-to-date with the testfest-1
tagged version?
Lagally: would advocate fixing the schema
McCool: would agree
... encourage people to update the schema
... would take an action to fix it
<McCool> https://github.com/mmccool/thingweb-playground/blob/dec-2018-testfest/WebContent/td-schema.json
McCool: remove writable and make readOnly/writeOnly not required
Lagally: one more thing
... on the assertions
... version in Ege's assertion is mandatory
McCool: there are bunch of versions
... optional in the tagged version TD
... that's good
... the Assertions
... we have a problem
... 3 outcomes
... pass/fail/not-implemented
... but how to handle those 3 options?
... version is good example
... property object
... there is another properties tag here
... type of the members properties and items MUST be serialized
as a JSON bject
... exist and has to have property
... the problem is
... in my test cases
... problem with camera
... bunch of properties
... simple integer type
... for brightness
... actually has sub property but not implemented
<McCool> https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/SimpleWebCamera.jsonld
McCool: one of my properties
... updated schema
<McCool> https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/BetterWebCamera.jsonld
McCool: we now have security
definition
... list of values
... and writable is wrong
... to be fixed
... did in contrast
... mixture of properties
... some of them have types
... JSON rule says property expressed as object is
required
... only some of the properties are expressed
... properties sub field
... this TD should be a PASS
... but it turns to be fail or not-impl
... was scratching my head...
... work around is
... (dec-2018... branch)
... TD validator for JSON Schema
... special option called $comment
... callback function with this value
... particular rule match
... for schema
... I can write a rule
... if it matches and valid, it's pass
... if it matches but invalid, it's fail
... good news is can tell how to fix
... bad news is bunch of things to do
<McCool> https://github.com/epoberezkin/ajv
<McCool> https://github.com/epoberezkin/ajv#options
Kaz: sorry but have started to wonder if it would make sense to try manual test first, and then this automatic testing mechanism as the second stage
McCool: kind of agree
... but
... we should have some automatic tooling mechanism for the
next testfest
Kaz: yeah
McCool: technically each implementer
can provide implementation report
... manually creating the CSV input is one possible
option
... do we have any gaps?
Kaz: you've already found one
possible solution, so we can try that at some point
... but maybe we should start manual check first
McCool: yeah
... can try to solve the problem by Friday
... worth while trying both approaches
... manually filling the CSV files and tooling
McCool: basically I need a CSV file
for each implementation
... that's kind of homework for people
Lagally: wondering about a couple of
TDs by others
... do we expect the other companies' TDs as well?
McCool: Siemens is expected
... Hitachi does TD consumer only
... don't know about Dave's implementation
... my main concern is
... we may not have gaps
... do we have 2 implementations for security features?
... might show up down the vocabulary
... may see 0 here
Lagally: what is the implication to have only one implementation?
Kaz: if we can get only one
implementation for some features, we need to drop the features
... and we have to republish a Candidate Rec after dropping them
... but we can identify some of the difficult features as "features at risk" when we transtion to CR
... and we can drop those "features at risk" and transition to PR without going back to CR
Lagally: how are optional features treated? What happens if they are not implemented at all?
Kaz: the trend is "even optional features should have 2 or more implementations"
McCool: also wondering about default values
Kaz: btw, the 2nd assertion "property, action, event MUST have..." should be split into 3 separate assertions, e.g., "property MUST have...", "action MUST have..." and "event MUST have..."
McCool: also thinking about that
point
... using child assertions
... can create some extra assertions
... and consolidate all the child assertions
Lagally: why don't we define the combined assertions separately in the spec and generate the tables
McCool: also can do so
Lagally: that might be easier
Kaz: that said, the more
important at the moment is checking the coverage of
features
... so that we can tell which features are possibly at risk
McCool: right
... at the moment, people should manually generate the CSV
reports
Lagally: the CSV format would work even after the TD spec is updated?
McCool: as long as the feature ID
doesn't change
... we need example and counter example as well
... these assertions should not change much
... btw, I went through the test spec at the appendix as
well
... need to go through all the details and see counter
examples
... parent vs sub assertions
... the parent assertion is true if all the child assertions
are true
... need a bit more work
... a few assertions are still vague
... confusing assertions to be away
Kaz: we should ask the others, Koster, Sebastian, Toumura-san, etc., to provide their TDs and reports
McCool: recap would be better
Koster: would like to recap
Sebastian: yeah
... also would like to talk about the f2f in Princeton
McCool: date of f2f and IG Charter as
well
... Kaz and I had discussion
<McCool> https://w3c.github.io/wot/charters/wot-ig-2019.html
McCool: we should send out a Call for
Consensus message to the whole IG for the updated draft IG
Charter
... we should wrap up the discussion by Friday
... regarding the f2f dates https://doodle.com/poll/5d4gbht3wm9v66dy
... result here
... 2 weeks
... 2 opposing conflicts
... Dave's conflict is he'll miss Thursday
... not sure about Taki
... or Sebastian
Sebastian: definitely OK
... just in favor with later meeting
... prefer latter one
McCool: yeah
... maybe Matthias also has difficulty with the earlier
option
... more preparation would be better
... Taki, what is your trouble?
Taki: north carolina for IIC
McCool: Taki can't make the latter
option at all
... Dave just has conflict with Thursday
Lagally: what about Matthias?
McCool: in the worst case, he can't
make either of the options
... given Taki's situation, my reincarnation is the first
option
Sebastian: let's make the decision by the end of this week
<McCool> 28 Jan - 2 Feb 2019
McCool: ok
... we'll finalize the date by the end of this week
... should send a message for CfC out
... should nail down by Friday
McCool: recap
Koster: most of the test cases are validating TD
<McCool> https://github.com/w3c/wot/blob/master/testing/criteria.md
McCool: either TD producer or a TD
consumer
... you have a device on the network and have a TD which
describes the device
... automatically generated TD or manually generated TD
... proxies take a TD and possibly change it
... somewhere here (in the report.html)
... we have to count distinct implementations
... and what is "distinct implementations"?
<McCool> https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing
McCool: e.g., even though there are
three implementation by Intel for security features, they're
based on the same code-base (node-wot), so should be counted as
one implementation
... we have 6 organizations at the implementations/
subdirectory
... would have the other implementers as well
... (and then shows optional interop/ area)
... arcs within the connection diagram should be described
here
... producer/consumer pairs
Kaz: yeah, so at some point, we
should split that interop part
... and create a separate WG Note
McCool: yeah, maybe an
interoperability Note
... (and then shows the implementation result CSV files)
https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results
Kaz summrizes what we should do:
1. submit TDs at https://github.com/w3c/wot/tree/master/testfest/2018-12-online/TDs
2. submit CSV-based reports: https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results
3. implementation descriptin: https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/implementations
McCool: that is kind of described
within the manual at:
https://github.com/w3c/wot/tree/master/testfest/2018-12-online
... but the automatic tool is not available yet
Koster: ok
... the CSV file is the result of assertions
McCool: (updates the testfest manual)
https://github.com/w3c/wot/tree/master/testfest/2018-12-online
McCool: not-impl means no
example
... null will be ignored
Koster: question about Ege's
tool
... using npm
... AssertionTester
McCool: there are 2 tools
... playground validator
... td-schema.json is almost updated for the testfest-1 TD
version
<McCool> https://github.com/mmccool/thingweb-playground/tree/dec-2018-testfest/WebContent
McCool: we need to get it
updated
... have a PR
... Ege also has a PR
... waiting for those PRs to be merged
... playground has web interface and command interface
... and then AssertionTester
... ajv schema validator
... some of my assertions here
... (McCool revisits his assertions)
... (and explains the issue with handling
pass/fail/not-impl)
Koster: TD schema with ajv hooks
Sebastian: need to leave
... wondering about the goal of the testfest this week
Kaz: thinks the goal of this first testfest effort is simply looking the implementability of each feature, i.e., can get one or more implementation for each feature (rather than 2 or more)
McCool: would add some more extra
assertions
... if you would like to split some of them, we can do so
... this is a snapshop as of today
... if node-wot is to be updated, that's fine
Sebastian: another question about TD
slot on Friday
... will we have a regular TD meeting?
McCool: other than a few issues, you can use the rest for the TD discussion
Sebastian: we have to make decision about some of the TD issues
Kaz: using the first 30m for
testfest wrap-up
... and using the remaining 1.5h for TD discussion
... is that ok?
Sebastian: ok
Toru: going back to the interop
test
... pass/fail/secure, etc.
... is it possible to handle partial result?
McCool: (visits interop/ area)
... right now we have producer, consumer, security,
status
... you can mention two rows and merge them
... hundred passes and one failure would be failure
... purpose of the summary table
... maybe more information to be captured
... you can add more columns
... my tool will ignore the information
... e.g., you can add columns like cause, comment, etc.
... some failure may be caused by "authentication invalid" with
"expired certificate"
... we can keep track those columns
Toru: maybe just one additional comment column would work
McCool: ok
... probably will become an interoperability document
Lagally: one suggestion
... overview document
... private repos
... very confusing
... and Ege not on the call
(Ege just joined)
McCool: Ege, have you checked my
PR?
... Ege's PR is the main one
... and I also have a PR
Ege: now I'm free to work on that
McCool: discovered bunch of
issues
... fundamental issue
Kaz: would suggest we resolve that issue via the PR/Issue since we've already discussed during this call twice :)
Lagally: which of the repos to have the latest tools, schemas, etc.?
McCool: here is the suggestion
... nothing is in a good shape at the moment...
... we need some more time to stabilize the situation
... (summarize the issue with schema)
Ege: updated the schema Monday evening
McCool: Ege will update the JSON
Schema
... will also add much bashing
... within the next two days
Lagally: thingweb resources also on ege's private repo
Ege: will merge it but still need to check
https://github.com/w3c/wot/tree/master/testfest/2018-12-online
Kaz: so we'd like to use only the
wot official repo for our testfest (if possible)
... note that TD files and CSV reports are already to be copied
under w3c/wot repo
... and so I was wondering about the implementation description
HTML
McCool: that can be also copied under the official w3c repo
<McCool> https://github.com/w3c/wot/blob/master/testfest/2018-12-online/implementations/README.md
McCool: people can install their
implementation description HTML as well to the above official
area
... and please submit your generated CSV report under:
https://github.com/w3c/wot/tree/master/testfest/2018-12-online/results
... each organization should create a subdirectory there
... and copy your generated CSV files to that subdirectory
Kaz: is it OK to start with this manual approach on the w3c/wot repo?
Lagally: ok
... so we're encouraged to copy the implementation description
HTML as well to the w3c/wot repo
McCool: yes
... can copy a template for your help
... and I'll copy all the resources to my mmccool repo to
regenerate the report.html
Lagally: ok
Taki: particular assertion, e.g.,
property, action and event
... not sure we really should split it
McCool: yeah
... can have one combined assertion as a parent with several
child assertions
... can just generate them based on the current TD
Taki: can also have some discussion on Friday
McCool: yeah
Ege: working on the schema fix
McCool: taki, maybe you can merge Ege's PR
<McCool> https://github.com/w3c/wot-thing-description/pull/323
Taki: issue on the JSON Schema
McCool: yeah
... writable
... ah, even older which just include writable
... another PR
https://github.com/w3c/wot-thing-description/pull/324
Taki: can merge it
McCool: anything else for today?
(none)
McCool: please work on your CSV
files
... will continue to work on the automation
... Ege, we need further collaboration
[the official testfest call adjourned]