<kaz> scribenick: McCool
two topics: plugfest prep and testing
McCool: I suggest we do Fredrico's pres on testing first, so he can leave if he has to
Federico: presentation, then a
little demonstration on what is being done for CoAP
... WoT may be different, but it will give you an idea
... F-Interop is an H2020 project
... almost done, Oct 2018, but tools will be maintained
... main objectives is to develop and provide online testing
tools
... including interoperabilty testing
... members into INRIA, ETSI, etc.
... currently, SoA is F2F events ("plugfests")
... but F2F plugfests have issues...
... short, if you hit a bug, can wreck entire event
... so F-interop wants to provide online and remote
process
... so can do continuous testing
... system description
... runs in cloud, can connect both through Web GUI and to IUT
(Item under test)
... can also have simulated IUTs to connect to running in
cloud
... various tools: communication, including tunneling (bypass
NATs and Firewalls); coordinating tests based on test
descriptions; traffic sniffing; dissecting messages; traffic
analysis
... results -> issue PASS/FAIL/INCONCLUSIVE based on test
description
... there is a tutorial available online that you can look
at
... initially looked at a couple of IoT standards: CoAP and
6TiSCH
... started with ETSI plugfest test descriptions
... now looking at extensions: oneM2M, OMA LwM2M, 6LoWPAN,
and... hopefully WoT
... Ex: CoAP test case (test spec)
... did not make these up, used ETSI test specs
... gives objectives, configuration, pretest condition, then
test sequence
... in this case, test sequence starts with a stimulus, then a
set of checks on responses
... so what are WoT test cases?
... could not find anything similar to CoAP
... would be nice to have things more formal
Ege: looks a bit like invoke
simple action
... but autogenerated from TD
... can generate payload from description
... can send payload and check response
... for example, can check that output satisfies schema
Federico: that makes
sense...
... but not predefining coverage, looking at capabilities of
device, making sure those work
Ege: test description basically is given by the TD
Federico: nice approach, but at
the end of the day doing the same kind of tests that were in
the BJ plugfest
... but a more advanced automation
McCool: time check, only 5-10m more
Federico: ok, I'd like to take
the time to do a demo
... can access for free
Federico: can do both single user
and user-to-user tests
... or can use a reference-based test
... pick device, and test case
... there is also a playground to launch requests without a
predefined test
... start it, deploys instance in Docker containers and runs
tests
... need to set up environment, communication, etc.
... communication can include a virtual network
... including support for UDP
... based on very formal set of test cases
... at the end get a report
scribenick: kaz
McCool: we should clarify which part is opensource and which is not
scribenick: McCool
McCool: also, I think a small team (Ege, Federico, Dave) should go off and put together a small proof-of-concept using just the OSS F-Interop tools and then report back
scribenick: kaz
Kaz: agree we
should clarify which is open and which is not.
... also we should think about levels of testing again, and consider which
level/part is this work (F-Interop) should be applied
... regarding the smaller call, would suggest at least all the co-Chairs and the staff contacts should be included
scribenick: McCool
McCool: I was thinking a working meeting with just the implementors, but let's discuss via email
Koster: we should get in the
habit of publishing an agenda
... so people who are just interested in the pf, for instance,
know when to join
Koster: did prepare one slide as a discussion point
<ryuichi> https://github.com/w3c/wot/blob/master/plugfest/2018-bundang/preparation.md
Matsukura: want to first explain
preparation.md structure
... sections: first is intro, info from last pf
... second section is new information
... 2.1 is a table for participants to fill out
... please insert your information in this table
... 2.2 is checking points
... these should also be discussed in the results.md for your
report-out
... can follow this template and carry information
forward
... then 2.3 are other issues
... if any suggestions to improve please make them
... section 4 are logistical requirements: network, power,
etc.
... I also made an example, fujitsu-preparation.md
... information is the same as last plugfest
Koster: table is useful, people
can just add a row
... checkpoints are also useful for results.md
... but not everyone used the same template
... or maybe it was
... but edited
... section 2.3, other issues... my material
... kind of a mixture of different things; experimental
features, validation, binding templates, services, etc.
... this section will go away/be integrated
McCool: I think we should
prioritize, e.g. security should be mandatory
... also, "other" maybe should be broken into "extras" and
"experimental features". Latter are things that are candidates
for inclusion in the standard but need so experimentation
Sebastian: I think we need fewer
"demos" and more "plugging"
... we really need to check how things can work together
... that would really help to improve the standard
... we need to actually focus on testing interoperability and
the standard
... would like to see more "client" implementations
... a lot of devices, not enough systems trying to use other
devices
Koster: agree. we also need more
formal test plans and better record keeping
... agree also that we need more applications
... we are building a system, we need to have all the
components present
... need so see more interactions
Sebastian: rather than bringing a
lot of things, we should focus on the content and
interactions
... we should concentrate on the functionality
Federico: table of features that
implementations should implement
... at the end of the day, pulling tables to see what features
are being tested
Koster: when you have a set of
test cases, it can stand in for a standard application
... but we also need some ad-hoc applications to test
integration
... and use cases
... for example, using semantic search to find a bunch of
sensors and analyse them for some purpose
... in other words, system-level applications
... first cover functional, then cover system
... functional: list features you want to test, and then
generate tests against them
... then, assuming functional tests done, how do we orchestrate
them to solve a higher-level problem
... we don't really know what "interoperability" is yet
McCool: I think the second part
is more like "what is it good for", eg. use case analysis
... functional testing is just "does the implementation match
the spec"
... system testing is "does the spec have the right things"
scribenick: kaz
Kaz: was on the queue and wanted to mention something like McCool :)
... probably we should have a combined demo scenario on the main preparation.md, and clarify what kind of use cases/features are included in that. And then clarify details in the company-preparation.md and test/report if it works after the plugfest
scribenick: McCool
Koster: so maybe we need to think
about to structure this in the preparation.md file
... would still like to do system-level scenario
exploration
... but should think about how to structure it better
Matsukura: adding a "use case"
section might be useful
... added, please insert your idea
Koster: my presentation... more
of the same
... we are building a system, e.g. an application that
orchestrates multiple things
... but also have components and roles
Koster: to test components need
to have them in context, eg. in a system (application)
... several components are not servients...
... or are they?
... Servients use the scripting API, interacts with Thing
Directory
... if it only exposes a TD, e.g. is an endpoint device, then
it's not a servient
... system architecture includes both local and remote
parts
... I myself am going to bring fewer things but focus on
applications
... but applications have a bunch of common components and
patterns, e.g. local/remote directories, proxies, etc.
... also some new concepts, like "Bridge Servient"; can also
think of as a "role"
... also notable is that proxy-proxy connection doesn't
matter
... but need to go through the proxy to get to local devices
from the cloud
... other way does not necessarily need the proxy
... want to list the patterns separately
... but would like to take an application-layer approach
... register things locally
... tell the proxy I want to make available
... can use the information in the TD to figure out how to
implement
... missing an arrow here, some extra orchestration
McCool: noticed that the
directory and the proxy are paired
... and also notable is that TD may have to be modified in
various ways before being registered in the could TDir
... could be the other way around: could register in the TDir,
then proxy can pick it up
Koster: another point: we want to
think about applications and how to deploy them
... I am thinking about using node-wot to deploy
applications
... what I suggest we start thinking about
... we need goals that are bigger than just turning a light
on
... what are the higher-level problems we are trying to solve,
and what are their requirements?
... for example, what are the security requirements
Sebastian: don't think we need to
spend so much time on the application
... I am thinking more that we just do more 1:1 testing
... and try to use all the interaction patterns
... we could go further and do mash-ups
... even that simple approach exposes many issues
Koster: I was thinking about how
to accomplish what you say...
... can create a control panel in a client that lets you
connect to all the
McCool: could be a simple tool that generates a web dashboard given a TD
Kaz: can clarify use cases,
brief, in preparation.md
... so people can understand the "story"
Sebastian: if you recall in
BJ
... we tried to identified what worked well and what
didn't
... I am a little worried about our timeline
... we only have a couple more plugfests to get experience with
TD in practice
... then we are supposed to be done
... not convinced we are testing things completely enough
... we need to at least do all the basic component tests
scribenick: kaz
Kaz: would agree with your concern. however, each use case doesn't have to be very fancy but can be something simple, and the main preparation use case scenario simply refer to them and clarify concrete demo scenarios. sometimes in parallel and sometimes sequentially using dashboard, etc.
scribenick: McCool
Koster: this is definitely a
different level
... we also need the basic testing at the plugfest
... but... the low-level testing is obviously a higher
priority
... although we also need to move forward with system testing
too
Toru: do we really need scripting API?
McCool: my opinion is that the
minimum is that it "consumes" a TD
... if it only exposes a TD it's an endpoint device
Koster: I do want to motivate
using the TDir as well
... but, yeah...
<Zakim> kaz, you wanted to ask about the availability of Koster's slides :)
Matsukura: our implementation
doesn't use scripting API, so... but we need to better focus on
"applications"
... our implementation doesn't use scripting API, though
Kaz: can you make slides available?
Lagally: suggestion
... suggest to add a legend to the architecture slides, to be able to understand the meaning of the colours of the boxes
Koster: sure
Matsukura-san: AOB?
ok, adjourn