See also: IRC log
<urata_access> https://www.w3.org/auto/wg/wiki/20170316_TestFramework_Teleco
Urata reviews agenda
ted: I reviewed this after last
call. it is a very nice presentation, much of it is from W3C's
Process Document
... we should follow W3C's Process Document and not create a
new process
Hira reviews slidedeck for call
[there are great examples in slides from W3C MMI activity http://www.w3.org/2002/mmi/]
[slide 5 Implementation report objectives]
Peter: is this implementation report we as a group produce or something the different implementers?
Ted: ideally the implementers
since they are more familiar with their own
implementation
... if they do not and are not part of the group, we can. we
don't have to write reports for all implementations
Hira: w3c requires two implementations as proof of interoperability
Ted: to enter CR we need a plan for creating implementation reports and complete them during CR phase
[slide 6]
Process Document on Implementation experiences (report)
[slide 7]
Urata asks about VIAS entering CR
[discussion of loss of editor, needs wider group review and be published as FPWD]
[peer pressure by group on Urata to step up. best would be to find two editors]
Urata: I will consider and discuss with my supervisor
Wonsuk: would we need browser implementations for VIAS as that would be a problem?
Ted: we need two implementations, they do not have to be in a browser
Hira: (from slide 7) signal generator out of scope
Ted: agree but people will need a source, record and replay, generator or actual vehicle
Peter: we use Open DS
<urata_access> https://www.opends.eu/
Hira: we have a list of 30-40
assertions that can be tested and store them in a
spreadsheet (see archived copy)
... please review the list yourselves
Urata: I am difficulty
implementing some tests
... authorization system is not clear
... access control state is implementation dependent concept
and not defined clearly in our spec
... it is hard to realize in a prototype and create test for
it
... a mock security provider can be used with predefined fake
tokens
... this is one possible way I am considering
... I would like to hear from others
Ted: I think it is a workable
idea and would suggest instead of 'aaaaaa' 'ddddd' using
strings that explain the token type
... eg 'this_is_full_access_token' 'this_is_invalid_token' and
'this_is_limited_access_token'
Hira: what issues remain?
Urata: I will report in github issues
[slide 12]
Hira: what do you think of the diagram?
Peter: that architecture is
similar but we will use real signals instead of a
generator
... we will not have a full tree but will be limited by what
the vehicle makes available
... we will be able to test with different gets/sets
Urata: a signal generator can create all signals. they might not be as realistic as from a production vehicle but still useful
Hira: how does your generator connect to VISS server?
Urata: I used web sockets
[slide 14]
Ted: I am confused by the diagram
in slide 14. We are not doing anything with Geolocation, not
sure what is meant by Web Storage, we do not do HTTP at present
so wonder why there are multiple HTTP servers
... we need to test web socket. that can but doesn't have to be
done from a browser using a test harness
Hira: Web storage and geolocation not needed
Urata: I created this picture to explain
[Urata broadcasting in WebEx editing diagram, clarifying that the tests will be served up by http]
[html ui for testing. loads JS harness, tests and then makes ws connection for json]
Urata: Geolocation was an example of mixing in other API
Wonsuk: for the signal generator, can we specify our own data for testing?
Urata: that is a problem. I have to use specific signals, engine speed for example, in my tests
Wonsuk: we need to test for different data types, integer, string... their size limits
Urata: (highlighting a specific test in spreadsheet) we also set min/max values
Ted: what might be useful since
people will have different data sets as mentioned is to have a
generic test for checking a signal and then set what they want
to test in configuration file
... which signal, expected type and range of legitimate
values
... also tokens can be configuration parameters
... it would be good to have defaults - signals to expect and
test tokens
Urata: that is one possible way
Ted: this way your test suite can be used to check production environments with what signals the underlying vehilce provides (and does not allow access)
<hira> Sorry to step away for a second, and will be back soon.
Wonsuk: client test cases should
start with getVSS and based on the response the client knows
what sort of signals it can test
... then you can have dynamic tests
Ted: +1 that would be very useful but might be more complicated to implement
Urata: that would be very good
but not sure I have enough time
... if getVSS method fails then it fails the whole test
... configuration based is reasonable to do
Peter: agree you want to be able to specify signals to test and would ideally want to run getVSS and then test each signal
Ted: if that is too complicated
then individual tester could start with getVSS and use the
response and send through a configuration generator so a
subsequent run will test all those signals
... also remember VSS can be extended with custom signals and
it would be nice for people to be able to test those as
well
<hira> https://www.w3.org/auto/wg/wiki/20170316_TestFramework_Teleco
<urata_access> Test suite repository
Urata: wrote support status in
README of github. get, set, subscribe,
getVSS all supported
... not supported is filter and wildcards
... wss (secure web sockets) is not supported
Ted: problem is the certificates right?
Urata: yes
Ted: in many programming languages you can tell either in the connection method or as part of the platform configuration you can tell your code not to validate certificates
Peter: self signing is much
easier
... it is useful for us to test secure sockets
... you could also allow tester to upload/provide a cert and
store under /etc/ssl (or better somewhere under /tmp/ssl and
have your OS set to check there before trying to validate
online) but that is more complicated. easiest is if your code
can simply not check cert
Urata presenting html UI running in his browser. it is temporarily available on an IP URI shared with the group in IRC but will not be in minutes
Urata: through the UI I can get
or subscribe to just about anything
... I need to revise the README so others can install
<urata_access> Test Framework for VISS
https://github.com/w3c/automotive/viss-test-suite
Urata: do I need to have test suite in W3C repo, when, process for review
Ted: it doesn't have to be but
agree better. please do initial import and that way people can
review and handle merge requests for new tests
... it is clearer others can use and contribute by having it in
w3c repo. you can continue with your fork as well
Urata: ok to do in April?
Ted: yes, when you're ready
Urata: what should we do with VIAS test suite?
Ted: as you say, FPWD after we
get editors
... we are far off on our timeline. also we have been reaching
out and trying to engage PSA+IBM, Visteon (who applied but
hasn't joined yet) and GM who all have competing efforts to
VIAS that are all further along
... we should continue with VIAS like VISS because we cannot be
sure of their participation. VISS is looking good for schedule,
VIAS not
<urata_access> Test cases for VISS
Wonsuk: when do you expect you
can make this available for use?
... I have a need to use these myself
Urata: you can clone the repo and
use it now
... you will need to install on a linux machine
[in WebEx chat Mike indicated he can contribute to JS & Node, can contribute to testing and Junichi tries to recruit him as second VIAS editor. junichi-hashimoto++ ! ]
Hira: when do you expect the remaining test cases you indicated to be available?
Urata: hopefully by next week
Ted: that would be great but
thinking improving README so others can run and write tests
might be higher priority
... people may think of additional useful tests
Peter: we're interested
Wonsuk: browser may block self signed certificates, there are some tricks when launching browser I can help with in Mac
Junichi: on Mac you can open
login keychain that Chromium would accept
... we have to use self signed because we are not using a fully
qualified domain name that a certificate authority could
sign
<wonsuk> browser launching option I use: /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/tmp/demo/ --ignore-certificate-errors &
Urata: local server can have self signed
Junichi: in our case it isn't possible since it is a special name
Urata: are there any other topics that people want to discuss?
[adjourned]