WebDriver F2F, June 14th 2013 -- 14 Jun 2013

<wilhelm> Scribe: wilhelm

Test suite

<scribe> Scribe: plh

David: it was more working on it and writing tests

John: not sure if this is the venue or email but I'd like to understand the end-to-end plan for it

David: it's in mercurial

<sstewart6> https://dvcs.w3.org/hg/webdriver-test

David: we're trying to port the open source project tests
... those have grown quite organically over the years
... sort them, etc.
... we're trying to move them across
... there are parts when the tests and the spec diverge
... like at TPAC
... but we think the spec is correct and need to fix the tests

Simon: we're taking the tests andmaking sure they map to the spec
... once we're complete with the move, we can remove the tests in selenium

plh: license for the tests?

Simon: all Apache 2 tests
... we're clean on that front

John: is there a structure for submission?
... like in the CSS WG

Simon: correct, it would be better to have a holding

plh: [trying to motivate the group to move to web-platform-tests github]

<AutomatedTester> https://github.com/w3c/web-platform-tests

All: looks like a compelling case

Simon: let's do it then

Resolution: webdriver tests will merge into web-platform-tests

Simon: we already covered language in the test suite (python)

Wilhelm: we should distribute chapters

<jgraham> (not been following, but it's not clear to me that webdriver tests are much like the other tests there, or if they have the same license)

Simon: will look 9, 10, 11
... reading element state and executing JS

<jgraham> (although there are ofc adcvantages e.g. you get free documentation)

Wilhelm: I'll do 16 screenshots

David: I'll do 10 and 17

Simon: I'll do 9 and 11

David: I'll do 14 as well

<sstewart6> https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html

(confusion about the spec version :)

Ken: I'll do 5 and 6

John: I'll do 2 and 3

Chris: I'll do 15

<scribe> ACTION: Simon to figure testing for chapters 9 and 11 [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action01]

<scribe> ACTION: David to figure testing for chapters 10, 14 and 17 [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action02]

<scribe> ACTION: John to figure testing for chapters 2 and 3 [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action03]

<scribe> ACTION: Wilhelm to figure testing for chapters 16 [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action04]

<scribe> ACTION: Chris to figure testing for chapters 15 [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action05]

<scribe> ACTION: Ken to figure testing for chapters 5 and 6 [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action06]

<scribe> ACTION: John to figure out testing for chapter 1 [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action07]

<scribe> ACTION: Simon to create the web-platform-tests directory and move tests there [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action08]

Simon: we'll figure out the other chapters later

[figuring out the next agenda item]

SysApps

<sstewart6> http://www.w3.org/wiki/System_Applications

David: those are related to OS APIs. we're doing testing with marionette
... we have JS libraries for access to things, like contacts, set geolocation, etc.
... we do tests for batteries, bluetooth, livemaps, etc.
... not sure I don't think we need to worry about it. we didn't extend marionette for them, we use the execute script method

[Simon goes through the list of APIs]

David: in Firefox OS we have a browser within a browser

Simon: we must go deeper :)

David: everything is based in an iframe
... we try to prevent cross browser contamination
... chrome os is doing similar things
... a lot of these have come pout of Mozilla while working on firefox os

<sstewart6> Specs appear to be here: https://github.com/sysapps

David: they are certain OEM who see value in it, like Samsung
... don't think there are a lot of difference
... the webdriver is low enough and don't need tweaking
... Marionette is the only I think I can use to test firefox os

<scribe> ACTION: David to double check there is no need to extend WebDriver because of Firefox OS [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action09]

Simon: it works because the OS is written in JS...

David: agree but others would be out of scope

Simon: ok

Security dialogs

<sstewart6> driver.authtenticateAs(Credentials)

<sstewart6> http://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/Alert.html#authenticateUsing(org.openqa.selenium.security.Credentials)

Simon: it is currently unimplemented
... it's there only as a local end API at the moment
... this isn't handling things like OpenID
... you would just write that use normal primitive
... we model all modal dialogs as an alert

John: this is targetting the security dialog but we didn't target modal dialogs yet, right?

Simon: it's in the wired protocol

<sstewart6> https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/alert_text

<sstewart6> https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/accept_alert

<sstewart6> https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/dismiss_alert

David: this is the open source wire protocol, not the google one

Simon: this will be migrated into the missing appendix
... the way we handle dialog: we assume they all have the same level functionality: dismiss, cancel, send keys, get the text of the alter itself
... make sure to fill a field, ie finding out what the prompt is about
... because you're doing auth, you may do user/passwrd but there is a level of indirection
... do we want to allow webdriver to access basic digest? or is it out of scope?

Wilhelm: right now, I'm sending keys to the browser

John: automating a security dialog is a threat imho. it needs to be detailed and complete.
... doing something that sounds insecure wouldn't fly in the company
... we might not do it even if the spec says we have to
... the limitation here that wouldn't be able to use webdriver for auth

Wilhem: how can I auth my staging area?

John: would need to auth first

Simon: but we can't connect to a running server
... on services like Sauce Labs, you don't even have access...

David: for marionette, it's in the nightly. we got it approved: you would need to pass a flag
... for desktop, it's command line flag. this goes through a number of checks, including certain preferences
... if one is missing, marionette doesn't start
... ie you have to do several steps to put you in an insecure state
... for mobile, it's going to be different
... we'll visit that once we're ready for the general public
... that's how we got through the security review

Simon: the command line turns the browser into something that is listening

David: you need to have a number of commands in order to activate it
... you can't remote over a network
... you would need to have an intermediairy on the remote machine
... you can't speak to marionette over the network
... could be overriden but require deep knowledge of preferences
... of course, if you can do silly things

John: interesting approach even if 99.9% guarantee isn't enough
... in order to be hacked you need to make 3 or 4 wrong decisions

David: correct

Ken: it's a similar approach in Chrome
... for android, not sure how the options get passed
... since we don't speak wire protocol in chrome, you need a separate binary

David: yes, we have a shim for the http protocol as well
... preventing add-ons from switching on marionette was vital
... those aren't allowed in the store. those get automatically flagged
... if you don't go through the store, all bets are off
... I can find the security discussion meeting...

Simon: if you've got an API for automated browser, spammer can use it. there are JS variables to detect webdriver was enabled.
... should we agree on a fingerprint

John: seems like a must to me
... otherwise webdriver will be used against captcha

<AutomatedTester> https://bugzilla.mozilla.org/show_bug.cgi?id=870576 https://wiki.mozilla.org/Security/Reviews/MarionetteCLIAll

Simon: so, is our spec the right place to put that in and should it be the same in all browsers?
... initially, I was doing security through obscurity...

John: the obscurity will get clear very quickly...
... there has to be something else

Simon: if you set an attribute on a document, it can get overriden...
... in the ideal world, you would set a header
... and you would set an attribute that cannot overriden, like document.automated
... but then you can take advantage of the scoping rule...
... and substitute it

David: some large websites, like Amazon, do detect things like phantomjs already

Simon: I'm going to siuggest a header

John: we might learn from XHR/CORS
... it's sort of a similar problem. the combination of header and attribute could do it
... so we must have a fingerprint, with a header and an attribute, and do a security review

Simon: let's put a strawman in the spec and get review

<scribe> ACTION: Simon to put a proposal in the spec around the security mechanism header and attribute [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action10]

<scribe> ACTION: All get their security teams to review Simon's proposal [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action11]

<scribe> ACTION: Wilhelm to ping the Web Security WG on Simon's proposal [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action12]

John: we have non-modal notification, like saveAs, etc.

Simon: we didn't run into those yet, like geolocation. we just defaulted everything to a particular state

John: some dialogs have several states, like the activeX (no, never, yes once, yes always)

Simon: that's leading in extra capabilities in HTML5...
... does user media use the same prompt as geoloc? are those consistent?

David: we're consistent

John: as we are

David: we call those door hangers

[David shows the design]

[one button with several options]

Simon: so, what do we do? provide an API to allow a capability?
... we nuke the profile everytime in Chrome
... so always share doesn't really matter
... we could override switchTo, like switchTo the permission dialog and then use the existing command (dismiss, ok)
... if you ignore the dialog, it will default to No

Wilhelm: when I test the geoloc spec as a browser vendr, I'd want the four options but that's only 5 orgs in the world

Simon: you can set a profile...
... so we want to pick between the two accept states

Marc: really, it's all four
... so it's its own alert object
... if we change the wire protocol to allow an index for dialog...

Simon: so alert response is the end point?

All: yes

<scribe> ACTION: Simon to change the wire protocol to add the alert response and could take 4 responses (yes, yes always, no, no, always) [recorded in http://www.w3.org/2013/06/14-testing-minutes.html#action13]

John: what about ignore?
... it's functionally equivalent to No

Wilhelm: so 4 options

John: I still would like to test this...

Marc: what about a string with predefined values?

Simon: I'm fine with mandating four and allow extensions

John: I still would like to have ignore

<AutomatedTester> Ms2ger: I just showed a doorhanger :)

Simon: if we find it's useful, we can always come back to it in version 1.1
... a few things to consider now: mocking locations, for image capture, mimicking the data

Marc: orientation...

<sstewart6> Let's get coffee!

<kkania> Scribe: kkania

<sstewart6> ok

<sstewart6> https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/location

handling location, camera, and other sensors

sstewart6: we have a number of options to do here

location, and media capture are probably the big 2

that would solve must individual use cases right now

JohnJansen: there is a slightly different question

sstewart6: it sounds like we should ask the people writing the specs to tell us what the best way to supply the mock data is

<sstewart6> plh: there will be a cat picture

AutomatedTester: we can mock geolocation in js

<sstewart6> kkania: the link above ponts at the wire protocol

sstewart6: I'm suggesting the two we should bake in now are geolocation and media capture

plh: what about device orientation?

sstewart6: and device orientation
... so there's three we should bake

<plh> http://dev.w3.org/geo/api/spec-source-orientation.html

<sstewart6> Orientation as it is now: https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/orientation

fisherii: portrait/landscape is different from device orientation

sstewart6: ok, let's go back to our original two, and allow users to specify whether it is upright or landscape

gdennis: there's portrait, landscape, portrait secondary, and landscape secondary

<gdennis> http://www.w3.org/TR/screen-orientation/

fisherii: we should base orientation off of this spec

<sstewart6> http://www.w3.org/TR/screen-orientation/#allowed-orientations

JohnJansen: I don't know if we support all 4 states

gdennis: that could be a no-op

fisherii: it could depend on the browser and device
... does the browser actually support that rotation, or device

but i think it should throw an exception if it is unsupported orientation

gdennis: i don't think it should be an exception, a user can put it in the orientation

fisherii: but screen orientation is not the same as the device orientation

AutomatedTester: so if we were to do all of this, telling the browser to change the orientation is going to be extremely difficult

so in the example of ff os, we can do it on the emulator, but not on real devices

AutomatedTester: we can't get the device to change orientation programmatically

sstewart6: well you could
... there are things we'd like to say in the spec, even if the browser can't currently do it

AutomatedTester: can we expect the browsers to do this?

fisherii: well it's capability based, whether controlling the orientation is possible

if the device doesn't support it, the capability should return false

wilhelm: so what does happen if you do turn the device in question, does the orientation change
... if the device can actually do it, we should be able to automate it

sstewart6: the language i want in the spec, is if you support a capability, you must provide the functionality
... so are we happy with orientation, the four orientation values, for geolocation, latitude, longitude, altitidue

how long do these things stick for

fisherii: until change?
... rest of the session basically

sstewart6: ok
... for media capture, we just allow people to upload a screenshot or something?

AutomatedTester: that's interesting, since you can do video and stuff
... for webrtc, that's the biggest use case

fisherii: but the video could be a still picture
... and we don't do that well on dynamic things anyways

AutomatedTester: that's fine, as long as that is documented

<sstewart6> http://www.w3.org/TR/2012/WD-html-media-capture-20120529/

wilhelm: we could maybe make it simpler by not needing a picture from the user, but by just coloring the screen example

fisherii: camera is easy also, since it is only a single image

<sstewart6> http://www.w3.org/TR/mediacapture-streams/

fisherii: all of these are just file pickers on some sense

sstewart6: yes, they all upload a file of some sort, but how they capture is differnet

fisherii: so we don't have to do anything special about these, just treat them like any other file

sstewart6: ok, that makes life a lot easier

fisherii: i was thinking when we were talking about media we were talking about media streams

sstewart6: i don't think we're ready to tackle that one yet

<wilhelm> Photo demo: http://shinydemos.com/photo-booth/

Resolution: we'll support geolocation and screen orientation and handle media capture as standard file inputs; punt on everything else

Local storage and app cache and normal html caches

<sstewart6> https://code.google.com/p/selenium/issues/detail?id=40

sstewart6: this is one of the oldest selenium bugs

fisherii: you can clear the cookies now

sstewart6: so that's interesting, since we can't clear the httponly cookies
... so, clearing caches, can we do it, should we allow it
... at google, people often wanted pre-warmed app caches so when they hit the site they could just start doing things

instead of going through all the hoops

fisherii: so they wanted to start with a known state

can't you do that in ff by serializing the profile

sstewart6: so there's three different caches, html and cookie cache, app cache, and local storage

plh: what about indexed db?

sstewart6: and indexeddb

fisherii: i assumes this is because the don't want to incur overhead of restarting the browser

sstewart6: or if they're using IE, which doesn't use a clean profile per run
... or some mobile browsers

fisherii: I would find this useful personally, if we could nuke all of that state

plh: what about clearing security state or geolocation ...?

sstewart6: ok, so we don't support clearing caches midway thru, but we guarantee on start up things are in a clean state

fisherii: except IE can't guarantee clean state

sstewart6: the spec should aim high

John: you can tell IE to clean the state on exit

fisherii: ok, that's good; although there's still difficulty on mobile

sstewart6: should it be a should or a must

John: I think it should be a should

AutomatedTester: should be a must, but pragmatically a should

sstewart6: let's start with a must right now, and we'll have a chance for review

plh: is there a way to test this is implemented

sstewart6: yes, there is, load a file, delete it, and try to access it again
... or we could stick a proxy in the way, start a new session, configure with a proxy, request a file, and then request it again, the first time with proper caching headers; restarting the browser, it should download the file again

plh: or you can just do local storage

fisherii: we want tests for all these aspects of state
... when we talk about the state, we should define what we mean

sstewart6: i think you're right, anyone disagree?

Resolution: we're not going to support clearing caches midway through test, we will attempt to start browsers in a clean state, and we will define clean state, and we will write tests around these

plh: that's for clearing, what about setting

sstewart6: let's follow what we do for cookies now, you have to go to the domain
... it might be nice, but it's not a must

<sstewart6> https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/session_storage

fisherii: there is a spec in open source for local storage/session storage

sstewart6: i think just leave this in the open source spec, until there's a need for it

<plh> [lunch break]

<sstewart6> scribe sstewart6

<sstewart6> scribe:sstewart6

<scribe> scribe: SimonStewart

<scribe> scribe: gosh darn it, let me write

<scribe> ScribeNick: sstewart6

naming of log nodes

How do we prevent collisions in log type names?

wilhelm: is it a real problem?

It's probably a problem on grid

https://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html#logging

https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/log/types

https://code.google.com/p/selenium/wiki/JsonWireProtocol#Log_Type

"server" is overloaded

Options: namespace log types, or allow intermediate nodes to rewrite type names

JohnJansen: Why not use namespaces?

plh: what would be wrong? :)

wilhelm: mentions xml namespaces in less than glowing terms
... suggests namespacing by adding some unique key from the machine
... so why not use unique keys based on the host?

Because the purpose of the logs is to provide consistent diagnostic data. Hard to do with string parsing

Perhaps add "name" and "host" to the log entry

https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/log/types

https://code.google.com/p/selenium/wiki/JsonWireProtocol#/session/:sessionId/log

gdennis: Why do you need to find out the types of logs available to you?

Because you may not know which logs are available to you

gdennis: do the names contain meaning?

They can be arbitrary.

Conversation about what the purpose of the names are

gdennis: the "all" case needs to be handled, but could be done separately

JohnJansen: say I get back the strings and we've implemented this. I'd still do the "all" query if I wasn't sure which server was wanted

Conversation about highlighting different log types

(in a tool)

gdennis: you'd be okay if there was a distinctive log type called "all" since the type is in each entry
... not sure the "types" url is necessary

JohnJansen: as long as the log itself has a type, I'm not sure that the log types are necessary

Discussion about timestamps

http://selenium.googlecode.com/git/docs/api/java/org/openqa/selenium/logging/LogCombiner.html

kkania: you could have the browser and performance logs interleaved, provided everything was done on the same machine

gdennis: how do we prevent collision of log names
... add the ability to request a log name with a given prefix

JohnJansen: and that would be the namespace?

gdennis: possibly

kkania: but what about the case where there are two types of the same server on different machines? How would you distinguish?

JohnJansen: the log types don't have meaning, so you'd need to go through them and post-process

kkania: how would the client do the Right Thing for the user

gdennis: what's the format of a log type?

kkania: not defined. Normally just writing to some memory or a file
... chromedriver tells chrome to log to a file

gdennis: what's the log type there?

kkania: browser

gdennis: where do you say that?

kkania: the client would request that

How do I request all "performance" logs?

Suggestion: three pieces of data needed to be sufficiently unique: name of log type, hostname/identifier, and server type

kkania: what about the case where there are two identical servers on the same host?

How about adding port to the hostname?

JohnJansen: that might help

In that case we don't need the server type

Resolution: add hostname and port to LogEntry

kkania: LogEntry is currently time, level, message

Discussion about memory of servers

JohnJansen: okay to add type and host identifier

and data

kkania: and data

JohnJansen: "type" is about disambiguation

kkania: is there a way for the user to request logs from one particular server?
... the suggested approach wouldn't allow them to be aware of the types available?

No, they wouldn't be aware of the log types available to them

kkania: is it important that in the spec that the user has a way to determine the order of the intermediary nodes?

what about ordering the log entries, with remote end first and client end last

Conversation about ordering of logs

gdennis: you could have a "give me an ordering of the sources"
... a "source" described by hostname and port.
... there's no way in which logs of one type come before logs of another type from the same source?
... could have "log/sources", which would return the hostnames and ports in order

what about the case where you don't want a particular type of log?

gdennis: such as "I want all the logs, but not the performance logs?"

yes

kkania: what about being able to set verbosity of logging of individual nodes?
... how about only logging just the remote end?

We added logging to allow determining where flakiness in infrastructure was occuring

Are you suggesting, kkania, dropping logging entirely?

kkania: no, just collecting logs from the most remote end

JohnJansen: a browser automation spec need not worry about the components that the automation goes through

gdennis: can optionally provide it, though
... it's okay to provide information on sources (hostname and port)
... and you can then request logs from a particular source
... perhaps with an optional type for that source

Resolution: two end points for logging. First: "/log/:type/:source" where type and source will return entries for that type and source, allowing either or both to be wildcards.
... Second end point: "/log/available" (or similar): provides a list of sources and types.
... Wire format looks like "{source: {type: [entries], type2: [entries]}, source2: {type: [entries]}}"

gdennis: proposal isn't backward compatible?

It's not

Resolution: "source" is defined as "hostname + port"

gdennis: the "excluding" case is the one that means that the "available" end point is needed

sources need to returned in order

<JohnJansen> s/throws something at/throws something very heavy at

- DRAFT -

WebDriver F2F, June 14th 2013

14 Jun 2013

Attendees

Contents