Informal meeting on the WebDriver spec, London, April 19th -- 19 Apr 2012

<wilhelm> Scribe: simonstewart

I like cheese

introductions

wilhelm: starts the meeting

<wilhelm> guest: We're getting you set up now. Starting with introductions.

wilhelm: works as a independent consultant, chair of the browser tools and testing group

eranm: works at google on simonstewart's team. Works on user interactions

jarib: works on webdriver. works at finn.no. Does the ruby bindings

danielwh: works at google on webdriver

I do the same thing

jimevans: works at salesforce.com. Works on Webdriver (the IE and .net bindings)

freynaud: works at ebay on automation and Grid

jleyba: googled. Focuses on JS testing and does the JS bindings

andreastt: works at opera.

represents their browser automation team

raga: works at experia as a test lead

samit: works for TomTom as a tech lead, and the maintainer of the selenium IDE

wilhelm: why are we here, how the process works
... there are several reasons for writing the webdriver spec.
... wore the test manager hat at opera for several years
... Goal is still the same. The W3 should be able to write tests for the new specs
... The easy stuff has already been done (e.g.: rendering (!))
... Things that use JS assertions are already done too
... difficult part: testing things that require some sort of user interaction
... opera originally considered watir and webdriver. Didn't care about automating websites, just the browser
... in the testing IG at the w3c we're trying to reach a consensus about which tools should be used.
... Goal is to get all w3c standards to be able to use the webdriver api to write tests
... therefore we need a spec
... in addition, there's the benefits for the OSS project to have a proper and clear spec.
... Also, I believe we can use the spec as a tool to bring other people on board with the browser automation effort
... finally, the w3c has a patent policy that members of a working group must follow.
... reduces legal risk for people implementing the standard
... it's also my birthday.
... we'll talk until lunch and then clear up and disputes in the spec afterwards

raga: will this replace any other specs?

wilhelm: will just focus on browser automation, allowing others to test the things that webdriver does
... notice that this is an informal meeting, so anyone can join in. Would be happy to help new members come and join the group
... I hereby invite all of you as invited experts to the WG.

http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html

<wilhelm> Scribe: wilhelm

Spec overview

simonstewart: I've never done this before – if you have questions, just ask.
... Skipping the abstract…
... Leaping straight into the introduction.

… Missing expected use cases in the introduction.

… Three use cases should be added: people who want to test web sites, …

… people who are working on a browser itself and want to verify that the browser works correctly, people who write specifications that define how a browser should behave.

… We would be dead in the water if we were to define everything independenlty. We cross-reference other specs where applicable.

… If you find an error in the spec where we should be referring to another spec, please let us know.

… Naming the two sides of the API. Traditionally you define client and server. These are overloaded terms.

… One end is the test or the user, one end is the browser that is being automated.

… Deliberately not mentioning how things are routed. How commands are formatted are up to implementors.

… There will be a non-normative section on the JSON wire protocol, which the most common implementation.

… We intend to provide as much as possible in Web IDL format.

… The webdriver spec will lay out commands that can be used, as well as a way these can be extended by vendors, avoiding clashes.

… (Detailing steps of commands and responses – see spec for exact details.)

… We'd like people to be able to do send additonal information with commands and responses. (Screenshots on error, authentication, &c.)

… Open question: should we require implementors to preserve the order of the fields?

others: Not important.

simonstewart: The numeric error codes have gaps due to legacy.
... … There's definitely a hierarchy of error codes, similar to in HTTP.

… Yes, perhaps we should group them together.

jleyba: We have implicit error codes as well due to the underlying HTTP.

jimevans: We're not specifiying that the transport will be HTTP.

jleyba: We should validate commands. If you don't give the expected parameters, server may blow up.

simonstewart: Missing part here.
... Status codes should encapsulate all possible errors.

jimevans: Currently we don't.

jleyba: We should have classes of error codes where vendors may extend (say 600 range).

simonstewart: Users generally don't do the right thing with error codes.

kevin: Are integers defined to be unsigned in WebIDL?

simonstewart: These are unsigned.

rambighananthan: When our scripts look for elements, they may not be in the right place. Is there a way to assert whether an element is within a given area?

simonstewart: This is already possible on the client using the available information about position and size.

… Goal is to keep the spec small. We want to define the building blocks people can build more complex things with.

<scribe> ACTION: Go through the status codes [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action01]

simonstewart: Spec uses capabilities in two different ways. Desired capabilities. Capabilities object for responses – this is what I support.

… The interface has a map of a dictionary of capabilities.

… Occasional objects.

… Two methods: boolean has method, get method.

kevin: In section 3.1.1 doesn't allow arbitrary objects to be returned.

simonstewart: How do you encode objects to send across the wire?

… Maybe we should tighten up the spec here.

andreastt: The capabilities we have now is an unholy mess.

kevin: How is null represented in Web IDL?

simonstewart: I don't know Web IDL very well. Feel free to correct things.

samit: Are we planning to standardize capability names?

simonstewart: We will standardize a set.

… We need a non-normative appendix of capability names with a link back to the section that defines it.

simonstewart: Some vendors may use other protocol formats.
... Section 3.1.1 – must be of type boolan, string, numerical, dictionary.

… Dictionary can contain keys of type string and types above.

… Serialization format must be clearly defined. Simplest is dictionaries.

… This section is merely defining data types.

kevin: It should be called out that the absence of a value is NULL.

jleyba: get returns the raw value of a capability and makes an interpretation of that value.

simonstewart: Many questions here are due to me not understanding how to write Web IDL. Text may be easier to read.

<jari> (union types: http://dev.w3.org/2006/webapi/WebIDL/#idl-union)

samit: Can you give an example of when mutable capabilities are needed?

simonstewart: “I want Firefox 3.6 with this particular profile”.

… You might not know this when you start.

dawagner: Is this an implementation detail?

simonstewart: I'm happy to drop this.

jleyba: I vote for dropping it.

simonstewart: I'm fine with that.

<scribe> Scribe: jari

simonstewart: topic: sessions

sessions

simonstewart: each session given uuid, opaque string

discussion removing the requirement of a uuid

simonstewart: (describing sessions - see spec)

jleyba: do we want to define a format for the command name?

simonstewart: want a format that allows for vendor extension

jleyba: current names based on firefox implementation, should we standardize on that?

simonstewart: nice to have human readable command names
... do we want command names to be case sensitive?

eranm: no benefit

<scribe> ACTION: section 3 - specify command names as case sensitive [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action02]

simonstewart: (continues reading from the section 4.1)

SamitBadle: change sessionId from SHOULD to MUST?

<SamitBadle> not, just asking.

simonstewart: why is explained later in same section

<SamitBadle> clear?

jleyba: asking if the remote end needs to respect sessionId created by the local end
... some capabilities should be set in stone

simonstewart: client should check what it gets back

discussion around fuzzy capability matching

<scribe> ACTION: Add a "required capabilities" parameter when creating the session (Section 4.1) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action03]

<guest> pmfji for 'required capabilities' you might define an 'isRequired(capability)' method.

discussing problems around required parameters when the capabilities object is an unordered set - the first capability checked would fail

jleyba: spec should say the remote end should collect as many capabilities as it can before it fail

simonstewart: i'm ok with that

discussing required capabilities in the context of grid implementation (i.e. intermediate nodes)

simonstewart: (continues reading from section 4.1)

eranm: what was conclusion on UUID?

simonstewart: remote end free to ignore UUIDs provided by the client

jleyba: don't see why local end should ever be able to set the UUID

simonstewart: known use cases exist, do we want to close the door on those?

<scribe> ACTION: server should make sure the UUID provided by the client is unique, not used before (Section 4.1) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action04]

<scribe> ACTION: fix "0" string in 4.1. point 6 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action05]

andreastt: how to specify different products in the capabilities?

simonstewart: use browserName

jleyba: are we standardizing browserNames?

simonstewart: no. ultimately each implementation will pick a browserName
... (reading section 4.1.1.1)

<guest> pmfji: perhaps use a dotted notation for capability names, e.g. "platform.version" instead of platformVersion etc.

simonstewart: just taking what's already there

SamitBadle: could we leave out "ANY" and just not send a platform capability?

kmenard: spec says "may be used"

simonstewart: may also be used to indicate that current OS is unknown from the remote end

hugs: what capability would differentiate e.g. iPhone from iPad?

simonstewart: spec says implementors may add additional platform names

(reading through section 4.1.2)

simonstewart: missing from this error handling section, required capabilitiy not fulfilled
... (reading section 4.1.3)

discussing the non-normativity of this section

Navigation (Section 5)

simonstewart: (reading from Section 5)

<scribe> ACTION: Navigating between HTTP vs HTTPS domains [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action06]

andreastt: are we restricting ourselves to dealing with HTML?

simonstewart: opens up a can of worms dealing with other things

wilhelm: needs to deal with XML content

simonstewart: page load strategies deals with HTML/HTTP
... can have non-normative section on XML

wilhelm: ok

jleyba: is the intent that these strategies apply only when explicitly say they apply, or do they apply to every command?

simonstewart: see section 5.2

<dawagner> jari: http://www.whatwg.org/specs/web-apps/current-work/multipage/dom.html#current-document-readiness

<scribe> ACTION: link to HTML spec for document.readyState [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action07]

<scribe> ACTION: Say explicitly that vendors can add additional page load strategies [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action08]

<scribe> ACTION: Link Section 5 to Section 13 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action09]

simonstewart: (reading section 5.1)
... (reading section 5.1.1)

<eranm> ACTION: Clarify in section 5.1.1 that this applies to all kinds of invalid certificates, not only self-signed ones. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action10]

<scribe> ACTION: Section 5.1.1 shouldn't use WebDriverException [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action11]

simonstewart: page load strategy specifies when the next command will execute

<scribe> ACTION: Swap order of 5.1 and 5.2 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action12]

breaking for lunch

<eranm> scribe: eranm

wilhelm: morning session was useful, suggest continue in the same format - going over the chapters

Controlling windows (section 6)

simonstewart: browser tabs are counted as separate windows

jleyba: window.top is a frame, happens to be the top-most frame. Say that when switching to a window you switch to a top-most frame.

wilhelm: the html specification uses the term "browsing context"

simonstewart: todo: define frame.
... 6.2 window handles.

<wilhelm> http://www.whatwg.org/specs/web-apps/current-work/#windows

simonstewart: suggesting a window class to refer to

jimevans: window IDs returned from the IE driver are no longer the HWND, this is an opaque string

<scribe> ACTION: we should make clear that the window handle is unique within the session [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action13]

andreastt: assuming we're talking about visible windows

jleyba: chrome also has hidden windows, we may have to be more restrictive

<scribe> ACTION: andreastt and jleyba, re-word the language in 6.3 on the windows returned from there [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action14]

jimevans: suggests specifying that the order of handles returned from getWindowHandles does not imply anything about the order of opened windows

simonstewart: the returned value will be changed to a set of strings to emphasize the order is not defined

freynaud: calling window.close is difficult for the grid, because it does not know the driver is done when the last window is closed

<scribe> ACTION: in section 6.4, take out the equivalence to quit. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action15]

use case brought up by SamitBadle is the last window of a mobile browser

(use case brought up by andreastt )

<scribe> ACTION: maximise should be in 6.5 as well [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action16]

andreastt: what happens when the command is called for a browser that cannot be resized?

<scribe> ACTION: define an exception if the browser cannot be resized to the specified size [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action17]

(case: chrome cannot be resized to less than 200 pixels)

<scribe> ACTION: maximise and full screen are needed as commands. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action18]

jimevans: maximising and resizing to the screen size are not the same

simonstewart: leave positioning out for now, not enough use cases to justify it
... explaining os-specific scaling (window zoom).

jleyba: is that like pinch gesture on mobile? others: no

<scribe> ACTION: jleyba and andreastt will write section 6.6 about window zoom [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action19]

jleyba: what happens if the window is no longer there? simonstewart : an exception is thrown if the window does not exist anymore

eranm: when a new window opens, we should specify commands are still processed in the original window. simonstewart : handled in section 7.

7 Where commands are handled

<scribe> ACTION: fix typo in 7.1, window._top -> window.top [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action20]

jleyba: section 7.2, if the id attribute is not standard on window, we should not check for it

wilhelm: editorial comment: use normative words in section 7.2, (must/should)

<scribe> ACTION: use normative words in section 7.2, (must/should) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action21]

<scribe> ACTION: jleyba : document which commands are safe to execute without a window. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action22]

simonstewart: closing the last window is not possible only on some platforms (mobile), definitely possible on others (OS X)

eranm: does the section about multiple windows in 7.2 only about the multiple windows being opened on start-up?

simonstewart: could also happen when attaching to an existing browser
... when you create a new session, if that new session creates only one window, this is the current window.
... in case more than one window is being opened, then you have to make a choice

<scribe> ACTION: change throws clauses to error clauses (errors are returned, could be translated to exceptions in the client side) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action23]

<scribe> ACTION: add null as an option in 7.3 (frame specification can be null, in which case it's the top-most one) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action24]

<scribe> ACTION: be explicit in 7.3 that domain boundaries can be crossed [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action25]

<scribe> ACTION: section 7.3, step 2.1 should be changed to null (see action 24) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action26]

simonstewart: section 7.3, step 3: so frame/iframe elements could be found and used in a switchToFrame command

8, running without focus.

jleyba: do we want the ability to switch up? simonstewart: users can find the way down again

SamitBadle: what happens when switching to a frame then switching to another window and switching back? simonstewart: specification says switching to a window switches to the top-most frame in it.

<dawagner> #2863

9 elements

jleyba: suggest specifying that all IDs, unless specified otherwise, are UUIDs.

<scribe> ACTION: specify that all IDs, unless specified otherwise, are UUIDs. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action27]

klm: introducing himself: Michael Klepnikov, working at Google on latency testing, interested in WebDriver. Specifically interested in browsers exporting existing information about timing of actions.

<simonstewart> http://dvcs.w3.org/hg/webdriver/raw-file/tip/webdriver-spec.html#logging

switching to section D, logging:

D, logging (klm's interest)

simonstewart: the basic idea is that there's a logs interface which is queried for a specific kind of logs: browser logs, etc.
... this is a uniform interface for getting logs for any stage, from the test itself , through the grid, to the browser launching itself

wilhelm: which problem are we trying to solve?

simonstewart: the test has failed, user would like to see the reason of the failure: out-of-memory in an intermediate step, commands executed before finding an event timed out.

hugs: another example is getting the javascript console before looking at the failure of the tests

<simonstewart> davidburns: do you want to be dialed into this meeting?

<davidburns> yea for a bit if that is ok?

<davidburns> simonstewart: ^

<simonstewart> eranm or dawagner: do you know the incantation to get davidburns in?

dawagner, can you invite?

<davidburns> last time we did a Google hangout

<dawagner> Yep, davidburns: Email address?

simonstewart: define a mechanism for moving logs

<davidburns> david.burns@theautomatedtester.co.uk

davidburns, is that your google+ account?

<davidburns> eranm: yup

<dawagner> You should have email

simonstewart: when calling getlogs(), do I get the partial logs since the last invocation or all logs ?

wilhelm: mentions another spec about logging.

<wilhelm> http://www.w3.org/2011/08/browser-testing-charter.html

SamitBadle: which performance data are we interested in?

klm: will instruct the browser to start collecting the performance data. Once that's collected, it should be in a separate channel, not polluted with other logging data

<davidburns> simonstewart: fyi https://twitter.com/#!/AutomatedTester/status/192764437348880384

freynaud: provide incremental logs, so you know which period of logs you're getting

klm: don't see a difference between console logs and timing logs

simonstewart: suggest passing a filter when fetching the messages
... start recording X log type, stop recording X log type, get all categories of log types, clear logs.

klm: if there are two parts of the test framework that don't know about each other, incremental vs. full could be a problem

simonstewart: clear is called for a specific type, subsequent get will return logs from the last clear or start record commands

andreastt: some logs are document-specific, some logs are server-global, do we mix them?

simonstewart: call clear and get logs for logs you care about.

simonstewart, andreastt : when a session is deleted, the logs should be deleted as well.

simonstewart, the spec should define how to clear and collect the logs, each log type will have specific details which will not be specified here.

<scribe> ACTION: simonstewart : write section D on logs [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action28]

9, Elements (back)

simonstewart: the spec does not define the protocols or the transport layer, how commands are transmitted.
... different types of encodings are suitable for different kinds of devices.

<scribe> ACTION: eranm: section 9.1, the WebElementList SHOULD be immutable. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action29]

<scribe> ACTION: simonstewart switch WebElementList to array of WebElement (native WebIDL) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action30]

jari: there's even a readonly type.

<jari> http://www.w3.org/TR/WebIDL/#idl-array

hugs, can we move XPath to be last? simonstewart : other locator strategies may be implemented using that. Agrees to moving it down.

<scribe> ACTION: 9.2.2. change to javascript. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action31]

<scribe> ACTION: jleyba section 9.2.3 shorten capability name [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action32]

<scribe> ACTION: jleyba , section 9.2.4 re-phrase to use CSS selectors other than xpath. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action33]

dawagner: we currently disallow compound CSS selectors, not disallowed by the spec.

wilhelm: clarifies that specifying a css selector "h1,h2" will return elements of either type.

<scribe> ACTION: simonstewart : allow compound css selectors. Get rid of class selectors. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action34]

<scribe> ACTION: section 9.2.5, spell out availability of functionality against html and xhtml [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action35]

<scribe> ACTION: section 9.2.5 drop step 3 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action36]

simonstewart: 9.2.7 aria - using the accessibility API.

<simonstewart> http://www.w3.org/TR/wai-aria/

<scribe> ACTION: write section 9.2.7 , if you're familiar with the spec. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action37]

wilhelm: we have 21.5 minutes left

<scribe> ACTION: dawagner : visibility ancestry is different than dom ancestry [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action38]

jleyba: section 10 bullet 3, is in accurate.

inaccurate.

<scribe> ACTION: dawagner : follow up on 10, last bullet point - element inside a fixed-sized parent, outside the size. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action39]

discussion about properties, attributes, regarding section 10.1

jari, dawagner : we should not conflate attributes and properties.

<scribe> ACTION: 10.1, re-work this section so that only style and aliases are left over. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action40]

<jari> slightly relevant section of the HTML spec on properties vs attributes: http://www.whatwg.org/specs/web-apps/current-work/#reflecting-content-attributes-in-idl-attributes

dawagner, we could do aliasing in the client libraries. simonstewart : let's leave in the spec

<scribe> ACTION: section 10.1.2.1 css should always be lower case, colour should always be rgba. [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action41]

<jari> scribe: jari

<scribe> ACTION: 10.1.1 - reword to be consistent with the HTML spec (see http://www.whatwg.org/specs/web-apps/current-work/#reflecting-content-attributes-in-idl-attributes) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action42]

jleyba: what about non-standard boolean attributes?

dawagner: example: "disabled" on input elements different from "disabled" on a div

<scribe> ACTION: 10.1.1 - general cleanup [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action43]

simonstewart: (reading section 10.1.2)
... HTML4 and HTML5 differs in the boolean value of disabled="cheese" and disabled="disabled"

<scribe> ACTION: wilhelm investigates HTML4 vs HTML5 and boolean attributes (esp. disabled on input vs non-input elements) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action44]

simonstewart: moving to section 10.1.3

<scribe> ACTION: getElementAttribute algorithm should be explicit why we're special casing URLs (reason: always return absolute URLs) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action45]

<scribe> ACTION: 10.1.3 - "algorithm defined above" should be named and linked [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action46]

<wilhelm> + iframes, frame, link, map, area

simonstewart: moving on to 10.2

<davidburns> I am heading out, will try join in about 15 mins again

reading through the algorithm for getting the visible text of an element

<dawagner> ACTION: dawagner: Fix 1.1 of the rendering text algorithm [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action47]

jimevans: is there need for a command to determine the presence of an element?

jleyba: the efficient way to do this has already been established

not going to be added.

jleyba: reads section 11, Executing JavaScript

<scribe> ACTION: use "DOMString" WebIDL type over "string" [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action48]

jleyba: (reading section 11.1, 11.2, 11.3)

dawagner: 11.1: Argument type; dictionary should be of Argument types

<scribe> ACTION: limit recursion depth in recursive algorithms (a SHOULD) [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action49]

discussing alert() in user-provided javascript

simonstewart: should return null to user and browser is in undefined state

jleyba: section 11.4, what do we do with unhandled javascript exceptions on the page? need API for capturing and checking

simonstewart: should be able to use captured logs

jleyba: if there's an unhandled error and you haven't queried for it, the next command should fail

simonstewart: no

jleyba: could be optionally enabled
... similar to how alerts are handled
... store errors that have occured on session; add getJavaScriptErrors
... webdriver perfectly capable of propagating failure when JS errors occur

simonstewart: not sure; gut feeling, seems like overkill
... not enough people care about this

leaving this as an open question

dawagner: could have more specific type than UnknownError when we know what's happening (Section 11.3 step 2)

jleyba: could also not throw and just return null

dawagner: makes more sense to throw

<scribe> ACTION: link to DOM4 instead of DOM3 [recorded in http://www.w3.org/2012/04/19-testing-minutes.html#action50]

- DRAFT -

Informal meeting on the WebDriver spec, London, April 19th

19 Apr 2012

Attendees

Contents