IRC log of testing on 2019-09-18
Timestamps are in UTC.
- 04:32:22 [RRSAgent]
- RRSAgent has joined #testing
- 04:32:22 [RRSAgent]
- logging to https://www.w3.org/2019/09/18-testing-irc
- 04:32:48 [foolip]
- RRSAgent: make logs public
- 04:32:55 [jorydotcom]
- jorydotcom has joined #testing
- 04:33:11 [myles]
- myles has joined #testing
- 04:33:39 [rachelandrew_]
- rachelandrew_ has joined #testing
- 04:33:44 [gsnedders]
- ScribeNick: boaz
- 04:33:51 [plh]
- plh has joined #testing
- 04:33:54 [astearns]
- “The name comes from earlier variants on the same theme”
- 04:33:56 [boaz]
- boaz has joined #testing
- 04:33:57 [boaz]
- present+
- 04:34:09 [boaz]
- scribenick: boaz
- 04:34:11 [foolip]
- present+
- 04:34:14 [joemedley]
- joemedley has joined #testing
- 04:34:27 [AmeliaBR]
- present+
- 04:34:28 [myles]
- present+
- 04:34:30 [boaz]
- presentation beginning
- 04:34:31 [cybai]
- cybai has joined #testing
- 04:34:33 [JohnRiv]
- JohnRiv has joined #testing
- 04:34:38 [jsbell]
- present+
- 04:34:38 [cbiesinger]
- present+
- 04:34:41 [shimazu]
- shimazu has joined #testing
- 04:34:44 [JohnRiv]
- present+
- 04:34:48 [bkardell_]
- bkardell_ has joined #testing
- 04:34:52 [myles]
- present+ myles
- 04:34:53 [bwalderman]
- bwalderman has joined #testing
- 04:34:59 [boaz]
- foolip: ill be presenting some updates on our work on wpt for the last year
- 04:35:10 [boaz]
- foolip: wpt is our collection of tests for most of the web platform
- 04:35:38 [boaz]
- foolip: in the last year we've added 5k tests (a lot from css)
- 04:36:10 [boaz]
- foolip: also since last year is more results on more browsers
- 04:36:11 [aboxhall_]
- aboxhall_ has joined #testing
- 04:36:29 [rachelandrew_]
- rachelandrew_ has joined #testing
- 04:36:33 [boaz]
- q?
- 04:36:48 [Zakim]
- Zakim has joined #testing
- 04:36:49 [rachelandrew_]
- present+ Rachel Andrew
- 04:37:17 [SimonSapin]
- present+
- 04:37:30 [boaz]
- foolip: worked with bocoup to improve the documentation this last year. we added search. we deduped and made the content easier to navigate.
- 04:37:31 [foolip]
- slides are https://docs.google.com/presentation/d/1DqzI_GhOE86CKBjJj0SPhf2oUxNE5w94dYzuACGa2oc/edit?usp=sharing
- 04:37:42 [gsnedders]
- https://web-platform-tests.org/ for new documentation
- 04:37:46 [boaz]
- foolip: slides are https://docs.google.com/presentation/d/1DqzI_GhOE86CKBjJj0SPhf2oUxNE5w94dYzuACGa2oc/edit?usp=sharing
- 04:37:47 [zcorpan]
- zcorpan has joined #testing
- 04:37:49 [zcorpan]
- zcorpan has joined #testing
- 04:38:00 [boaz]
- foolip: we improved fuzzy reftests
- 04:38:09 [boaz]
- astearns: how many have fuziness turned on
- 04:38:26 [cathiechen]
- cathiechen has joined #testing
- 04:39:16 [boaz]
- jgraham: a bunch
- 04:39:25 [boaz]
- foolip: added comparison tool for ref test results
- 04:39:50 [boaz]
- foolip: we also added filtering to the the search bar of wpt.fyi
- 04:40:29 [denis]
- denis has joined #testing
- 04:41:19 [boaz]
- foolip: live demo of the filtering and reftest comparison
- 04:41:47 [boaz]
- foolip: priorities for 2019/2020 are to 1) improve interop, looking at driving down browser specific failures
- 04:42:11 [JohnJansen]
- present+
- 04:44:18 [boaz]
- foolip: ... 2) making PR results more actionable and reliable by taking our pr bot which runs tests, and making the output tell you what to do
- 04:45:06 [boaz]
- foolip: 3-7) also test failure triage, search test by metadata, mobile testing, address flakiness, and wpt.fyi ui improvments (ease of use, ally)
- 04:45:44 [xiaoqian]
- xiaoqian has joined #testing
- 04:45:50 [foolip]
- q?
- 04:45:59 [rtoyg_m2]
- rtoyg_m2 has joined #testing
- 04:46:17 [zghadyali]
- zghadyali has joined #testing
- 04:46:19 [boaz]
- astearns: there are a ton of open PRs on wpt. where does getting through that bag log fit in your priorities?
- 04:47:15 [boaz]
- jgraham: we have a reviewer system that consists of yml files that name people as reviewers on a per directory basis
- 04:47:38 [boaz]
- zcorpan: if there is a subject matter expert not in there, you should ask them to add themselves
- 04:48:05 [boaz]
- jgraham: we have a fundamental problem with this, because we can't get reviewers for certain technologies
- 04:48:44 [foolip]
- q+ AmeliaBR
- 04:48:52 [boaz]
- astearns: I think thats fair. I know a lot of the PRs are on CSS tests. so I think it is on my to push people towards adding themselves as reviewers.
- 04:48:54 [foolip]
- boaz ^
- 04:49:27 [foolip]
- q+
- 04:50:17 [boaz]
- ack AmeliaBR
- 04:50:36 [jorydotcom]
- jorydotcom has joined #testing
- 04:51:13 [karl]
- karl has joined #testing
- 04:51:44 [boaz]
- AmeliaBR: on the issue of pr backlogs, I think it is worth having a discussion about making better tooling to have things like reassigning issues with no reviews after a week, or on a semireg basis sending an email to the working group about their unreviewed tests
- 04:51:55 [boaz]
- astearns: I could try that manually for CSS
- 04:52:26 [JohnJansen]
- q+
- 04:52:44 [jgraham]
- q+
- 04:52:48 [foolip]
- present lukebjerring
- 04:52:55 [foolip]
- present+ lukebjerring
- 04:53:02 [jgraham]
- present+
- 04:53:39 [gsnedders]
- q?
- 04:53:41 [gsnedders]
- ack foolip
- 04:53:44 [boaz]
- lukebjerring: there is a tool called github-health.appspot.com that uses the google philosophy.
- 04:53:57 [zcorpan]
- https://github-health.appspot.com
- 04:54:17 [diemol]
- diemol has joined #testing
- 04:54:44 [boaz]
- foolip: there is a tradeoff, sometimes you have to close out reviews that you can't get to, or sometimes you have to go to amillion. crbug is at amillion.
- 04:55:02 [boaz]
- JohnJansen: there is not an easy mapping between wpt dir structure and wgs and specs.
- 04:55:14 [boaz]
- jgraham: it is supposed to be good.
- 04:55:26 [boaz]
- JohnJansen: asks a q to prove that it is not
- 04:55:51 [boaz]
- jgraham: the PRs get labeled
- 04:55:58 [boaz]
- JohnJansen: ok im satisfied
- 04:56:13 [boaz]
- s/JohnJansen: ok im satisfied/ /
- 04:56:26 [AmeliaBR]
- q+ to ask about managing not-ready-to-merge PRs
- 04:56:56 [boaz]
- JohnJansen: at my work, I want to go review some tests. I'm not in the wg or working on a spec, but I want to find the technology area that I am interested in
- 04:57:02 [boaz]
- jgraham: im not sure this is possible
- 04:57:13 [boaz]
- jgraham: labels should address this
- 04:57:36 [boaz]
- zcorpan: this is similar to how to find a test for the feature I'm interested in
- 04:57:43 [boaz]
- zcorpan: you can use grep to do this
- 04:57:44 [foolip]
- q?
- 04:57:48 [foolip]
- ack JohnJansen
- 04:57:50 [boaz]
- ack JohnJansen
- 04:58:13 [foolip]
- ack jgraham
- 04:58:31 [cybai_]
- cybai_ has joined #testing
- 04:58:36 [boaz]
- jgraham: good idea to email wgs, if you try that in css, please let us know how it goes.
- 04:58:54 [foolip]
- q?
- 04:59:07 [boaz]
- astearns: experimenting manually will help me figure out what to do
- 04:59:31 [foolip]
- ack AmeliaBR
- 04:59:31 [Zakim]
- AmeliaBR, you wanted to ask about managing not-ready-to-merge PRs
- 04:59:32 [AmeliaBR]
- ack me
- 04:59:33 [foolip]
- q?
- 04:59:36 [boaz]
- boaz: will you summarize?
- 04:59:38 [boaz]
- astearns: yes
- 05:00:02 [boaz]
- AmeliaBR: in addition to not getting reviews, and sometimes the review leads to a spec issue.
- 05:00:33 [boaz]
- gsnedders: we have a label for this in the PR
- 05:00:41 [boaz]
- AmeliaBR: can I get docs to that?
- 05:00:45 [foolip]
- q+
- 05:00:55 [wjmaclean]
- wjmaclean has joined #testing
- 05:01:13 [foolip]
- boaz: let's talk later
- 05:01:34 [boaz]
- scribnic none
- 05:01:36 [boaz]
- foolip: yah, or link that in the 2020 priorities doc
- 05:01:40 [boaz]
- scribenic: boa
- 05:01:42 [boaz]
- scribenic: boaz
- 05:02:15 [JohnJansen]
- scribenick: boaz
- 05:02:35 [JohnJansen]
- q?
- 05:02:37 [JohnJansen]
- q+
- 05:02:39 [foolip]
- q-
- 05:02:40 [boaz]
- foolip: yes I have an issue open
- 05:02:44 [foolip]
- q?
- 05:03:06 [boaz]
- JohnJansen: it sounds like some members of css wg have filed tests expecting someone outside of the wg to review
- 05:03:14 [boaz]
- astearns: no, its an internal complaint
- 05:03:32 [foolip]
- q?
- 05:03:37 [JohnJansen]
- ack JohnRiv
- 05:03:38 [foolip]
- ack JohnJansen
- 05:04:39 [boaz]
- astearns: what about the meta data tagging system
- 05:04:56 [boaz]
- lukebjerring: we are thinking of adding a manifest
- 05:05:29 [boaz]
- lukebjerring: we have a pr for showing open PRs in wpt.fyi
- 05:05:30 [jugglinmike]
- jugglinmike has joined #testing
- 05:05:45 [foolip]
- q?
- 05:06:24 [JohnJansen]
- +1 to lukebjerring design to let me see PRs open for current directory. that would be much better for my mental model than searching labels
- 05:06:45 [jorydotcom]
- jorydotcom has joined #testing
- 05:06:46 [boaz]
- JohnJansen: +1s to lukebjerring design to let me see PRs open for current directory. that would be much better for my mental model than searching labels
- 05:07:28 [foolip]
- q?
- 05:07:33 [boaz]
- q+
- 05:08:06 [boaz]
- foolip: we have identified in the past that test automation is tricky. is that something people are blocked by?
- 05:08:31 [boaz]
- AmeliaBR: im not sure what you mean by automation, but I love the wptests.center
- 05:08:37 [boaz]
- lukebjerring: we're going to integrate that
- 05:08:53 [boaz]
- AmeliaBR: to be able to use that for reftests that'd be greate
- 05:09:01 [boaz]
- s/greate/great
- 05:09:22 [JohnJansen]
- temporary url for feedback on test authoring: https://wptest.center/#/new
- 05:09:38 [boaz]
- foolip: I was refering more to manual tests. how often do you have to write a tests for manual tests
- 05:09:59 [boaz]
- AmeliaBR: we have a legacy issue with svg, where our existing tests were from a manual testing era
- 05:10:13 [boaz]
- AmeliaBR: manual comes up with interaction tests. there could be a way to automate those.
- 05:10:43 [boaz]
- foolip: we have a way to automate mouse/keyboard input, we support that in all the browsers so long as you are writing testharness.js tests
- 05:10:58 [boaz]
- foolip: do you need it for ref tests?
- 05:11:00 [BitBot]
- (14wpt) [PR] jgraham 03merged 13#19029 into 07master: [Gecko Bug 1574690] P3 longer timeout to ease intermittent COEP test - https://git.io/JemhQ
- 05:11:28 [boaz]
- AmeliaBR: we do have declaritive animations that should work with js turned off
- 05:11:32 [boaz]
- foolip: that is tricky
- 05:11:49 [boaz]
- foolip: how many manual svg tests to you have
- 05:11:51 [boaz]
- ?
- 05:12:04 [boaz]
- gsnedders: 514 manual tests in svg
- 05:12:36 [boaz]
- boaz: has anyone recently written a manual test?
- 05:12:58 [boaz]
- AmeliaBR: we did one recently because doing it in an automated way was too hard
- 05:13:09 [BitBot]
- (14wpt) [PR] jgraham 03merged 13#19116 into 07master: Fixed lint error in #18886 - https://git.io/JeOgS
- 05:13:52 [boaz]
- foolip: tell us when you run into this and we can advise you
- 05:14:04 [boaz]
- zcorpan: also please file an issue with type untestable
- 05:14:14 [gsnedders]
- s/type untestable/type:untestable/
- 05:14:28 [boaz]
- lukebjerring: if we fix the issues that make svg need 514 manual tests, whats the liklihood that they will be replaced?
- 05:14:36 [boaz]
- AmeliaBR: that's a big to do
- 05:15:18 [gsnedders]
- q+ SimonSapin
- 05:15:20 [foolip]
- q?
- 05:15:31 [gsnedders]
- ack SimonSapin
- 05:15:32 [foolip]
- ack SimonSapin
- 05:16:07 [boaz]
- SimonSapin: in the case of SVG I think its just conversion work. buttt its hard to make a reference for some svg features without using the svg features
- 05:16:12 [bwalderman]
- bwalderman has left #testing
- 05:16:18 [foolip]
- q?
- 05:16:20 [foolip]
- q+
- 05:16:35 [boaz]
- AmeliaBR: this has definitly been an issue
- 05:16:42 [shimazu]
- shimazu has joined #testing
- 05:17:07 [foolip]
- q+ lukebjerring
- 05:17:09 [foolip]
- ack boaz
- 05:17:11 [SimonSapin]
- s/just conversion/not just conversion/
- 05:17:17 [boaz]
- zcorpan: my immediate thought about how to automate those tests is to have an image
- 05:17:29 [foolip]
- boaz: let's keep going with this topic for a bit
- 05:17:30 [foolip]
- q?
- 05:17:32 [boaz]
- zcorpan: and then add fuzziness
- 05:17:34 [foolip]
- q-
- 05:17:57 [astearns]
- Replacing actual eyeballs with fuzzy matching
- 05:18:27 [foolip]
- q?
- 05:18:34 [boaz]
- lukebjerring: is there tooling that could walk you through manual tests and generate reftests from that
- 05:18:34 [foolip]
- q-
- 05:18:37 [foolip]
- ack lukebjerring
- 05:18:44 [boaz]
- gsnedders: problem with that is text
- 05:18:46 [boaz]
- q+
- 05:19:08 [boaz]
- jgraham: using images as references isn't a great pattern
- 05:20:04 [foolip]
- q?
- 05:20:17 [boaz]
- boaz: how about mdn survey data and wpt data
- 05:20:26 [foolip]
- q+
- 05:20:46 [boaz]
- kadir: now that we know that interop is a big problem it could be nice to dive into what the specific issues are
- 05:20:56 [zcorpan]
- ack boaz
- 05:20:58 [zcorpan]
- ack foolip
- 05:21:16 [boaz]
- foolip: work on a compat/interop follow up survey, then look at what we have in wpt and figure out what to follow up on
- 05:21:20 [AmeliaBR]
- q+
- 05:21:25 [rtoyg_m2]
- rtoyg_m2 has joined #testing
- 05:21:26 [boaz]
- q+
- 05:21:43 [boaz]
- foolip: we want to see these things line up but my guess is that it wont
- 05:21:52 [foolip]
- q?
- 05:21:55 [boaz]
- ack AmeliaBR
- 05:21:56 [foolip]
- ack AmeliaBR
- 05:22:04 [foolip]
- q+ lukebjerring
- 05:22:32 [boaz]
- AmeliaBR: one thing on this is that the wpt approach to interop is different than the web developer's. because wpt only focuses on latest browser, where as web devs have to support old versions.
- 05:23:00 [boaz]
- rachelandrew_: I agree, that a lot of the issues will be older browsers
- 05:23:12 [foolip]
- ack boaz
- 05:23:14 [foolip]
- q?
- 05:23:44 [foolip]
- ack lukebjerring
- 05:23:52 [boaz]
- boaz: we should have a consistent vocab between wpt and mdn survey
- 05:24:02 [boaz]
- lukebjerring: yes, i agree
- 05:24:24 [foolip]
- q?
- 05:24:44 [boaz]
- lukebjerring: it could be good to name a feature in wpt, and ask a dev if they think the feature is interoperable
- 05:25:11 [boaz]
- q?
- 05:25:24 [boaz]
- boaz: not sure if we can ask web devs about this in terms of wpt
- 05:25:32 [foolip]
- q+ karl
- 05:25:35 [foolip]
- q- karl
- 05:25:37 [boaz]
- AmeliaBR: I think we'll need to group specs together to ask people
- 05:25:53 [boaz]
- AmeliaBR: I also think we'll need to separate frustrations from legacy browsers vs current ones
- 05:26:22 [BitBot]
- (14wpt) [PR] Honry 04closed 13#18886: [sensor] Upstream sensor layout tests to wpt - https://git.io/fjjly
- 05:26:26 [BitBot]
- (14wpt) [PR] Honry reopened 13#18886: [sensor] Upstream sensor layout tests to wpt - https://git.io/fjjly
- 05:26:50 [gsnedders]
- RRSAgent, make minutes
- 05:26:50 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/18-testing-minutes.html gsnedders
- 05:26:54 [zcorpan]
- zcorpan has joined #testing
- 05:26:55 [gsnedders]
- RRSAgent, stop