IRC log of testing on 2019-09-18

Timestamps are in UTC.

04:32:22 [RRSAgent]
RRSAgent has joined #testing
04:32:22 [RRSAgent]
logging to https://www.w3.org/2019/09/18-testing-irc
04:32:48 [foolip]
RRSAgent: make logs public
04:32:55 [jorydotcom]
jorydotcom has joined #testing
04:33:11 [myles]
myles has joined #testing
04:33:39 [rachelandrew_]
rachelandrew_ has joined #testing
04:33:44 [gsnedders]
ScribeNick: boaz
04:33:51 [plh]
plh has joined #testing
04:33:54 [astearns]
“The name comes from earlier variants on the same theme”
04:33:56 [boaz]
boaz has joined #testing
04:33:57 [boaz]
present+
04:34:09 [boaz]
scribenick: boaz
04:34:11 [foolip]
present+
04:34:14 [joemedley]
joemedley has joined #testing
04:34:27 [AmeliaBR]
present+
04:34:28 [myles]
present+
04:34:30 [boaz]
presentation beginning
04:34:31 [cybai]
cybai has joined #testing
04:34:33 [JohnRiv]
JohnRiv has joined #testing
04:34:38 [jsbell]
present+
04:34:38 [cbiesinger]
present+
04:34:41 [shimazu]
shimazu has joined #testing
04:34:44 [JohnRiv]
present+
04:34:48 [bkardell_]
bkardell_ has joined #testing
04:34:52 [myles]
present+ myles
04:34:53 [bwalderman]
bwalderman has joined #testing
04:34:59 [boaz]
foolip: ill be presenting some updates on our work on wpt for the last year
04:35:10 [boaz]
foolip: wpt is our collection of tests for most of the web platform
04:35:38 [boaz]
foolip: in the last year we've added 5k tests (a lot from css)
04:36:10 [boaz]
foolip: also since last year is more results on more browsers
04:36:11 [aboxhall_]
aboxhall_ has joined #testing
04:36:29 [rachelandrew_]
rachelandrew_ has joined #testing
04:36:33 [boaz]
q?
04:36:48 [Zakim]
Zakim has joined #testing
04:36:49 [rachelandrew_]
present+ Rachel Andrew
04:37:17 [SimonSapin]
present+
04:37:30 [boaz]
foolip: worked with bocoup to improve the documentation this last year. we added search. we deduped and made the content easier to navigate.
04:37:31 [foolip]
slides are https://docs.google.com/presentation/d/1DqzI_GhOE86CKBjJj0SPhf2oUxNE5w94dYzuACGa2oc/edit?usp=sharing
04:37:42 [gsnedders]
https://web-platform-tests.org/ for new documentation
04:37:46 [boaz]
foolip: slides are https://docs.google.com/presentation/d/1DqzI_GhOE86CKBjJj0SPhf2oUxNE5w94dYzuACGa2oc/edit?usp=sharing
04:37:47 [zcorpan]
zcorpan has joined #testing
04:37:49 [zcorpan]
zcorpan has joined #testing
04:38:00 [boaz]
foolip: we improved fuzzy reftests
04:38:09 [boaz]
astearns: how many have fuziness turned on
04:38:26 [cathiechen]
cathiechen has joined #testing
04:39:16 [boaz]
jgraham: a bunch
04:39:25 [boaz]
foolip: added comparison tool for ref test results
04:39:50 [boaz]
foolip: we also added filtering to the the search bar of wpt.fyi
04:40:29 [denis]
denis has joined #testing
04:41:19 [boaz]
foolip: live demo of the filtering and reftest comparison
04:41:47 [boaz]
foolip: priorities for 2019/2020 are to 1) improve interop, looking at driving down browser specific failures
04:42:11 [JohnJansen]
present+
04:44:18 [boaz]
foolip: ... 2) making PR results more actionable and reliable by taking our pr bot which runs tests, and making the output tell you what to do
04:45:06 [boaz]
foolip: 3-7) also test failure triage, search test by metadata, mobile testing, address flakiness, and wpt.fyi ui improvments (ease of use, ally)
04:45:44 [xiaoqian]
xiaoqian has joined #testing
04:45:50 [foolip]
q?
04:45:59 [rtoyg_m2]
rtoyg_m2 has joined #testing
04:46:17 [zghadyali]
zghadyali has joined #testing
04:46:19 [boaz]
astearns: there are a ton of open PRs on wpt. where does getting through that bag log fit in your priorities?
04:47:15 [boaz]
jgraham: we have a reviewer system that consists of yml files that name people as reviewers on a per directory basis
04:47:38 [boaz]
zcorpan: if there is a subject matter expert not in there, you should ask them to add themselves
04:48:05 [boaz]
jgraham: we have a fundamental problem with this, because we can't get reviewers for certain technologies
04:48:44 [foolip]
q+ AmeliaBR
04:48:52 [boaz]
astearns: I think thats fair. I know a lot of the PRs are on CSS tests. so I think it is on my to push people towards adding themselves as reviewers.
04:48:54 [foolip]
boaz ^
04:49:27 [foolip]
q+
04:50:17 [boaz]
ack AmeliaBR
04:50:36 [jorydotcom]
jorydotcom has joined #testing
04:51:13 [karl]
karl has joined #testing
04:51:44 [boaz]
AmeliaBR: on the issue of pr backlogs, I think it is worth having a discussion about making better tooling to have things like reassigning issues with no reviews after a week, or on a semireg basis sending an email to the working group about their unreviewed tests
04:51:55 [boaz]
astearns: I could try that manually for CSS
04:52:26 [JohnJansen]
q+
04:52:44 [jgraham]
q+
04:52:48 [foolip]
present lukebjerring
04:52:55 [foolip]
present+ lukebjerring
04:53:02 [jgraham]
present+
04:53:39 [gsnedders]
q?
04:53:41 [gsnedders]
ack foolip
04:53:44 [boaz]
lukebjerring: there is a tool called github-health.appspot.com that uses the google philosophy.
04:53:57 [zcorpan]
https://github-health.appspot.com
04:54:17 [diemol]
diemol has joined #testing
04:54:44 [boaz]
foolip: there is a tradeoff, sometimes you have to close out reviews that you can't get to, or sometimes you have to go to amillion. crbug is at amillion.
04:55:02 [boaz]
JohnJansen: there is not an easy mapping between wpt dir structure and wgs and specs.
04:55:14 [boaz]
jgraham: it is supposed to be good.
04:55:26 [boaz]
JohnJansen: asks a q to prove that it is not
04:55:51 [boaz]
jgraham: the PRs get labeled
04:55:58 [boaz]
JohnJansen: ok im satisfied
04:56:13 [boaz]
s/JohnJansen: ok im satisfied/ /
04:56:26 [AmeliaBR]
q+ to ask about managing not-ready-to-merge PRs
04:56:56 [boaz]
JohnJansen: at my work, I want to go review some tests. I'm not in the wg or working on a spec, but I want to find the technology area that I am interested in
04:57:02 [boaz]
jgraham: im not sure this is possible
04:57:13 [boaz]
jgraham: labels should address this
04:57:36 [boaz]
zcorpan: this is similar to how to find a test for the feature I'm interested in
04:57:43 [boaz]
zcorpan: you can use grep to do this
04:57:44 [foolip]
q?
04:57:48 [foolip]
ack JohnJansen
04:57:50 [boaz]
ack JohnJansen
04:58:13 [foolip]
ack jgraham
04:58:31 [cybai_]
cybai_ has joined #testing
04:58:36 [boaz]
jgraham: good idea to email wgs, if you try that in css, please let us know how it goes.
04:58:54 [foolip]
q?
04:59:07 [boaz]
astearns: experimenting manually will help me figure out what to do
04:59:31 [foolip]
ack AmeliaBR
04:59:31 [Zakim]
AmeliaBR, you wanted to ask about managing not-ready-to-merge PRs
04:59:32 [AmeliaBR]
ack me
04:59:33 [foolip]
q?
04:59:36 [boaz]
boaz: will you summarize?
04:59:38 [boaz]
astearns: yes
05:00:02 [boaz]
AmeliaBR: in addition to not getting reviews, and sometimes the review leads to a spec issue.
05:00:33 [boaz]
gsnedders: we have a label for this in the PR
05:00:41 [boaz]
AmeliaBR: can I get docs to that?
05:00:45 [foolip]
q+
05:00:55 [wjmaclean]
wjmaclean has joined #testing
05:01:13 [foolip]
boaz: let's talk later
05:01:34 [boaz]
scribnic none
05:01:36 [boaz]
foolip: yah, or link that in the 2020 priorities doc
05:01:40 [boaz]
scribenic: boa
05:01:42 [boaz]
scribenic: boaz
05:02:15 [JohnJansen]
scribenick: boaz
05:02:35 [JohnJansen]
q?
05:02:37 [JohnJansen]
q+
05:02:39 [foolip]
q-
05:02:40 [boaz]
foolip: yes I have an issue open
05:02:44 [foolip]
q?
05:03:06 [boaz]
JohnJansen: it sounds like some members of css wg have filed tests expecting someone outside of the wg to review
05:03:14 [boaz]
astearns: no, its an internal complaint
05:03:32 [foolip]
q?
05:03:37 [JohnJansen]
ack JohnRiv
05:03:38 [foolip]
ack JohnJansen
05:04:39 [boaz]
astearns: what about the meta data tagging system
05:04:56 [boaz]
lukebjerring: we are thinking of adding a manifest
05:05:29 [boaz]
lukebjerring: we have a pr for showing open PRs in wpt.fyi
05:05:30 [jugglinmike]
jugglinmike has joined #testing
05:05:45 [foolip]
q?
05:06:24 [JohnJansen]
+1 to lukebjerring design to let me see PRs open for current directory. that would be much better for my mental model than searching labels
05:06:45 [jorydotcom]
jorydotcom has joined #testing
05:06:46 [boaz]
JohnJansen: +1s to lukebjerring design to let me see PRs open for current directory. that would be much better for my mental model than searching labels
05:07:28 [foolip]
q?
05:07:33 [boaz]
q+
05:08:06 [boaz]
foolip: we have identified in the past that test automation is tricky. is that something people are blocked by?
05:08:31 [boaz]
AmeliaBR: im not sure what you mean by automation, but I love the wptests.center
05:08:37 [boaz]
lukebjerring: we're going to integrate that
05:08:53 [boaz]
AmeliaBR: to be able to use that for reftests that'd be greate
05:09:01 [boaz]
s/greate/great
05:09:22 [JohnJansen]
temporary url for feedback on test authoring: https://wptest.center/#/new
05:09:38 [boaz]
foolip: I was refering more to manual tests. how often do you have to write a tests for manual tests
05:09:59 [boaz]
AmeliaBR: we have a legacy issue with svg, where our existing tests were from a manual testing era
05:10:13 [boaz]
AmeliaBR: manual comes up with interaction tests. there could be a way to automate those.
05:10:43 [boaz]
foolip: we have a way to automate mouse/keyboard input, we support that in all the browsers so long as you are writing testharness.js tests
05:10:58 [boaz]
foolip: do you need it for ref tests?
05:11:00 [BitBot]
(14wpt) [PR] jgraham 03merged 13#19029 into 07master: [Gecko Bug 1574690] P3 longer timeout to ease intermittent COEP test - https://git.io/JemhQ
05:11:28 [boaz]
AmeliaBR: we do have declaritive animations that should work with js turned off
05:11:32 [boaz]
foolip: that is tricky
05:11:49 [boaz]
foolip: how many manual svg tests to you have
05:11:51 [boaz]
?
05:12:04 [boaz]
gsnedders: 514 manual tests in svg
05:12:36 [boaz]
boaz: has anyone recently written a manual test?
05:12:58 [boaz]
AmeliaBR: we did one recently because doing it in an automated way was too hard
05:13:09 [BitBot]
(14wpt) [PR] jgraham 03merged 13#19116 into 07master: Fixed lint error in #18886 - https://git.io/JeOgS
05:13:52 [boaz]
foolip: tell us when you run into this and we can advise you
05:14:04 [boaz]
zcorpan: also please file an issue with type untestable
05:14:14 [gsnedders]
s/type untestable/type:untestable/
05:14:28 [boaz]
lukebjerring: if we fix the issues that make svg need 514 manual tests, whats the liklihood that they will be replaced?
05:14:36 [boaz]
AmeliaBR: that's a big to do
05:15:18 [gsnedders]
q+ SimonSapin
05:15:20 [foolip]
q?
05:15:31 [gsnedders]
ack SimonSapin
05:15:32 [foolip]
ack SimonSapin
05:16:07 [boaz]
SimonSapin: in the case of SVG I think its just conversion work. buttt its hard to make a reference for some svg features without using the svg features
05:16:12 [bwalderman]
bwalderman has left #testing
05:16:18 [foolip]
q?
05:16:20 [foolip]
q+
05:16:35 [boaz]
AmeliaBR: this has definitly been an issue
05:16:42 [shimazu]
shimazu has joined #testing
05:17:07 [foolip]
q+ lukebjerring
05:17:09 [foolip]
ack boaz
05:17:11 [SimonSapin]
s/just conversion/not just conversion/
05:17:17 [boaz]
zcorpan: my immediate thought about how to automate those tests is to have an image
05:17:29 [foolip]
boaz: let's keep going with this topic for a bit
05:17:30 [foolip]
q?
05:17:32 [boaz]
zcorpan: and then add fuzziness
05:17:34 [foolip]
q-
05:17:57 [astearns]
Replacing actual eyeballs with fuzzy matching
05:18:27 [foolip]
q?
05:18:34 [boaz]
lukebjerring: is there tooling that could walk you through manual tests and generate reftests from that
05:18:34 [foolip]
q-
05:18:37 [foolip]
ack lukebjerring
05:18:44 [boaz]
gsnedders: problem with that is text
05:18:46 [boaz]
q+
05:19:08 [boaz]
jgraham: using images as references isn't a great pattern
05:20:04 [foolip]
q?
05:20:17 [boaz]
boaz: how about mdn survey data and wpt data
05:20:26 [foolip]
q+
05:20:46 [boaz]
kadir: now that we know that interop is a big problem it could be nice to dive into what the specific issues are
05:20:56 [zcorpan]
ack boaz
05:20:58 [zcorpan]
ack foolip
05:21:16 [boaz]
foolip: work on a compat/interop follow up survey, then look at what we have in wpt and figure out what to follow up on
05:21:20 [AmeliaBR]
q+
05:21:25 [rtoyg_m2]
rtoyg_m2 has joined #testing
05:21:26 [boaz]
q+
05:21:43 [boaz]
foolip: we want to see these things line up but my guess is that it wont
05:21:52 [foolip]
q?
05:21:55 [boaz]
ack AmeliaBR
05:21:56 [foolip]
ack AmeliaBR
05:22:04 [foolip]
q+ lukebjerring
05:22:32 [boaz]
AmeliaBR: one thing on this is that the wpt approach to interop is different than the web developer's. because wpt only focuses on latest browser, where as web devs have to support old versions.
05:23:00 [boaz]
rachelandrew_: I agree, that a lot of the issues will be older browsers
05:23:12 [foolip]
ack boaz
05:23:14 [foolip]
q?
05:23:44 [foolip]
ack lukebjerring
05:23:52 [boaz]
boaz: we should have a consistent vocab between wpt and mdn survey
05:24:02 [boaz]
lukebjerring: yes, i agree
05:24:24 [foolip]
q?
05:24:44 [boaz]
lukebjerring: it could be good to name a feature in wpt, and ask a dev if they think the feature is interoperable
05:25:11 [boaz]
q?
05:25:24 [boaz]
boaz: not sure if we can ask web devs about this in terms of wpt
05:25:32 [foolip]
q+ karl
05:25:35 [foolip]
q- karl
05:25:37 [boaz]
AmeliaBR: I think we'll need to group specs together to ask people
05:25:53 [boaz]
AmeliaBR: I also think we'll need to separate frustrations from legacy browsers vs current ones
05:26:22 [BitBot]
(14wpt) [PR] Honry 04closed 13#18886: [sensor] Upstream sensor layout tests to wpt - https://git.io/fjjly
05:26:26 [BitBot]
(14wpt) [PR] Honry reopened 13#18886: [sensor] Upstream sensor layout tests to wpt - https://git.io/fjjly
05:26:50 [gsnedders]
RRSAgent, make minutes
05:26:50 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/18-testing-minutes.html gsnedders
05:26:54 [zcorpan]
zcorpan has joined #testing
05:26:55 [gsnedders]
RRSAgent, stop