04:32:22 RRSAgent has joined #testing 04:32:22 logging to https://www.w3.org/2019/09/18-testing-irc 04:32:48 RRSAgent: make logs public 04:32:55 jorydotcom has joined #testing 04:33:11 myles has joined #testing 04:33:39 rachelandrew_ has joined #testing 04:33:44 ScribeNick: boaz 04:33:51 plh has joined #testing 04:33:54 “The name comes from earlier variants on the same theme” 04:33:56 boaz has joined #testing 04:33:57 present+ 04:34:09 scribenick: boaz 04:34:11 present+ 04:34:14 joemedley has joined #testing 04:34:27 present+ 04:34:28 present+ 04:34:30 presentation beginning 04:34:31 cybai has joined #testing 04:34:33 JohnRiv has joined #testing 04:34:38 present+ 04:34:38 present+ 04:34:41 shimazu has joined #testing 04:34:44 present+ 04:34:48 bkardell_ has joined #testing 04:34:52 present+ myles 04:34:53 bwalderman has joined #testing 04:34:59 foolip: ill be presenting some updates on our work on wpt for the last year 04:35:10 foolip: wpt is our collection of tests for most of the web platform 04:35:38 foolip: in the last year we've added 5k tests (a lot from css) 04:36:10 foolip: also since last year is more results on more browsers 04:36:11 aboxhall_ has joined #testing 04:36:29 rachelandrew_ has joined #testing 04:36:33 q? 04:36:48 Zakim has joined #testing 04:36:49 present+ Rachel Andrew 04:37:17 present+ 04:37:30 foolip: worked with bocoup to improve the documentation this last year. we added search. we deduped and made the content easier to navigate. 04:37:31 slides are https://docs.google.com/presentation/d/1DqzI_GhOE86CKBjJj0SPhf2oUxNE5w94dYzuACGa2oc/edit?usp=sharing 04:37:42 https://web-platform-tests.org/ for new documentation 04:37:46 foolip: slides are https://docs.google.com/presentation/d/1DqzI_GhOE86CKBjJj0SPhf2oUxNE5w94dYzuACGa2oc/edit?usp=sharing 04:37:47 zcorpan has joined #testing 04:37:49 zcorpan has joined #testing 04:38:00 foolip: we improved fuzzy reftests 04:38:09 astearns: how many have fuziness turned on 04:38:26 cathiechen has joined #testing 04:39:16 jgraham: a bunch 04:39:25 foolip: added comparison tool for ref test results 04:39:50 foolip: we also added filtering to the the search bar of wpt.fyi 04:40:29 denis has joined #testing 04:41:19 foolip: live demo of the filtering and reftest comparison 04:41:47 foolip: priorities for 2019/2020 are to 1) improve interop, looking at driving down browser specific failures 04:42:11 present+ 04:44:18 foolip: ... 2) making PR results more actionable and reliable by taking our pr bot which runs tests, and making the output tell you what to do 04:45:06 foolip: 3-7) also test failure triage, search test by metadata, mobile testing, address flakiness, and wpt.fyi ui improvments (ease of use, ally) 04:45:44 xiaoqian has joined #testing 04:45:50 q? 04:45:59 rtoyg_m2 has joined #testing 04:46:17 zghadyali has joined #testing 04:46:19 astearns: there are a ton of open PRs on wpt. where does getting through that bag log fit in your priorities? 04:47:15 jgraham: we have a reviewer system that consists of yml files that name people as reviewers on a per directory basis 04:47:38 zcorpan: if there is a subject matter expert not in there, you should ask them to add themselves 04:48:05 jgraham: we have a fundamental problem with this, because we can't get reviewers for certain technologies 04:48:44 q+ AmeliaBR 04:48:52 astearns: I think thats fair. I know a lot of the PRs are on CSS tests. so I think it is on my to push people towards adding themselves as reviewers. 04:48:54 boaz ^ 04:49:27 q+ 04:50:17 ack AmeliaBR 04:50:36 jorydotcom has joined #testing 04:51:13 karl has joined #testing 04:51:44 AmeliaBR: on the issue of pr backlogs, I think it is worth having a discussion about making better tooling to have things like reassigning issues with no reviews after a week, or on a semireg basis sending an email to the working group about their unreviewed tests 04:51:55 astearns: I could try that manually for CSS 04:52:26 q+ 04:52:44 q+ 04:52:48 present lukebjerring 04:52:55 present+ lukebjerring 04:53:02 present+ 04:53:39 q? 04:53:41 ack foolip 04:53:44 lukebjerring: there is a tool called github-health.appspot.com that uses the google philosophy. 04:53:57 https://github-health.appspot.com 04:54:17 diemol has joined #testing 04:54:44 foolip: there is a tradeoff, sometimes you have to close out reviews that you can't get to, or sometimes you have to go to amillion. crbug is at amillion. 04:55:02 JohnJansen: there is not an easy mapping between wpt dir structure and wgs and specs. 04:55:14 jgraham: it is supposed to be good. 04:55:26 JohnJansen: asks a q to prove that it is not 04:55:51 jgraham: the PRs get labeled 04:55:58 JohnJansen: ok im satisfied 04:56:13 s/JohnJansen: ok im satisfied/ / 04:56:26 q+ to ask about managing not-ready-to-merge PRs 04:56:56 JohnJansen: at my work, I want to go review some tests. I'm not in the wg or working on a spec, but I want to find the technology area that I am interested in 04:57:02 jgraham: im not sure this is possible 04:57:13 jgraham: labels should address this 04:57:36 zcorpan: this is similar to how to find a test for the feature I'm interested in 04:57:43 zcorpan: you can use grep to do this 04:57:44 q? 04:57:48 ack JohnJansen 04:57:50 ack JohnJansen 04:58:13 ack jgraham 04:58:31 cybai_ has joined #testing 04:58:36 jgraham: good idea to email wgs, if you try that in css, please let us know how it goes. 04:58:54 q? 04:59:07 astearns: experimenting manually will help me figure out what to do 04:59:31 ack AmeliaBR 04:59:31 AmeliaBR, you wanted to ask about managing not-ready-to-merge PRs 04:59:32 ack me 04:59:33 q? 04:59:36 boaz: will you summarize? 04:59:38 astearns: yes 05:00:02 AmeliaBR: in addition to not getting reviews, and sometimes the review leads to a spec issue. 05:00:33 gsnedders: we have a label for this in the PR 05:00:41 AmeliaBR: can I get docs to that? 05:00:45 q+ 05:00:55 wjmaclean has joined #testing 05:01:13 boaz: let's talk later 05:01:34 scribnic none 05:01:36 foolip: yah, or link that in the 2020 priorities doc 05:01:40 scribenic: boa 05:01:42 scribenic: boaz 05:02:15 scribenick: boaz 05:02:35 q? 05:02:37 q+ 05:02:39 q- 05:02:40 foolip: yes I have an issue open 05:02:44 q? 05:03:06 JohnJansen: it sounds like some members of css wg have filed tests expecting someone outside of the wg to review 05:03:14 astearns: no, its an internal complaint 05:03:32 q? 05:03:37 ack JohnRiv 05:03:38 ack JohnJansen 05:04:39 astearns: what about the meta data tagging system 05:04:56 lukebjerring: we are thinking of adding a manifest 05:05:29 lukebjerring: we have a pr for showing open PRs in wpt.fyi 05:05:30 jugglinmike has joined #testing 05:05:45 q? 05:06:24 +1 to lukebjerring design to let me see PRs open for current directory. that would be much better for my mental model than searching labels 05:06:45 jorydotcom has joined #testing 05:06:46 JohnJansen: +1s to lukebjerring design to let me see PRs open for current directory. that would be much better for my mental model than searching labels 05:07:28 q? 05:07:33 q+ 05:08:06 foolip: we have identified in the past that test automation is tricky. is that something people are blocked by? 05:08:31 AmeliaBR: im not sure what you mean by automation, but I love the wptests.center 05:08:37 lukebjerring: we're going to integrate that 05:08:53 AmeliaBR: to be able to use that for reftests that'd be greate 05:09:01 s/greate/great 05:09:22 temporary url for feedback on test authoring: https://wptest.center/#/new 05:09:38 foolip: I was refering more to manual tests. how often do you have to write a tests for manual tests 05:09:59 AmeliaBR: we have a legacy issue with svg, where our existing tests were from a manual testing era 05:10:13 AmeliaBR: manual comes up with interaction tests. there could be a way to automate those. 05:10:43 foolip: we have a way to automate mouse/keyboard input, we support that in all the browsers so long as you are writing testharness.js tests 05:10:58 foolip: do you need it for ref tests? 05:11:00 (14wpt) [PR] jgraham 03merged 13#19029 into 07master: [Gecko Bug 1574690] P3 longer timeout to ease intermittent COEP test - https://git.io/JemhQ 05:11:28 AmeliaBR: we do have declaritive animations that should work with js turned off 05:11:32 foolip: that is tricky 05:11:49 foolip: how many manual svg tests to you have 05:11:51 ? 05:12:04 gsnedders: 514 manual tests in svg 05:12:36 boaz: has anyone recently written a manual test? 05:12:58 AmeliaBR: we did one recently because doing it in an automated way was too hard 05:13:09 (14wpt) [PR] jgraham 03merged 13#19116 into 07master: Fixed lint error in #18886 - https://git.io/JeOgS 05:13:52 foolip: tell us when you run into this and we can advise you 05:14:04 zcorpan: also please file an issue with type untestable 05:14:14 s/type untestable/type:untestable/ 05:14:28 lukebjerring: if we fix the issues that make svg need 514 manual tests, whats the liklihood that they will be replaced? 05:14:36 AmeliaBR: that's a big to do 05:15:18 q+ SimonSapin 05:15:20 q? 05:15:31 ack SimonSapin 05:15:32 ack SimonSapin 05:16:07 SimonSapin: in the case of SVG I think its just conversion work. buttt its hard to make a reference for some svg features without using the svg features 05:16:12 bwalderman has left #testing 05:16:18 q? 05:16:20 q+ 05:16:35 AmeliaBR: this has definitly been an issue 05:16:42 shimazu has joined #testing 05:17:07 q+ lukebjerring 05:17:09 ack boaz 05:17:11 s/just conversion/not just conversion/ 05:17:17 zcorpan: my immediate thought about how to automate those tests is to have an image 05:17:29 boaz: let's keep going with this topic for a bit 05:17:30 q? 05:17:32 zcorpan: and then add fuzziness 05:17:34 q- 05:17:57 Replacing actual eyeballs with fuzzy matching 05:18:27 q? 05:18:34 lukebjerring: is there tooling that could walk you through manual tests and generate reftests from that 05:18:34 q- 05:18:37 ack lukebjerring 05:18:44 gsnedders: problem with that is text 05:18:46 q+ 05:19:08 jgraham: using images as references isn't a great pattern 05:20:04 q? 05:20:17 boaz: how about mdn survey data and wpt data 05:20:26 q+ 05:20:46 kadir: now that we know that interop is a big problem it could be nice to dive into what the specific issues are 05:20:56 ack boaz 05:20:58 ack foolip 05:21:16 foolip: work on a compat/interop follow up survey, then look at what we have in wpt and figure out what to follow up on 05:21:20 q+ 05:21:25 rtoyg_m2 has joined #testing 05:21:26 q+ 05:21:43 foolip: we want to see these things line up but my guess is that it wont 05:21:52 q? 05:21:55 ack AmeliaBR 05:21:56 ack AmeliaBR 05:22:04 q+ lukebjerring 05:22:32 AmeliaBR: one thing on this is that the wpt approach to interop is different than the web developer's. because wpt only focuses on latest browser, where as web devs have to support old versions. 05:23:00 rachelandrew_: I agree, that a lot of the issues will be older browsers 05:23:12 ack boaz 05:23:14 q? 05:23:44 ack lukebjerring 05:23:52 boaz: we should have a consistent vocab between wpt and mdn survey 05:24:02 lukebjerring: yes, i agree 05:24:24 q? 05:24:44 lukebjerring: it could be good to name a feature in wpt, and ask a dev if they think the feature is interoperable 05:25:11 q? 05:25:24 boaz: not sure if we can ask web devs about this in terms of wpt 05:25:32 q+ karl 05:25:35 q- karl 05:25:37 AmeliaBR: I think we'll need to group specs together to ask people 05:25:53 AmeliaBR: I also think we'll need to separate frustrations from legacy browsers vs current ones 05:26:22 (14wpt) [PR] Honry 04closed 13#18886: [sensor] Upstream sensor layout tests to wpt - https://git.io/fjjly 05:26:26 (14wpt) [PR] Honry reopened 13#18886: [sensor] Upstream sensor layout tests to wpt - https://git.io/fjjly 05:26:50 RRSAgent, make minutes 05:26:50 I have made the request to generate https://www.w3.org/2019/09/18-testing-minutes.html gsnedders 05:26:54 zcorpan has joined #testing 05:26:55 RRSAgent, stop