IRC log of testing on 2019-09-16
Timestamps are in UTC.
- 00:17:51 [RRSAgent]
- RRSAgent has joined #testing
- 00:17:51 [RRSAgent]
- logging to https://www.w3.org/2019/09/16-testing-irc
- 00:18:45 [jgraham]
- Agenda: https://docs.google.com/document/d/1_d2xUBgNn6nmiIXM6m9zSLjzYhvDS4LMrmgoxMQJKuU/edit#
- 00:19:09 [jgraham]
- RRSAgent: Make logs public
- 00:19:13 [ato]
- Is it "present+ <nick>"?
- 00:19:42 [ato]
- Komehyo
- 00:19:46 [ato]
- Uh, https://www.w3.org/2002/03/RRSAgent
- 00:20:01 [jgraham]
- Present+
- 00:20:12 [ato]
- Present+
- 00:22:42 [jgraham]
- RRSAgent: make minutes
- 00:22:42 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html jgraham
- 00:27:08 [jgraham]
- https://blog.mozilla.org/opendesign/firefox-the-evolution-of-a-brand/
- 00:27:12 [estes]
- estes has joined #testing
- 00:27:31 [BitBot]
- (14wpt) [PR] moz-wptsync-bot requested 13#19070 merge into 07master: [Gecko Bug 1315892] text-orientation: upright' forces used 'direction' to LTR. - https://git.io/JeOvA
- 00:31:17 [Hexcles]
- Hexcles has joined #testing
- 00:31:46 [JohnJansen]
- present+
- 00:31:57 [Hexcles]
- present+
- 00:31:57 [CalebRouleau]
- present+
- 00:31:59 [AutomatedTester]
- present+
- 00:32:00 [foolip]
- present+
- 00:32:04 [reillyg]
- present+
- 00:32:06 [zcorpan]
- zcorpan has joined #testing
- 00:32:10 [joyee]
- present+
- 00:32:19 [lukebjerring]
- present+
- 00:32:19 [JohnJansen]
- jgraham: that's how you do it.
- 00:32:24 [zcorpan]
- https://bocoup.com/blog/how-to-scribe-at-tpac
- 00:32:34 [JohnChen]
- present+
- 00:32:44 [zghadyali_]
- zghadyali_ has joined #testing
- 00:33:16 [zghadyali_]
- present+
- 00:33:20 [zcorpan]
- present+ Simon Pieters, Bocoup
- 00:33:22 [Zakim]
- Zakim has joined #testing
- 00:33:25 [dxie_]
- dxie_ has joined #testing
- 00:33:26 [AutomatedTester]
- present+ David Burns, Mozilla
- 00:33:39 [Hexcles]
- RRSAgent: halp!
- 00:33:39 [RRSAgent]
- I'm logging. I don't understand 'halp!', Hexcles. Try /msg RRSAgent help
- 00:33:39 [foolip]
- present+ Philip Jägenstedt, Google
- 00:33:44 [ato]
- present+ Andreas Tolfsen (Mozilla)
- 00:33:47 [dxie_]
- present+, Daniel Xie, Google
- 00:34:07 [Hexcles]
- present+ Robert Ma (Google)
- 00:34:09 [zghadyali_]
- present+ Zoher Ghadyali, Microsoft
- 00:34:25 [foolip]
- scribenick: foolip
- 00:34:44 [foolip]
- lukebjerring: we have an Agenda in a doc, will work on that after first break
- 00:34:50 [JohnChen]
- present+ John Chen, Google
- 00:34:52 [joyee]
- present+, Qiuyi(Joyee) Zhang, Igalia
- 00:34:57 [foolip]
- lukebjerring: first status updates: https://docs.google.com/presentation/d/10pP5UdurCE3_5YMk6ds8ksz5GIj_Bxixfi9GJP2LOkw/edit?usp=sharing
- 00:35:28 [yigu]
- yigu has joined #testing
- 00:37:32 [foolip]
- lukebjerring: the dents in the Safari graph are infrastructure issues that affected reults
- 00:37:46 [MikeSmith]
- present+
- 00:38:41 [MikeSmith]
- RRSAgent, make minutes
- 00:38:41 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 00:38:48 [MikeSmith]
- agenda?
- 00:39:18 [foolip]
- jgraham: big change for Firefox is rewrite of encodings
- 00:39:41 [ato]
- RRSAgent: make minutes
- 00:39:41 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 00:40:51 [foolip]
- lukebjerring: browser-specific failures and passes are interesting for interoperability
- 00:40:58 [MikeSmith]
- Present- Bocoup
- 00:41:06 [MikeSmith]
- Present- Mozilla
- 00:41:18 [MikeSmith]
- Present- Google
- 00:41:33 [MikeSmith]
- Present- Andreas
- 00:41:39 [MikeSmith]
- Present- Tolfsen
- 00:41:40 [yigu]
- yigu has joined #testing
- 00:41:47 [MikeSmith]
- Present- (Mozilla)
- 00:41:54 [MikeSmith]
- Present- (Google)
- 00:42:01 [MikeSmith]
- Present- Microsoft
- 00:42:09 [MikeSmith]
- Present- Igalia
- 00:42:24 [MikeSmith]
- RRSAgent, make minutes
- 00:42:24 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 00:42:52 [foolip]
- jugglinmike1: yes!
- 00:43:06 [foolip]
- jugglinmike1: meet.google.com/xyo-wzse-nss
- 00:43:15 [Orphis]
- Orphis has joined #testing
- 00:43:22 [foolip]
- jugglinmike1: I thought you'd be out for the day
- 00:43:45 [MikeSmith]
- present+ gsnedders
- 00:43:57 [foolip]
- example of is:different: https://wpt.fyi/results/?label=master&label=experimental&aligned&q=is%3Adifferent
- 00:44:03 [kevers]
- kevers has joined #testing
- 00:44:09 [gsnedders]
- Present+
- 00:44:17 [MikeSmith]
- jugglinmike1, I can webrtc you in
- 00:44:22 [foolip]
- example of the `all` query: https://wpt.fyi/results/?label=master&label=experimental&aligned&q=all%28status%3Aerror%29
- 00:44:54 [yigu]
- present+ Yi Gu, Google
- 00:45:05 [foolip]
- jugglinmike1: can you join? I'll have to click OK
- 00:45:21 [foolip]
- jugglinmike1: good?
- 00:45:40 [MikeSmith]
- s/jugglinmike1:/jugglinmike1,/
- 00:45:46 [MikeSmith]
- RRSAgent, make minutes
- 00:45:46 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 00:46:29 [kevers]
- present
- 00:46:55 [foolip]
- Hexcles: I'll do a live demo now!
- 00:47:04 [Orphis]
- Orphis has joined #testing
- 00:47:30 [MikeSmith]
- present+ Yi_Gu
- 00:47:38 [MikeSmith]
- Present- Google
- 00:47:42 [MikeSmith]
- RRSAgent, make minutes
- 00:47:42 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 00:48:19 [MikeSmith]
- Present- Ma
- 00:48:24 [kevers]
- present+ Kevin Ellis
- 00:48:39 [MikeSmith]
- present- Robert
- 00:48:49 [ato]
- Title: Web Platform Tests, Day 1, TPAC 2019
- 00:48:57 [MikeSmith]
- present+ Robert_Ma
- 00:49:09 [MikeSmith]
- present- Kevin_Ellis
- 00:49:15 [MikeSmith]
- present+ Kevin_Ellis
- 00:49:21 [MikeSmith]
- present- Kevin
- 00:49:28 [MikeSmith]
- present- Ellis
- 00:50:23 [MikeSmith]
- s/jugglinmike1: can you join? I'll have to click OK/
- 00:50:32 [MikeSmith]
- s/jugglinmike1, good?/
- 00:50:50 [MikeSmith]
- RRSAgent, make minutes
- 00:50:50 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 00:53:01 [MikeSmith]
- i/we have an Agenda/Topic: Intro from Luke Bjerring
- 00:53:19 [ato]
- Meeting: Web Platform Tests, Day 1, TPAC 2019
- 00:53:27 [ato]
- RRSAgent: make minutes
- 00:53:27 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 00:53:45 [MikeSmith]
- i/live demo/Topic: Live demo from Robert Ma
- 00:53:48 [MikeSmith]
- RRSAgent, make minutes
- 00:53:48 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 00:55:17 [ato]
- RRSAgent: agenda?
- 00:55:17 [RRSAgent]
- I'm logging. Sorry, nothing found for 'agenda'
- 00:57:52 [foolip]
- jgraham: Taskcluster is going to split into separate instances for Gecko CI and rest of world, will give us new URLs and new UI
- 00:58:08 [foolip]
- jgraham: hope to do Android x86 emulator on Taskcluster by end of year
- 00:59:00 [foolip]
- jgraham: want to run Chrome and Firefox. much more capacity limits than desktop
- 00:59:07 [foolip]
- jgraham: will probably start with daily runs
- 00:59:35 [foolip]
- foolip: also trying to get WebKitGTK running
- 01:00:20 [foolip]
- jgraham: trying to add decision task for Taskcluster, would allow us to schedule dependent tasks
- 01:00:33 [foolip]
- jgraham: for example, we can avoid getting different versions of Firefox
- 01:00:49 [foolip]
- jgraham: or could have a build job
- 01:02:21 [foolip]
- JohnJansen: Edge is now a Chromium browser
- 01:02:43 [foolip]
- JohnJansen: have looked to take advantage of wpt.fyi. 99.9% results are the same a Chrome, so differences are interesting
- 01:03:04 [foolip]
- JohnJansen: we have found bugs (regressions) where Edge is different to Chrome
- 01:03:18 [foolip]
- JohnJansen: we aim to be 24 hours behind Chrome
- 01:03:32 [foolip]
- JohnJansen: but we also have different features that can cause tests to break. usually turns into blocking bugs
- 01:03:39 [foolip]
- JohnJansen: really appreciate the interop view
- 01:04:01 [foolip]
- JohnJansen: when will Edge stable be released? we're bug driven, not date driven
- 01:04:18 [foolip]
- JohnJansen: beta felt very stable to me, very impressed by quality of Chromium out of the box.
- 01:04:45 [karl]
- karl has joined #testing
- 01:04:59 [foolip]
- JohnJansen: wpt.fyi has helped us immensely. interesting to see the pass rate increase so much, now differences are really important
- 01:05:20 [wanming]
- wanming has joined #testing
- 01:05:34 [foolip]
- https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=edge&aligned&q=is%3Adifferent is a view that might be helpful
- 01:06:04 [jugglinmike1]
- foolip: I can talk docs
- 01:07:11 [MikeSmith]
- agenda: https://docs.google.com/document/d/1_d2xUBgNn6nmiIXM6m9zSLjzYhvDS4LMrmgoxMQJKuU/edit#
- 01:08:10 [JohnJansen]
- https://wpt.fyi/interop/IndexedDB/idbobjectstore_createIndex15-autoincrement.htm?label=master&label=experimental&product=chrome&product=edge&aligned&diff&filter=ADC
- 01:08:21 [foolip]
- jugglinmike1: we've been working to improve the docs over the past year
- 01:09:33 [Orphis]
- Orphis has joined #testing
- 01:10:06 [foolip]
- jugglinmike1: we've switched for GitHub pages (Jekyll) to Sphinx, a Python project
- 01:11:02 [yigu]
- See Mike's screen for the figure here: https://meet.google.com/xyo-wzse-nss
- 01:11:50 [zcorpan]
- example search result https://web-platform-tests.org/search.html?q=assert_throws&check_keywords=yes&area=default
- 01:12:21 [foolip]
- jgraham: thanks for improving the docs, it's a big improvement compared to what we had
- 01:12:27 [jugglinmike1]
- Docs: "2019 WPT Documentation Improvements" https://docs.google.com/document/d/16KJbWVRtIjZQX80CQhOvwG2RHlIqVM1ADQTG1Q-QdlE/edit
- 01:12:34 [MikeSmith]
- RRSAgent, make minutes
- 01:12:34 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 01:13:05 [foolip]
- jgraham: Another update. We've now moved from Travis to GitHub Actions. Somewhat mixed results, some spurious failures. Mostly things that require a github token, because the secrets handling is easier
- 01:13:40 [plh]
- plh has joined #testing
- 01:13:58 [foolip]
- JohnJansen: some differences we see between Chrome and Edge might be because we run on Windows too.
- 01:14:18 [foolip]
- lukebjerring: let's break for food
- 01:14:30 [foolip]
- jgraham: let's be back here at 10:45
- 01:14:47 [Hexcles]
- (i.e. in 30min)
- 01:15:05 [Hexcles]
- Hexcles has joined #testing
- 01:17:56 [denis]
- denis has joined #testing
- 01:22:23 [plh_]
- plh_ has joined #testing
- 01:23:06 [flackr]
- flackr has joined #testing
- 01:24:00 [twisniewski]
- present+
- 01:24:12 [karl]
- present+
- 01:24:22 [miketaylr]
- present+ Mike Taylor
- 01:24:55 [miketaylr]
- RRSAgent, make minutes
- 01:24:55 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html miketaylr
- 01:25:20 [miketaylr]
- present- Mike
- 01:25:22 [miketaylr]
- present- Taylor
- 01:25:24 [miketaylr]
- present+
- 01:25:26 [miketaylr]
- RRSAgent, make minutes
- 01:25:26 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html miketaylr
- 01:29:33 [Orphis]
- Orphis has joined #testing
- 01:30:49 [plh]
- plh has joined #testing
- 01:33:54 [tidoust]
- tidoust has joined #testing
- 01:34:11 [Orphis]
- Orphis has joined #testing
- 01:36:23 [Orphis]
- Orphis has joined #testing
- 01:39:32 [karl]
- karl has joined #testing
- 01:43:22 [estes]
- estes has joined #testing
- 01:44:38 [Orphis]
- Orphis has joined #testing
- 01:46:10 [Hexcles]
- Hexcles has joined #testing
- 01:47:26 [kevers]
- kevers has joined #testing
- 01:51:21 [JohnJansen]
- Meeting: Web-Platform-Tests TPAC 2019
- 01:51:36 [JohnJansen]
- RRSAgent, make minutes
- 01:51:36 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen
- 01:55:47 [zghadyali]
- zghadyali has joined #testing
- 01:56:00 [JohnChen]
- JohnChen has joined #testing
- 01:57:32 [zcorpan]
- zcorpan has joined #testing
- 01:58:26 [JohnJansen]
- back. intros...
- 02:01:42 [foolip]
- JohnJansen: if you add `(edge:!pass&edge:!ok)` to the diff query you can see the stuff that's more likely a problem
- 02:02:37 [karlcow_]
- karlcow_ has joined #testing
- 02:03:42 [jorydotcom]
- jorydotcom has joined #testing
- 02:03:52 [jorydotcom]
- +present Jory Burson, Bocoup
- 02:04:04 [spectranaut]
- spectranaut has joined #testing
- 02:04:08 [jorydotcom]
- @jugglinmike1 sleep tight :D
- 02:04:20 [jorydotcom]
- lol
- 02:04:25 [spectranaut]
- present+ Valerie Young, Bocoup
- 02:04:47 [jorydotcom]
- present+ Jory Burson, Bocoup
- 02:05:59 [marcosc]
- marcosc has joined #testing
- 02:06:54 [gsnedders]
- so who have we agreed CSS WG joint meeting with on the CSS WG side?
- 02:13:05 [MikeSmith]
- RRSAgent, make minutes
- 02:13:05 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 02:13:50 [MikeSmith]
- present- Bocoup
- 02:14:03 [MikeSmith]
- RRSAgent, make minutes
- 02:14:03 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 02:14:29 [foolip]
- We're now doing Agenda smithing
- 02:15:20 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19071 merge into 07master: Port two webkit-xxx-interpolation.html to wpt/ - https://git.io/JeOJr
- 02:15:53 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19072 merge into 07master: Delete three webkit-xxx-interpolation.html - https://git.io/JeOJK
- 02:16:00 [jugglinmike]
- jugglinmike has joined #testing
- 02:17:03 [gsnedders]
- because https://wiki.csswg.org/planning/tpac-2019#tuesday doesn't have any joint meeting with us there?
- 02:21:58 [plh_]
- plh_ has joined #testing
- 02:23:46 [JayZhou]
- JayZhou has joined #testing
- 02:31:44 [yigu_]
- yigu_ has joined #testing
- 02:34:41 [xiaoqian]
- xiaoqian has joined #testing
- 02:36:22 [MikeSmith]
- https://www.w3.org/wiki/TPAC/2019/SessionIdeas
- 02:36:34 [JohnJansen]
- https://w3c.github.io/tpac-breakouts/sessions.html
- 02:39:38 [foolip]
- scribenick: foolip
- 02:39:43 [foolip]
- Review of 2019 priorities from last TPAC starting
- 02:39:57 [foolip]
- https://docs.google.com/document/d/1UE2KB7gvaEw5gvp4aAQNS9TrFYmnzgDzh5Sr4LQqaQo/edit?usp=sharing
- 02:40:38 [JohnJansen]
- TOPIC: Review of 2019 priorities from last TPAC starting
- 02:40:44 [zcorpan]
- scribenick: zcorpan
- 02:42:48 [zcorpan]
- zcorpan: documentation has been worked on
- 02:42:56 [zcorpan]
- lukebjerring: i like the improved docs!
- 02:43:14 [zcorpan]
- AutomatedTester: improve debugging, this has to do with reftests
- 02:43:24 [zcorpan]
- AutomatedTester: improving tooling about getting debugging info out
- 02:43:31 [zcorpan]
- jgraham: duplicate with the later bullet point
- 02:44:06 [zcorpan]
- JohnJansen: last year i couldn't figure out how to debug a test in python
- 02:45:25 [zcorpan]
- ato: multiprocess debugging in python is fundamental limitation
- 02:45:37 [zcorpan]
- JohnJansen: nobody knows how to debug a test running through wptrunner?
- 02:45:43 [zcorpan]
- jgraham: in wdspec test?
- 02:45:45 [spectranaut]
- spectranaut has joined #testing
- 02:45:46 [zcorpan]
- JohnJansen: yes
- 02:45:51 [zcorpan]
- jgraham: i don' tknow how to do that
- 02:46:01 [zcorpan]
- foolip: printf()
- 02:46:17 [zcorpan]
- jgraham: dunno if something spins up a new python process
- 02:46:19 [zcorpan]
- gsnedders: yes
- 02:46:40 [zcorpan]
- jgraham: maybe pause before the process starts and get the process id
- 02:46:44 [MikeSmith]
- RRSAgent, make minutes
- 02:46:44 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 02:47:09 [zcorpan]
- JohnJansen: reftests have clearly improved in wpt.fyi
- 02:47:29 [zcorpan]
- ato: pytest subprocess
- 02:47:40 [zcorpan]
- ato: command line flag to something something
- 02:48:12 [zcorpan]
- jgraham: the wdspec case is really one process, other things is waiting on it
- 02:48:26 [zcorpan]
- jgraham: if waiting time is indefinite, which i think we can, then it could work
- 02:48:46 [zcorpan]
- ato: also webdriver related timeouts you need to tweak
- 02:48:56 [zcorpan]
- jgraham: for other test types we try to do that
- 02:49:11 [zcorpan]
- jgraham: disabling timeouts when running gdb debugger
- 02:49:32 [zcorpan]
- ato: we've set timeout multiplier to something very high
- 02:49:43 [zcorpan]
- lukebjerring: feature request?
- 02:49:48 [zcorpan]
- JohnJansen: i'm logging one
- 02:50:04 [JohnJansen]
- RRSAgent, make minutes
- 02:50:04 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen
- 02:50:07 [zcorpan]
- foolip: async_cleanup is major thing that happened
- 02:50:29 [zcorpan]
- foolip: how many have used reftests?
- 02:50:33 [zcorpan]
- (show of hands)
- 02:51:42 [BitBot]
- (14wpt) [issue] thejohnjansen opened 13#19073: We need a way to attach a debugger to the executing test when run from wptrunner - https://git.io/JeOUk
- 02:51:58 [zcorpan]
- gsnedders: "has the assignee reviewed" is a metric we can check
- 02:52:01 [JohnJansen]
- I logged an issue for debugging tests: https://github.com/web-platform-tests/wpt/issues/19073
- 02:52:14 [zcorpan]
- foolip: i set a filter to prio PRs where i'm assigned
- 02:52:32 [Hexcles]
- https://github-health.appspot.com/
- 02:52:49 [zcorpan]
- lukebjerring: for people who use github-health this helps
- 02:53:15 [zcorpan]
- jgraham: interesting to see if it's working for ppl who are not on infrastructure
- 02:53:37 [ato]
- What Mozilla uses: https://addons.mozilla.org/en-US/firefox/addon/myqonly/
- 02:53:52 [zcorpan]
- foolip: i look at things i'm assigned to
- 02:53:58 [zcorpan]
- zcorpan: me too, or reassign
- 02:54:01 [ato]
- Shows a notification in your browser for GitHub items you are assigned to.
- 02:54:08 [zcorpan]
- foolip: ideas for wpt-pr-bot?
- 02:54:21 [zcorpan]
- jgraham: improved a bit, but hasn't shifted a lot
- 02:55:02 [zcorpan]
- gsnedders: how many of the 250 PRs from 2017 are still open?
- 02:55:41 [zcorpan]
- gsnedders: we need people who work on this
- 02:55:46 [zcorpan]
- gsnedders: we've tried for years
- 02:56:54 [zcorpan]
- foolip: teaching people how to do email filtering works
- 02:57:35 [foolip]
- zcorpan: we know that new contributors don't have a good time
- 02:57:42 [AutomatedTester]
- scribenick: foolip
- 02:57:42 [foolip]
- zcorpan: their PRs are stuck for years
- 02:57:55 [foolip]
- zcorpan: do we want their contributions from people who don't already work on browser engines?
- 02:58:20 [foolip]
- zcorpan: if the answer is no we shouldn't pretend that we accept PRs, but if we do we should allocate time to do the review
- 02:59:09 [gsnedders]
- s/work on this/work on the web features in question to review them/
- 02:59:20 [foolip]
- jgraham: the problem is who are the we who should do the reviews?
- 02:59:43 [foolip]
- jgraham: ownership is fine-grained and we don't have a way to enforce that people do their job
- 03:00:12 [foolip]
- zcorpan: we can explore incentives to do reviews, like celebrate those who do the most reviews
- 03:00:22 [jorydotcom]
- +1
- 03:02:40 [tidoust]
- tidoust has joined #testing
- 03:04:17 [zcorpan]
- jgraham: it might be not worthwhile in the moment, but in the long run it may be
- 03:04:42 [zcorpan]
- jgraham: choice between possible impact over a long period vs definite impact over short period
- 03:05:02 [zcorpan]
- ato: are we sure that a stale review has been considered
- 03:05:12 [zcorpan]
- ato: if nobody has looked at it, how do we know if it's important
- 03:05:24 [zcorpan]
- miketaylr: has anyone retriaged?
- 03:05:39 [zcorpan]
- miketaylr: might be 50 out of the 440 that are amazing
- 03:05:59 [zcorpan]
- jgraham: i think we classify things in terms of their status (waiting for review, waiting for OP)
- 03:06:14 [zcorpan]
- jgraham: don't classify by importance, size
- 03:06:47 [zcorpan]
- miketaylr: if i say to other person at moz that 50 tests are important, it can be prioritized
- 03:06:58 [zcorpan]
- jgraham: if we can sort things by stuff that show browser issues
- 03:07:08 [zcorpan]
- jgraham: that is valuable
- 03:07:14 [zcorpan]
- foolip: come back to this?
- 03:07:17 [zcorpan]
- jgraham: yes
- 03:07:24 [tidoust]
- tidoust has joined #testing
- 03:08:03 [zcorpan]
- jgraham: next bullet point: beginners onboarding
- 03:08:14 [zcorpan]
- foolip: more to say that docs?
- 03:08:38 [zcorpan]
- jgraham: mdn survey
- 03:09:08 [zcorpan]
- foolip: i think that survey can be helpful to prioritize our work
- 03:09:27 [zcorpan]
- jgraham: "make CI more robust"
- 03:09:30 [zcorpan]
- foolip: i think we did
- 03:09:48 [zcorpan]
- foolip: reliability question, can talk about PRs in next session
- 03:09:59 [zcorpan]
- foolip: running more, getting more complete results
- 03:10:04 [zcorpan]
- jgraham: problems with safari
- 03:10:05 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot 04closed 13#18999: Revert "Reland "Started implementing the STAPIT algorithm"" - https://git.io/Jemrw
- 03:10:27 [zcorpan]
- foolip: github actoins acting up, problems for PRs
- 03:10:45 [zcorpan]
- foolip: PR results and full run are better now than 1 year ago
- 03:10:58 [zcorpan]
- jgraham: "give web platform engineers the tools they need to prio"
- 03:11:21 [zcorpan]
- jgraham: is done or in progress
- 03:11:28 [zcorpan]
- gsnedders: don't have bug links
- 03:11:32 [zcorpan]
- jgraham: so that's in progress
- 03:12:17 [zcorpan]
- lukebjerring: every bug that i triage that has a crbug is fixed or explanation why it won't be in the short term
- 03:12:28 [zcorpan]
- lukebjerring: made some progress there
- 03:12:54 [zcorpan]
- foolip: would you encourage others to do the same?
- 03:12:56 [zcorpan]
- lukebjerring: yes
- 03:13:13 [foolip]
- lukebjerring: https://wpt.fyi/insights
- 03:13:23 [zcorpan]
- lukebjerring: double down on what is an important failure, not yet really solved
- 03:13:37 [Hexcles]
- lukebjerring: RFC https://github.com/web-platform-tests/rfcs/pull/16 will move the needle more significantly
- 03:13:50 [zcorpan]
- jgraham: we've looked at firefox-only failures
- 03:14:08 [zcorpan]
- jgraham: triage that but resistance is always "ok these things fail but are they relevant to compat?"
- 03:14:21 [zcorpan]
- jgraham: we know if we fix compat bugs they often end up with new tests
- 03:14:32 [zcorpan]
- jgraham: don't have the reverse: does fixing a test fix web compat
- 03:15:10 [zcorpan]
- jgraham: data exists that building up internal understanding of when it's important
- 03:15:27 [zcorpan]
- jgraham: things that are different between firefox and safari removes some objections
- 03:15:46 [zcorpan]
- jgraham: or chrome and safari. firefox-specific failures
- 03:16:13 [zcorpan]
- jgraham: have better ways to tell if a failure is important, is valuable to us
- 03:16:35 [zcorpan]
- jgraham: we still don't understand how given a test failure impacts web compat
- 03:17:12 [zcorpan]
- JohnJansen: if we reduce a site bug and fix that, then track back to which tests now pass
- 03:17:16 [zcorpan]
- JohnJansen: but not reverse
- 03:17:35 [zcorpan]
- JohnJansen: interaction between features also
- 03:17:57 [zcorpan]
- jgraham: we could test that, but don't test all possible intersections
- 03:18:33 [zcorpan]
- jgraham: reducing things that go into the suite
- 03:18:51 [zcorpan]
- foolip: beyond harness errors, can we talk about flaky tests?
- 03:18:56 [zcorpan]
- jgraham: infra has improved
- 03:19:17 [denis_]
- denis_ has joined #testing
- 03:20:23 [zcorpan]
- jgraham: some ideas that we havent' followed up
- 03:20:58 [zcorpan]
- jgraham: if we can get to a situation where a library works around a browser bug, if they also file a browser bug
- 03:21:03 [zcorpan]
- jgraham: would be impactful
- 03:21:18 [zcorpan]
- foolip: triage metadata and bug linking, if there's labeling and start counts
- 03:21:52 [zcorpan]
- foolip: test to spec linking. bikeshed and respec have something!
- 03:23:40 [zcorpan]
- jgraham: progress, but relevant people aren't in the room
- 03:23:47 [zcorpan]
- jgraham: test coverage
- 03:24:13 [zcorpan]
- Hexcles: in blink we have coverage comparison between wpt and legacy layout tests
- 03:24:20 [zcorpan]
- Hexcles: results are pretty good overall
- 03:24:41 [zcorpan]
- lukebjerring: 10% difference between running all tests (wpt+layout) and only running wpt
- 03:25:06 [zcorpan]
- lukebjerring: identify low hanging fruit to upstream tests to wpt
- 03:25:08 [Hexcles]
- https://storage.googleapis.com/blink-wpt-coverage/201812/index.html
- 03:25:34 [zcorpan]
- Hexcles: we can easily measure coverage data
- 03:25:49 [zcorpan]
- Hexcles: i think there's effort to collect data on ongoing basis
- 03:25:59 [whsieh]
- whsieh has joined #testing
- 03:26:03 [zcorpan]
- JohnJansen: region coverage?
- 03:26:09 [zcorpan]
- JohnJansen: what does it mean?
- 03:26:16 [zcorpan]
- gsnedders: basic blocks
- 03:27:07 [zcorpan]
- jgraham: for gecko we have a coverage metric
- 03:27:12 [foolip]
- Tootip is "Region coverage is the percentage of code regions which have been executed at least once. A code region may span multiple lines (e.g in a large function body with no control flow). However, it's also possible for a single line to contain multiple code regions (e.g in 'return x || y && z')."
- 03:27:18 [zcorpan]
- jgraham: daily coverage runs maybe
- 03:27:34 [zcorpan]
- jgraham: can see per testsuite
- 03:27:41 [ato]
- RRSAgent: make minutes, please
- 03:27:41 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 03:27:44 [zcorpan]
- jgraham: the UI doesn't yet display diffs between testsuites
- 03:27:53 [zcorpan]
- jgraham: I have some of that data for gecko
- 03:28:04 [ato]
- ScribeNick: zcorpan
- 03:28:05 [zcorpan]
- jgraham: some bits are better covered by mochitests, like gamepad
- 03:28:18 [zcorpan]
- jgraham: some areas where wpt has better coverage
- 03:28:19 [ato]
- RRSAgent: make minutes
- 03:28:19 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 03:28:37 [zcorpan]
- Hexcles: one thing stood out
- 03:28:44 [zcorpan]
- Hexcles: accessibility is poorly tested in wpt
- 03:28:48 [zcorpan]
- gsnedders: not possible to test
- 03:29:09 [zcorpan]
- jgraham: coverage is a way to identify places where wpt is weak
- 03:29:20 [zcorpan]
- jgraham: might not always be fixable (like maybe GC)
- 03:29:35 [zcorpan]
- jgraham: gamepad or accessibility should be testable
- 03:29:52 [zcorpan]
- foolip: that's test automation
- 03:30:07 [zcorpan]
- foolip: has coverage improved?
- 03:30:17 [_11]
- _11 has joined #testing
- 03:30:33 [zcorpan]
- MikeSmith: how many tests do we have that are using ???
- 03:30:44 [zcorpan]
- MikeSmith: cases manual tests are converted
- 03:31:05 [zcorpan]
- MikeSmith: incentive to groups to automate manual tests
- 03:31:16 [zcorpan]
- MikeSmith: awareness, people don't know this is available
- 03:31:23 [Hexcles]
- s/???/testdriver.js
- 03:31:28 [zcorpan]
- foolip: 400 files testdriver.js
- 03:32:08 [zcorpan]
- jgraham: automating stuff that we can't currently automate... some success
- 03:32:19 [zcorpan]
- ato: in terms of testability extensions of webdriver api
- 03:32:30 [zcorpan]
- ato: we've seen other specs e.g. permissions, write extensions
- 03:32:43 [zcorpan]
- ato: but no implementations
- 03:32:58 [zcorpan]
- foolip: generate test report has been implemented
- 03:33:29 [zcorpan]
- jgraham: gecko hasn't implemented that yet
- 03:33:51 [zcorpan]
- foolip: ~200 files testdriver.js a year ago
- 03:34:14 [zcorpan]
- foolip: pointer events, painful?
- 03:34:34 [zcorpan]
- NavidZ_: let anyone add switch to testdriver protocol
- 03:34:42 [zcorpan]
- NavidZ_: gives user activation to the page
- 03:34:51 [zcorpan]
- NavidZ_: nobody can do that
- 03:35:05 [zcorpan]
- NavidZ_: one way to ask the test author to click or something
- 03:35:19 [zcorpan]
- NavidZ_: question is, how much of this do we want to expose to the testers
- 03:35:32 [zcorpan]
- NavidZ_: not testing parts of the browser
- 03:35:42 [zcorpan]
- NavidZ_: some apis already that can expose that
- 03:35:53 [zcorpan]
- NavidZ_: if we want to go down that path, do it all the way
- 03:36:03 [zcorpan]
- NavidZ_: on windows, behaves differently than linux on mac
- 03:36:22 [zcorpan]
- NavidZ_: chromium tests only, test can choose based on platform
- 03:36:32 [zcorpan]
- NavidZ_: do we want to expose that in testdriver?
- 03:36:51 [zcorpan]
- NavidZ_: focus next: can't get that without user interaction. difference between platforms
- 03:37:31 [zcorpan]
- NavidZ_: adding automation for specific things, expose more and more of inner workings of the browser
- 03:37:49 [zcorpan]
- foolip: do we want to add API for create user interaction defined in html?
- 03:37:58 [zcorpan]
- ato: big discussion in browser testing and tools
- 03:38:14 [Hexcles]
- Hexcles has joined #testing
- 03:38:18 [zcorpan]
- <lunch>
- 03:38:31 [zcorpan]
- s/<lunch>/<std-lunch>/
- 03:40:03 [zcorpan]
- zcorpan has joined #testing
- 04:01:30 [cybai]
- cybai has joined #testing
- 04:02:46 [spectranaut]
- spectranaut has joined #testing
- 04:06:13 [denis]
- denis has joined #testing
- 04:07:07 [cybai_]
- cybai_ has joined #testing
- 04:12:36 [cybai]
- cybai has joined #testing
- 04:14:50 [Hexcles]
- Hexcles has joined #testing
- 04:20:21 [kevers]
- kevers has joined #testing
- 04:24:08 [BitBot]
- (14wpt) [PR] dirkschulze requested 13#19074 merge into 07master: Add smfr as reviewer for CSS Transforms - https://git.io/JeOTN
- 04:24:11 [denis]
- denis has joined #testing
- 04:24:30 [prushforth]
- prushforth has joined #testing
- 04:25:03 [prushforth]
- present+ Peter Rushforth, Natural Resources Canada, Observer
- 04:30:56 [whsieh]
- whsieh has joined #testing
- 04:31:02 [tidoust]
- tidoust has joined #testing
- 04:34:28 [zcorpan]
- zcorpan has joined #testing
- 04:40:23 [BitBot]
- (14wpt) [PR] dirkschulze 03merged 13#19074 into 07master: Add smfr as reviewer for CSS Transforms - https://git.io/JeOTN
- 04:42:45 [zcorpan]
- RRSAgent: make minutes
- 04:42:45 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html zcorpan
- 04:45:11 [BitBot]
- (14wpt) [PR] moz-wptsync-bot requested 13#19075 merge into 07master: [Gecko Bug 1579993] Add WPT subgrid tests and a few regular Grid baseline alignment tests. - https://git.io/JeOkz
- 04:45:34 [marcosc]
- marcosc has joined #testing
- 04:45:57 [zcorpan]
- participants & their position to help with scribing https://docs.google.com/spreadsheets/d/1cqPK6ze2OCLsho4twJHNLZUPktfejIiiDlMwv0TaZBg/edit#gid=0
- 04:46:46 [Hexcles]
- Hexcles has joined #testing
- 04:51:20 [zghadyali]
- zghadyali has joined #Testing
- 04:56:20 [plh]
- plh has joined #testing
- 05:00:41 [cybai]
- cybai has joined #testing
- 05:01:54 [Hexcles]
- Hexcles has joined #testing
- 05:04:52 [saschanaz]
- saschanaz has joined #testing
- 05:07:40 [ato]
- RRSAgent: make minutes, please
- 05:07:40 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 05:07:41 [jorydotcom]
- jorydotcom has joined #testing
- 05:08:25 [marcosc]
- marcosc has joined #testing
- 05:09:29 [foolip]
- Just added https://www.w3.org/wiki/TPAC/2019/SessionIdeas#web-platform-tests_update_.26_discussion
- 05:09:29 [denis_]
- denis_ has joined #testing
- 05:10:11 [CalebRouleau]
- CalebRouleau has joined #testing
- 05:10:23 [CalebRouleau]
- present+ Caleb Rouleau, Google
- 05:10:55 [JohnJansen]
- present+
- 05:10:57 [ato]
- ScribeNick: ato
- 05:11:15 [cb]
- cb has joined #testing
- 05:11:34 [zcorpan]
- present- Google
- 05:11:40 [ato]
- Topic: Infra: making full use of the test results on PRs
- 05:11:55 [ato]
- foolip: PR checks are a bit noisy, so at the moment you want to ignore them. Fourteen in total.
- 05:12:11 [ato]
- ... Flaky tests also. Should we do something about them?
- 05:12:25 [ato]
- jgraham: Let's talk about what is there already.
- 05:12:44 [foolip]
- Example: https://github.com/web-platform-tests/wpt/pull/19067
- 05:12:57 [MikeSmith]
- RRSAgent, make minutes
- 05:12:57 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 05:13:12 [whsieh]
- whsieh has joined #testing
- 05:13:19 [MikeSmith]
- present- Natural_Resources_Canada
- 05:13:28 [MikeSmith]
- present- Observer
- 05:14:37 [BitBot]
- (14wpt) [PR] dirkschulze 03merged 13#11169 into 07master: Remove tests for SVG transform with CSS syntax - part 1 - https://git.io/JeOk6
- 05:14:44 [ato]
- foolip: The Azure pipeline is noisy, just produces results for Safari.
- 05:14:53 [ato]
- foolip: This is a request for the Azure team, to make this less noisy.
- 05:15:08 [ato]
- JohnJansen: You'd like the report to be merged into one? All the pipelines into one?
- 05:15:26 [ato]
- gsnedders: In GitHub checks it makes sense for them to be separate?
- 05:15:38 [ato]
- foolip: Talking to Microsoft about this makes sense I think.
- 05:16:15 [ato]
- jgraham: With TaskCluster you have to click through to get the specific URL for the job.
- 05:16:26 [ato]
- ... This is not natural to everyone.
- 05:16:43 [ato]
- ... I think switching that to checks will make it noisy, but may make it easier to figure out what's failing.
- 05:17:07 [ato]
- ... A decision task would help also, because it would just run the tasks that are relevant.
- 05:17:08 [JohnJansen]
- ACTION: JohnJansen follow up with Azure Pipelines team for this
- 05:17:14 [ato]
- ... Rather than the tasks we have decided to run for you.
- 05:17:31 [ato]
- [talk about priorities]
- 05:18:17 [ato]
- jgraham: With a decision task [on Taskcluster] it will run only the things that are dependencies, such as lints when source file changes, but essentially just the test jobs related.
- 05:18:27 [cybai_]
- cybai_ has joined #testing
- 05:18:40 [ato]
- jgraham: We've run up quite hard against the limits of GitHub.
- 05:18:51 [ato]
- jgraham: Checks vs. non-blocking checks
- 05:19:40 [ato]
- foolip: Final part, wpt.fyi.
- 05:20:04 [ato]
- foolip: Sometimes there are more wpt.fyi non-blocking checks, due to deployment etc.
- 05:20:07 [cb_]
- cb_ has joined #testing
- 05:20:19 [ato]
- ... Could we merge this into a single check somehow?
- 05:20:35 [ato]
- lukebjerring: If it's desirable we could reuse the same name and aggregate the information.
- 05:21:00 [ato]
- lukebjerring: Pending results will tie into these, so we will have pending checks whilst the process [?] is still ongoing.
- 05:21:19 [ato]
- foolip: Sometimes I see the same problem in all three.
- 05:21:29 [ato]
- jgraham: I've never clicked on these that have neutral status.
- 05:21:38 [ato]
- jgraham: So I don't think it’s an effective signal to give users.
- 05:22:19 [Hexcles]
- s/process [?]/result processor/
- 05:22:36 [ato]
- ... Being able to look at the Firefox results specifically has been useful, but I don't think that the neutral results provide any value.
- 05:22:48 [ato]
- foolip: How do we indicate this in a better way? Make it fail?
- 05:23:07 [ato]
- jgraham: Failing with a button to "un-fail" it might be OK, but confusing.
- 05:23:15 [ato]
- ... Maybe submit an issue on the code, somewhere?
- 05:23:19 [ato]
- foolip: Review comment?
- 05:23:21 [karl]
- karl has joined #testing
- 05:23:39 [ato]
- jgraham: "This test appears to be erroring in this browser, if that’s fine you should dismiss this review and accept the PR."
- 05:24:19 [foolip]
- https://developer.github.com/v3/checks/runs/ has screenshot at top of possible outcomes
- 05:24:29 [ato]
- lukebjerring: If you click on the details for the wpt.fyi job, there’s a recompute button right now, and for the case that is neutral which I would hope to upgrade to blocking, it would add a comment saying which user clicked the button.
- 05:24:39 [ato]
- ... "Luke marked this as passing using the ignore button"
- 05:25:20 [ato]
- foolip: There are a few different options, a red triangle and provide text.
- 05:25:38 [ato]
- foolip: I suppose it wouldn’t be blocking then. We could decide not to make it blocking, I mean.
- 05:26:05 [ato]
- jgraham: I’m happy to experiment with stuff here, but I’m very cynical about how much people writing these tests are going to care to dig into the issues.
- 05:26:20 [ato]
- ... At least initially people will be whining because their PRs are stuck and don’t bother investigating.
- 05:26:28 [rego]
- rego has joined #testing
- 05:26:36 [ato]
- foolip: Low-frequency but serious solution we’re willing to experiment with?
- 05:26:48 [ato]
- jgraham: No apss everywhere.
- 05:27:27 [ato]
- foolip: Let's go with "consistent error" everywhere.
- 05:27:31 [ato]
- s/apss/pass/
- 05:27:53 [ato]
- foolip: A lot of tests that Chrome adds are harness errors in Safari.
- 05:27:54 [jesopo]
- jesopo has joined #testing
- 05:28:11 [ato]
- jgraham: Maybe it should be more cynical about things that come from browser syncs.
- 05:28:29 [ato]
- ... A test for an API that only exists in Chrome might be problematic.
- 05:28:34 [Hexcles]
- ACTION: lukebjerring Hexcles: explore setting wpt.fyi check results to failures when e.g. tests error everywhere
- 05:29:21 [ato]
- lukebjerring: It has to previously been failing and now passing at the moment, and I think it would be more useful to make it blocking than neutral.
- 05:30:01 [ato]
- lukebjerring: We have merged PRs that failed tests, but they were neutral in the PR and didn't check. We had to unroll a big list of changes.
- 05:30:17 [zcorpan]
- zcorpan has joined #testing
- 05:30:26 [ato]
- jgraham: There needs to be some mechanism whereby the author has to say that “this is what I intended”.
- 05:30:41 [ato]
- ... Pass going to fail happens often. The test could’ve been wrong.
- 05:30:58 [CalebRouleau]
- q+
- 05:31:05 [ato]
- ... The test might’ve been passing in all browsers, but is now always failing.
- 05:31:43 [ato]
- lukebjerring: If you fix a test that was passing incorrectly and becomes failing, you’ll be blocking becaus ethe statistics will be wrong [?]
- 05:32:04 [ato]
- gsnedders: It’s not the job failing within the Chromium CI system.
- 05:32:28 [ato]
- jgraham: People are less invested in gettnig their change landed in Chromium than they are in WPT.
- 05:32:38 [ato]
- jgraham: Or is your CL blocked until it lands in WPT?
- 05:32:46 [ato]
- lukebjerring: That’s a discussion we want to have today.
- 05:32:48 [JohnJansen]
- q?
- 05:33:00 [CalebRouleau]
- q-
- 05:33:11 [ato]
- lukebjerring: We want to propagate that into the original CL before it lands, before the export happens. This reduces human manual intervention.
- 05:33:31 [ato]
- jgraham: [explains Mozilla process]
- 05:34:29 [ato]
- lukebjerring: Forcing someone to click an extra button is worth it balanced against the pain it puts on us later.
- 05:34:38 [xiaoqian]
- xiaoqian has joined #testing
- 05:34:55 [ato]
- ... Because of the privilege that WPT is given, it is important for WPT not to land into a bad state.
- 05:35:16 [ato]
- jgraham: Occasionally we land something that fails all our tests.
- 05:35:29 [ato]
- ... Coming up with good heuristics is hard, but I agree it’s a problem.
- 05:35:40 [ato]
- ... The problem is that this is a system platform developrs don’t interact with often.
- 05:35:50 [CalebRouleau]
- q+
- 05:36:37 [foolip]
- q+
- 05:36:37 [ato]
- ato: Should upstream PRs be scrutinised harder?
- 05:37:04 [ato]
- jgraham: Maybe that case should have different heuristics.
- 05:37:29 [gsnedders]
- ack CalebRouleau
- 05:37:49 [ato]
- CalebRouleau: Whitelist of things that is supposed to be passing?
- 05:37:59 [ato]
- CalebRouleau: This would be a change in the code.
- 05:38:15 [ato]
- jgraham: The expectation metadata could be put in WPT perhaps.
- 05:38:22 [ato]
- ... But it would meet a lot of resistance.
- 05:38:26 [ato]
- Hexcles: I agree.
- 05:38:50 [ato]
- jgraham: It would be like for the infra tests. You would have to go and update the expectations when you mean for something to change.
- 05:39:00 [ato]
- jgraham: It would work, because that’s how we do it for Gecko.
- 05:39:09 [estes]
- estes has joined #testing
- 05:39:32 [ato]
- CalebRouleau: The proposal was _whitelist_, not expectation data for all.
- 05:39:55 [foolip]
- q-
- 05:40:06 [ato]
- lukebjerring: You’re going to end up exhaustively listing the metadata anyway, and you’ll want this per-browser basis.
- 05:40:31 [ato]
- jgraham: It would also increase the workload on authors because they would have to update the metadata also for other web browsers.
- 05:41:01 [ato]
- lukebjerring: Someone can submit something to the codebase and suddenly everything fails, without any warning.
- 05:41:37 [Hexcles]
- q?
- 05:41:39 [ato]
- jgraham: Human intervention needed at some level.
- 05:41:52 [ato]
- ... Different rules for different directories?
- 05:43:11 [ato]
- ato: It’s not unprecedented that tests are wrong and we make them go from pass to fail expecedtly.
- 05:43:39 [ato]
- jgraham: [explains a recent case]
- 05:45:10 [Hexcles]
- foolip: if you are heavily involved you can become a reviewer or even codeowner on github
- 05:45:57 [ato]
- lukebjerring: I think it’s reasonable to ask people to explain why they are making a test go from pass to fail.
- 05:46:48 [ato]
- ... It’s easy to demonstrate why we would impose a blocking check for it that people would understand.
- 05:47:01 [ato]
- jgraham: I would need to see statistics to see that it’s usually problematic [?]
- 05:47:51 [ato]
- ... Another case is where someone adds a test that passes only in a single browser, but times out or fails in all others.
- 05:48:06 [ato]
- lukebjerring: Do we make these a failing action on GitHub and allow it to be ignored?
- 05:48:19 [ato]
- ... And when a regression happens, they are propagated into the Firefox CI?
- 05:48:53 [ato]
- jgraham: We’re starting up a project so that we can hopefully surface this stuff as feedback in the code review, as opposed to finding out after a change lands.
- 05:49:03 [CalebRouleau]
- CalebRouleau has joined #testing
- 05:49:25 [ato]
- ... I think we can surface this stuff to developers earlier and get them to look at it more readily.
- 05:49:49 [ato]
- ... It’s unclear what the best mechanism for the feedback in WPT on GH should be.
- 05:50:37 [ato]
- ... If you choose to ignore you have to give a reason I think.
- 05:50:41 [ato]
- ... With checks you can't maybe.
- 05:51:15 [ato]
- foolip: The GH checks are made for this, they have a desired outcome and you have to give a reason.
- 05:51:21 [ato]
- foolip: That flow is sort of built for this.
- 05:51:33 [ato]
- jgraham: I’m unconvinced they have got the UI right.
- 05:51:38 [ato]
- foolip: Well we haven’t tried it out.
- 05:52:23 [ato]
- jgraham: OK, maybe we should try the checks thing first, otherwise try the code review thing.
- 05:52:38 [ato]
- Hexcles: We could have different rules for different directories also.
- 05:53:10 [ato]
- jgraham: If we could initially roll it out for directories with developers we know are engaged.
- 05:53:43 [ato]
- foolip: If we see the flow in a PR, maybe this is going to be easier to assess.
- 05:54:45 [ato]
- lukebjerring: I do a deliberate regression PR to wpt.fyi for this
- 05:55:04 [ato]
- RRSAgent: make minutes
- 05:55:04 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 05:56:25 [ato]
- RRSAgent: Make minutes
- 05:56:25 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 05:57:28 [Hexcles]
- ACTION foolip: write an RFC for making regressions detected by wpt.fyi require actions
- 05:57:40 [ato]
- foolip: Understanding the flakiness is super-hard.
- 05:58:07 [ato]
- foolip: We could make the logs less verbose, but then we’d have to increase it to find out what’s wrong.
- 05:58:22 [ato]
- jgraham: We have a log handler that picks out the things it thinks is important.
- 05:58:49 [hyojin]
- hyojin has joined #testing
- 05:59:26 [ato]
- lukebjerring: Custom interpretation jobs is what wpt.fyi is doing by definition.
- 05:59:55 [cb]
- cb has joined #testing
- 05:59:57 [Hexcles]
- RRSAgent make minutes
- 06:00:31 [ato]
- lukebjerring: Instead of designing TaskCluster to have custom log interpretation, we should do this in wpt.fyi.
- 06:01:07 [ato]
- jgraham: But if we had this in TaskCluster, we would use this consistently also for other things. It could produce an artifact we could reuse elsewhere.
- 06:02:14 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html gsnedders
- 06:03:53 [Hexcles]
- ACTION: Hexcles: switch Taskcluster to GitHub Checks
- 06:04:51 [ato]
- jgraham: [explains TaskCluster]
- 06:06:46 [ato]
- RRSAgent, draft minutes
- 06:06:46 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 06:07:47 [ato]
- jgraham: Regarding flakey tests, recently the expected test status at Mozilla have support for multiple test statuses.
- 06:07:59 [ato]
- ... For example, this test can either pass or fail.
- 06:08:19 [zghadyali]
- zghadyali has joined #testing
- 06:08:27 [ato]
- lukebjerring: What is the main reason you’re against having expectation data upstream in WPT?
- 06:08:48 [ato]
- jgraham: On WPT we’re running things mostly on one platform.
- 06:08:53 [ato]
- ... On Gecko we’re running a vast number of platforms which is specific.
- 06:09:25 [ato]
- ... This could cause double work because you would have to care for other browsers’ expectation data in addition to your own.
- 06:10:25 [ato]
- Hexcles: There's a more fundamental problem: browser versions matter.
- 06:10:43 [ato]
- Hexcles: If you have an upstream expectation that applies to a specific version, it will be even more difficult.
- 06:11:12 [JohnJansen]
- present+
- 06:13:02 [ato]
- lukebjerring: There's an implicit falkiness if you have multiple acceptable statuses, and having a totally separate place to say soemthing is flaky seems kind of bad because it duplicates information.
- 06:15:36 [ato]
- jgraham: If you had the WPT flakiness data in tree you could build some interesting tooling.
- 06:16:02 [ato]
- jgraham: "You’ve marked this test as not-flaky in Chrome, but it’s still flaky elsewhere.”
- 06:16:34 [ato]
- foolip: Flakiness on master vs. flakiness on PRs?
- 06:17:20 [cb]
- cb has joined #testing
- 06:17:26 [ato]
- lukebjerring: We run a cron job that looks at the last ten runs and checks for flaky tests and lets some person know about it.
- 06:18:08 [ato]
- jgraham: Can we have a checks page?
- 06:18:14 [ato]
- form on*
- 06:18:22 [ato]
- lukebjerring: Recompute, ignore
- 06:18:54 [ato]
- ... You could also have "flag as flaky" which would change the metadata and re-run the computation, say.
- 06:19:42 [ato]
- ... If you have a flaky test on TC and people aren’t bothered looking into it, they are already ignoring this and force merging.
- 06:19:49 [ato]
- ... So it would not make the current situation any worse.
- 06:20:11 [ato]
- foolip: I don’t think we necessarily have anything we disagree on with regards to flaky tests.
- 06:21:37 [ato]
- JohnJansen: Explanation on how to deal with flaky tests.
- 06:21:57 [cybai]
- cybai has joined #testing
- 06:22:06 [ato]
- lukebjerring: Documentation for “so you’ve been told your tests are flaky” sounds like a good idea.
- 06:22:27 [ato]
- Hexcles: A tutorial linked from the GH checks.
- 06:23:08 [ato]
- foolip: If we have a button that marks as flaky, you can be sure people are going to click it.
- 06:23:17 [ato]
- ... This will eventually make the system useless if overused.
- 06:23:31 [ato]
- lukebjerring: It’s hard to identify false-positives.
- 06:23:54 [ato]
- jgraham: If you haven’t seen flakiness in the last month, then we probably don’t care.
- 06:24:07 [ato]
- jgraham: You can remove expectation data when the flakiness goes away.
- 06:24:32 [cb]
- cb has joined #testing
- 06:25:02 [ato]
- jgraham: In Chromium you run every test on every commit, you can get backed out if the test becomes flaky as a result of the CL.
- 06:25:29 [ato]
- ... In the Gecko case it matters less if the metadata is a little bit inaccurate.
- 06:26:28 [ato]
- Topic: Python 3
- 06:26:53 [ato]
- jgraham: Python Foundation are stop going to stop maintaining Python 2 sometime next year.
- 06:27:07 [ato]
- ... There is no need for immediate panic because RedHat will continue providing updates for another four years.
- 06:27:20 [ato]
- ... But we should have a plan for migration to Python 3.
- 06:27:38 [ato]
- s/are stop/are/
- 06:27:58 [ato]
- ... Gecko is starting to move things to Python 3 slowly, and there is increased need for us to have a roadmap for this as well.
- 06:28:21 [ato]
- Hexcles: There was no conclusion what WebKit is going to do based on the email thread.
- 06:28:43 [ato]
- jgraham: Previously we had assumed that WebKit was a blocking concern.
- 06:29:12 [ato]
- ... But now it looks like they are switching to Python 3, or possibly going to stop shipping Python altogether.
- 06:29:18 [ato]
- Hexcles: macOS 10.11
- 06:29:30 [ato]
- Hexcles: We should find someone from the WebKit community.
- 06:29:33 [Hexcles]
- s/10.11/10.15
- 06:29:43 [ato]
- s/.11/10.15/
- 06:29:57 [ato]
- RRSAgent: make minutes
- 06:29:57 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 06:31:07 [ato]
- jgraham: We need support for both Python 2 and 3.
- 06:31:39 [ato]
- ... For example making the WPT frontend run in either, then make the commands it despatches to run Python 3.
- 06:32:10 [ato]
- ... That seems to be the way Gecko works.
- 06:32:29 [ato]
- Break.
- 06:33:44 [tidoust]
- tidoust has joined #testing
- 06:35:10 [Hexcles]
- Hexcles has joined #testing
- 06:40:23 [BitBot]
- (14wpt) [PR] autofoolip requested 13#19076 merge into 07master: Update interfaces/IndexedDB.idl - https://git.io/JeOIW
- 06:40:29 [BitBot]
- (14wpt) [PR] autofoolip requested 13#19077 merge into 07master: Update interfaces/gamepad.idl - https://git.io/JeOI8
- 06:40:33 [BitBot]
- (14wpt) [PR] autofoolip requested 13#19078 merge into 07master: Update interfaces/geometry.idl - https://git.io/JeOI4
- 06:40:37 [BitBot]
- (14wpt) [PR] autofoolip requested 13#19079 merge into 07master: Update interfaces/webmidi.idl - https://git.io/JeOIB
- 06:40:41 [BitBot]
- (14wpt) [PR] autofoolip requested 13#19080 merge into 07master: Update interfaces/webrtc-stats.idl - https://git.io/JeOIR
- 06:42:33 [plh]
- plh has joined #testing
- 06:43:45 [xiaoqian]
- xiaoqian has joined #testing
- 06:44:50 [spectranaut]
- spectranaut has joined #testing
- 06:46:06 [zcorpan]
- zcorpan has joined #testing
- 06:56:06 [cybai]
- cybai has joined #testing
- 07:02:10 [Manishearth]
- Manishearth has joined #testing
- 07:02:41 [Hexcles]
- Hexcles has joined #testing
- 07:04:13 [ato]
- Topic: WebXR
- 07:04:23 [ato]
- RRSAgent: make minutes
- 07:04:23 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 07:05:05 [foolip]
- A testing API exists: https://github.com/immersive-web/webxr-test-api
- 07:05:14 [CalebRouleau]
- CalebRouleau has joined #testing
- 07:05:57 [ato]
- RRSAgent: this meeting spans midnight
- 07:05:58 [karl]
- karl has joined #testing
- 07:05:59 [BitBot]
- (14wpt) [PR] foolip 03merged 13#19076 into 07master: Update interfaces/IndexedDB.idl - https://git.io/JeOIW
- 07:05:59 [Manishearth]
- present+
- 07:06:06 [ato]
- RRSAgent: listen
- 07:06:07 [JohnJansen]
- present+
- 07:06:10 [ato]
- RRSAgent: make minutes
- 07:06:10 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 07:06:12 [ato]
- present+
- 07:06:13 [JohnChen]
- JohnChen has joined #testing
- 07:06:17 [mounir]
- mounir has joined #testing
- 07:06:20 [Manishearth]
- present+
- 07:06:21 [mounir]
- Present+
- 07:06:41 [ato]
- jgraham: How to test things that you can't model to simple interaction.
- 07:07:03 [ato]
- ... WebXR is an example of a spec implemented in more than just Chrome, that the tests won't work in other browsers because they have this Mojo crap in them.
- 07:07:07 [Manishearth]
- https://github.com/immersive-web/webxr-test-api/blob/master/explainer.md
- 07:07:13 [ato]
- ... The question is what is the testing strategy for WebXR.
- 07:07:31 [ato]
- Present+ mounir
- 07:07:39 [ato]
- Present+ Manishearth
- 07:07:45 [BitBot]
- (14wpt) [PR] foolip 03merged 13#19078 into 07master: Update interfaces/geometry.idl - https://git.io/JeOI4
- 07:07:50 [ato]
- mounir: There is a testing API in Chrome.
- 07:08:01 [ato]
- ... The backend of that in Chrome is using Mojo.
- 07:08:08 [ato]
- ... So this is not directly exposed to test code.
- 07:08:23 [ato]
- ... The solution we have is to have an internal API or something.
- 07:08:36 [ato]
- Manishearth: There was a testing API proposal that was out of date, and no one implemented it.
- 07:08:37 [jorydotcom]
- jorydotcom has joined #testing
- 07:08:45 [BitBot]
- (14wpt) [PR] foolip 03merged 13#19080 into 07master: Update interfaces/webrtc-stats.idl - https://git.io/JeOIR
- 07:08:50 [JohnJansen]
- RRSAgent, make minutes
- 07:08:50 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen
- 07:08:52 [ato]
- ... There were WPT that had a utils folder that did include specific things, but the tests were written on a shared API.
- 07:08:56 [ato]
- ... It was backed by some Mojo sstuff.
- 07:09:19 [ato]
- ... We implemented a new API for testing that has a backend in Chrome.
- 07:09:28 [ato]
- ... So that's why there's still Mojo in there.
- 07:09:44 [ato]
- ... We're able to run the tests just fine [in Firefox] because we have a native implementation of the API.
- 07:10:14 [ato]
- jgraham: I heard an expectation that Mojo had to load, and the tests would fail irrespective of what the browser did.
- 07:10:31 [ato]
- Manishearth: I've certainly been able to run the tests without Mojo on Servo with success.
- 07:11:46 [ato]
- ... In Servo it's a regular WebIDL that we expose with a pref when needed.
- 07:12:04 [ato]
- foolip: We haven't figured out how to get this to work in Chrome on regular Chrome build.
- 07:12:24 [ato]
- jgraham: Do the WebXR people have any needs from us?
- 07:13:15 [ato]
- mounir: We are challenged about where to put the Mojo bits [?]
- 07:13:34 [ato]
- jgraham: If you want us to install a magic extension we could facilitate that.
- 07:14:04 [ato]
- Hexcles: This seems like a very Chrome specific problem.
- 07:14:05 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19081 merge into 07master: [LayoutNG] Allow overflow-/word-wrap to work with keep-all - https://git.io/JeOIp
- 07:14:20 [ato]
- ... We're discussing archiving and fetching Mojo for testing.
- 07:14:38 [ato]
- jgraham: If you want to set a pref, put a file on the filesystem, or install an extension we could do that.
- 07:14:59 [ato]
- ... For Gecko tests you can use internal APIs by loading a web extension first.
- 07:15:42 [ato]
- Hexcles: There are some challenges lining up the right Mojo version with the right Chrome version.
- 07:15:53 [ato]
- ... We would need to map it on a revision by revision basis.
- 07:16:10 [ato]
- jgraham: It sounds like there are no fundamental WPT issues here.
- 07:16:40 [ato]
- lukebjerring: We build the chromedriver binary [unsigned?] in [some Google system].
- 07:17:09 [ato]
- Hexcles: We will need to add some logic to WPT to figure out the URL to fetch Mojo from.
- 07:17:14 [ato]
- ... I suppose there's no objection to that.
- 07:17:24 [ato]
- jgraham: We're happy to do browser specific stuff for some tests.
- 07:18:00 [ato]
- mounir: Why don't you guys use a content shell for testing?
- 07:18:17 [ato]
- jgraham: The question is how representative it is of the user experience.
- 07:18:26 [ato]
- Hexcles: Internally we're moving towads running the full browser.
- 07:18:39 [ato]
- jgraham: It used to be the case that you couldn't run chromedriver against content shell.
- 07:18:43 [ato]
- Hexcles: It's supposed to work.
- 07:18:46 [marcosc]
- marcosc has joined #testing
- 07:19:03 [ato]
- JohnChen: I can't speak to whether it works today, but it's meant to.
- 07:20:09 [ato]
- ato: The complication with Firefox is that it reads a bunch of prefs at startup time.
- 07:20:23 [ato]
- Manishearth: Is this the first time such a testing API is implemented?
- 07:20:39 [ato]
- jgraham: I think it is for something that is tested cross-browser.
- 07:20:49 [ato]
- Manishearth: It's not that we've made a grave mistake?
- 07:20:51 [ato]
- jgraham: No.
- 07:22:11 [ato]
- mounir: What is the time line for moving to wptrunner for the Chrome infrastructure, so we run full builds?
- 07:22:25 [ato]
- foolip: I don't know the timeline, but we're working on it and have for some time.
- 07:23:39 [ato]
- [discussion about how to fetch the latest Chrome]
- 07:24:22 [ato]
- gsnedders: It would be useful to have Chromium nightly builds running in WPT.
- 07:24:50 [Ms2ger]
- Ms2ger has joined #testing
- 07:25:07 [Hexcles]
- ^ That's unsigned Chromium, where EME etc. does not work
- 07:25:22 [ato]
- https://download-chromium.appspot.com/ provides chromium builds
- 07:25:58 [ato]
- foolip: Just to fix the Mojo problem, there has to be some JS to inject the Mojo files...
- 07:26:21 [ato]
- jgraham: We could mark certain directories to require the Mojo stuff or something.
- 07:27:29 [ato]
- ... Unfortunately for the prefs stuff, this is encoded in the Mozilla metadata and this is not upstreamed to WPT.
- 07:27:52 [ato]
- ... But there is an argument that this could be upstreamed, because at the moment you will occassionally see differences when running Firefox tests upstream.
- 07:28:48 [ato]
- RRSAgent: make minutes
- 07:28:48 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 07:31:24 [ato]
- Topic: Python 3
- 07:31:33 [whsieh]
- whsieh has joined #testing
- 07:31:38 [CalebRouleau]
- CalebRouleau has joined #testing
- 07:32:23 [ato]
- whsieh: Older versions of macOS might not have Python 3 installed.
- 07:32:54 [ato]
- ... It might still be years before we can drop Python 2.
- 07:33:12 [ato]
- jgraham: We need a strategy for us to move to Python 3 in a finite amount of time.
- 07:33:35 [ato]
- ... But we can't move to Python 2.
- 07:33:49 [ato]
- ... Does the manifest generation work in 3 now?
- 07:33:55 [ato]
- gsnedders: It generates a completely different manifest in 2 and 3.
- 07:34:09 [ato]
- gsnedders: Is Python 2 going to be maintained past 2024?
- 07:34:30 [ato]
- jgraham: Gecko is moving away from Python 2, but it's going to be years.
- 07:34:44 [ato]
- ... There are for example also dependencies on wptrunner.
- 07:35:15 [ato]
- foolip: So wptrunner can support only 3?
- 07:35:26 [ato]
- jgraham: I think it needs to support 2 and 3 for some time.
- 07:35:57 [ato]
- foolip: It would be nice if wptrunner keeps working on Python 2 on older Macs.
- 07:36:05 [ato]
- ... So you don't have to download anything special.
- 07:36:13 [ato]
- jgraham: That has historically been a requirement.
- 07:36:24 [ato]
- Hexcles: It sounds like this is not an urgent matter.
- 07:36:32 [zghadyali]
- zghadyali has joined #testing
- 07:36:33 [ato]
- gsnedders: Until Apple stops shipping Python 2.
- 07:37:57 [ato]
- jgraham: In Gecko, all new commands has to be Python 3.
- 07:38:05 [ato]
- ... In practice I'm not sure if it matters.
- 07:38:14 [ato]
- ... But there's a push to move to Python 3.
- 07:38:45 [ato]
- ... One first step would be to run the infrastructure tests in both versions.
- 07:38:58 [ato]
- ... We should maybe start writing new code in Python 3?
- 07:39:11 [ato]
- ... For example, require entry-points to be Python 2+3 compatible.
- 07:39:21 [ato]
- Hexcles: There's currently no incentive.
- 07:39:41 [ato]
- ... Every vendor seems to be postponing the migration indefinitely.
- 07:39:47 [ato]
- jgraham: Not sure that is the case for Mozilla.
- 07:41:02 [ato]
- ... wptrunner is Python 2 only.
- 07:41:24 [ato]
- ... The web server stuff works on 3, but the handler scripts might be fine but no one has checked.
- 07:41:28 [ato]
- Hexcles: They are definitely not fine!
- 07:42:10 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19082 merge into 07master: [webnfc] Add tests for NFCPushOptions.ignoreRead - https://git.io/JeOL8
- 07:42:31 [ato]
- jgraham: For new entry-points we could require 3.
- 07:42:45 [ato]
- ... That's not a big ask, because people add these relatively seldom.
- 07:43:13 [ato]
- ... Once we get the manifest generator to generate the same results in 2 and 3, there should be a unit test for the behaviour.
- 07:43:25 [ato]
- Hexcles: Doesn't sound like there's a modular approach.
- 07:43:57 [ato]
- gsnedders: Manifests does a lot of string manipulation, but there is less string conversion happening in wptrunner.
- 07:44:14 [ato]
- jgraham: I've started getting random patches for this.
- 07:44:41 [ato]
- ... Often I'm scared of accepting these because testing is hard.
- 07:46:05 [ato]
- Hexcles: It's hard to modularise wptrunner.
- 07:46:25 [ato]
- ... Someone needs to spend time to make it work on Python 3, then have integration tests for the Python 3 fixes.
- 07:46:47 [ato]
- jgraham: One first step would be to get it to import cleanly in Python 3 without SyntaxErrors.
- 07:46:56 [ato]
- Hexcles: Do you have an estimate?
- 07:47:08 [ato]
- jgraham: It's not a small amount of work.
- 07:47:12 [ato]
- gsnedders: There's a long tail of work.
- 07:47:25 [ato]
- jgraham: I can imagine some team at Mozilla might get an intern to do this.
- 07:47:59 [ato]
- ... There seems to be some agreement that we need to do this work, and that it's acceptable to stand up tests for the Python 3 behaviour.
- 07:48:19 [ato]
- ... I'm saying there are people working on this, and we should support the people doing work on this. Not that we should do the work right now.
- 07:48:27 [MikeSmith]
- RRSAgent, make minutes
- 07:48:27 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
- 07:48:38 [ato]
- ... And maybe in two years macOS might be more of a force to dictate further progress.
- 07:48:49 [Manishearth]
- Manishearth has left #testing
- 07:49:27 [ato]
- MikeSmith: On macOS, homebrew installs Python 3 by default.
- 07:49:41 [ato]
- ... This is a huge hurdle for contributors to WPT.
- 07:49:49 [ato]
- ... Because it overrides the system default Python 2.
- 07:50:08 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot 03merged 13#19081 into 07master: [LayoutNG] Allow overflow-/word-wrap to work with keep-all - https://git.io/JeOIp
- 07:50:33 [foolip]
- Doubt in the room of whether this is correct.
- 07:53:06 [ato]
- gsnedders: The constraint comes from the WebKit community, who are opposed to installing any other software on the system.
- 07:53:19 [ato]
- MikeSmith: Increasingly there are more and more brew packages relying on Python 3.
- 07:54:33 [ato]
- [discussion about misguided Linux distributions about how they are shipping Python]
- 07:57:09 [gsnedders]
- MikeSmith: https://docs.brew.sh/Homebrew-and-Python#python-3x-or-python-2x says it souldn't
- 07:58:20 [ato]
- gsnedders: The manifest migration is easily a month's work.
- 07:59:12 [ato]
- ... There are performance challenges involved.
- 07:59:28 [ato]
- ... And hard to do without making a complete mess of it.
- 07:59:58 [ato]
- [technical discussion about type annotations]
- 08:00:22 [ato]
- RRSAgent: make minutes
- 08:00:22 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
- 08:00:38 [ato]
- RRSAgent: stop