IRC log of testing on 2019-09-16

Timestamps are in UTC.

00:17:51 [RRSAgent]
RRSAgent has joined #testing
00:17:51 [RRSAgent]
logging to https://www.w3.org/2019/09/16-testing-irc
00:18:45 [jgraham]
Agenda: https://docs.google.com/document/d/1_d2xUBgNn6nmiIXM6m9zSLjzYhvDS4LMrmgoxMQJKuU/edit#
00:19:09 [jgraham]
RRSAgent: Make logs public
00:19:13 [ato]
Is it "present+ <nick>"?
00:19:42 [ato]
Komehyo
00:19:46 [ato]
Uh, https://www.w3.org/2002/03/RRSAgent
00:20:01 [jgraham]
Present+
00:20:12 [ato]
Present+
00:22:42 [jgraham]
RRSAgent: make minutes
00:22:42 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html jgraham
00:27:08 [jgraham]
https://blog.mozilla.org/opendesign/firefox-the-evolution-of-a-brand/
00:27:12 [estes]
estes has joined #testing
00:27:31 [BitBot]
(14wpt) [PR] moz-wptsync-bot requested 13#19070 merge into 07master: [Gecko Bug 1315892] text-orientation: upright' forces used 'direction' to LTR. - https://git.io/JeOvA
00:31:17 [Hexcles]
Hexcles has joined #testing
00:31:46 [JohnJansen]
present+
00:31:57 [Hexcles]
present+
00:31:57 [CalebRouleau]
present+
00:31:59 [AutomatedTester]
present+
00:32:00 [foolip]
present+
00:32:04 [reillyg]
present+
00:32:06 [zcorpan]
zcorpan has joined #testing
00:32:10 [joyee]
present+
00:32:19 [lukebjerring]
present+
00:32:19 [JohnJansen]
jgraham: that's how you do it.
00:32:24 [zcorpan]
https://bocoup.com/blog/how-to-scribe-at-tpac
00:32:34 [JohnChen]
present+
00:32:44 [zghadyali_]
zghadyali_ has joined #testing
00:33:16 [zghadyali_]
present+
00:33:20 [zcorpan]
present+ Simon Pieters, Bocoup
00:33:22 [Zakim]
Zakim has joined #testing
00:33:25 [dxie_]
dxie_ has joined #testing
00:33:26 [AutomatedTester]
present+ David Burns, Mozilla
00:33:39 [Hexcles]
RRSAgent: halp!
00:33:39 [RRSAgent]
I'm logging. I don't understand 'halp!', Hexcles. Try /msg RRSAgent help
00:33:39 [foolip]
present+ Philip Jägenstedt, Google
00:33:44 [ato]
present+ Andreas Tolfsen (Mozilla)
00:33:47 [dxie_]
present+, Daniel Xie, Google
00:34:07 [Hexcles]
present+ Robert Ma (Google)
00:34:09 [zghadyali_]
present+ Zoher Ghadyali, Microsoft
00:34:25 [foolip]
scribenick: foolip
00:34:44 [foolip]
lukebjerring: we have an Agenda in a doc, will work on that after first break
00:34:50 [JohnChen]
present+ John Chen, Google
00:34:52 [joyee]
present+, Qiuyi(Joyee) Zhang, Igalia
00:34:57 [foolip]
lukebjerring: first status updates: https://docs.google.com/presentation/d/10pP5UdurCE3_5YMk6ds8ksz5GIj_Bxixfi9GJP2LOkw/edit?usp=sharing
00:35:28 [yigu]
yigu has joined #testing
00:37:32 [foolip]
lukebjerring: the dents in the Safari graph are infrastructure issues that affected reults
00:37:46 [MikeSmith]
present+
00:38:41 [MikeSmith]
RRSAgent, make minutes
00:38:41 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
00:38:48 [MikeSmith]
agenda?
00:39:18 [foolip]
jgraham: big change for Firefox is rewrite of encodings
00:39:41 [ato]
RRSAgent: make minutes
00:39:41 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
00:40:51 [foolip]
lukebjerring: browser-specific failures and passes are interesting for interoperability
00:40:58 [MikeSmith]
Present- Bocoup
00:41:06 [MikeSmith]
Present- Mozilla
00:41:18 [MikeSmith]
Present- Google
00:41:33 [MikeSmith]
Present- Andreas
00:41:39 [MikeSmith]
Present- Tolfsen
00:41:40 [yigu]
yigu has joined #testing
00:41:47 [MikeSmith]
Present- (Mozilla)
00:41:54 [MikeSmith]
Present- (Google)
00:42:01 [MikeSmith]
Present- Microsoft
00:42:09 [MikeSmith]
Present- Igalia
00:42:24 [MikeSmith]
RRSAgent, make minutes
00:42:24 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
00:42:52 [foolip]
jugglinmike1: yes!
00:43:06 [foolip]
jugglinmike1: meet.google.com/xyo-wzse-nss
00:43:15 [Orphis]
Orphis has joined #testing
00:43:22 [foolip]
jugglinmike1: I thought you'd be out for the day
00:43:45 [MikeSmith]
present+ gsnedders
00:43:57 [foolip]
example of is:different: https://wpt.fyi/results/?label=master&label=experimental&aligned&q=is%3Adifferent
00:44:03 [kevers]
kevers has joined #testing
00:44:09 [gsnedders]
Present+
00:44:17 [MikeSmith]
jugglinmike1, I can webrtc you in
00:44:22 [foolip]
example of the `all` query: https://wpt.fyi/results/?label=master&label=experimental&aligned&q=all%28status%3Aerror%29
00:44:54 [yigu]
present+ Yi Gu, Google
00:45:05 [foolip]
jugglinmike1: can you join? I'll have to click OK
00:45:21 [foolip]
jugglinmike1: good?
00:45:40 [MikeSmith]
s/jugglinmike1:/jugglinmike1,/
00:45:46 [MikeSmith]
RRSAgent, make minutes
00:45:46 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
00:46:29 [kevers]
present
00:46:55 [foolip]
Hexcles: I'll do a live demo now!
00:47:04 [Orphis]
Orphis has joined #testing
00:47:30 [MikeSmith]
present+ Yi_Gu
00:47:38 [MikeSmith]
Present- Google
00:47:42 [MikeSmith]
RRSAgent, make minutes
00:47:42 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
00:48:19 [MikeSmith]
Present- Ma
00:48:24 [kevers]
present+ Kevin Ellis
00:48:39 [MikeSmith]
present- Robert
00:48:49 [ato]
Title: Web Platform Tests, Day 1, TPAC 2019
00:48:57 [MikeSmith]
present+ Robert_Ma
00:49:09 [MikeSmith]
present- Kevin_Ellis
00:49:15 [MikeSmith]
present+ Kevin_Ellis
00:49:21 [MikeSmith]
present- Kevin
00:49:28 [MikeSmith]
present- Ellis
00:50:23 [MikeSmith]
s/jugglinmike1: can you join? I'll have to click OK/
00:50:32 [MikeSmith]
s/jugglinmike1, good?/
00:50:50 [MikeSmith]
RRSAgent, make minutes
00:50:50 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
00:53:01 [MikeSmith]
i/we have an Agenda/Topic: Intro from Luke Bjerring
00:53:19 [ato]
Meeting: Web Platform Tests, Day 1, TPAC 2019
00:53:27 [ato]
RRSAgent: make minutes
00:53:27 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
00:53:45 [MikeSmith]
i/live demo/Topic: Live demo from Robert Ma
00:53:48 [MikeSmith]
RRSAgent, make minutes
00:53:48 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
00:55:17 [ato]
RRSAgent: agenda?
00:55:17 [RRSAgent]
I'm logging. Sorry, nothing found for 'agenda'
00:57:52 [foolip]
jgraham: Taskcluster is going to split into separate instances for Gecko CI and rest of world, will give us new URLs and new UI
00:58:08 [foolip]
jgraham: hope to do Android x86 emulator on Taskcluster by end of year
00:59:00 [foolip]
jgraham: want to run Chrome and Firefox. much more capacity limits than desktop
00:59:07 [foolip]
jgraham: will probably start with daily runs
00:59:35 [foolip]
foolip: also trying to get WebKitGTK running
01:00:20 [foolip]
jgraham: trying to add decision task for Taskcluster, would allow us to schedule dependent tasks
01:00:33 [foolip]
jgraham: for example, we can avoid getting different versions of Firefox
01:00:49 [foolip]
jgraham: or could have a build job
01:02:21 [foolip]
JohnJansen: Edge is now a Chromium browser
01:02:43 [foolip]
JohnJansen: have looked to take advantage of wpt.fyi. 99.9% results are the same a Chrome, so differences are interesting
01:03:04 [foolip]
JohnJansen: we have found bugs (regressions) where Edge is different to Chrome
01:03:18 [foolip]
JohnJansen: we aim to be 24 hours behind Chrome
01:03:32 [foolip]
JohnJansen: but we also have different features that can cause tests to break. usually turns into blocking bugs
01:03:39 [foolip]
JohnJansen: really appreciate the interop view
01:04:01 [foolip]
JohnJansen: when will Edge stable be released? we're bug driven, not date driven
01:04:18 [foolip]
JohnJansen: beta felt very stable to me, very impressed by quality of Chromium out of the box.
01:04:45 [karl]
karl has joined #testing
01:04:59 [foolip]
JohnJansen: wpt.fyi has helped us immensely. interesting to see the pass rate increase so much, now differences are really important
01:05:20 [wanming]
wanming has joined #testing
01:05:34 [foolip]
https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=edge&aligned&q=is%3Adifferent is a view that might be helpful
01:06:04 [jugglinmike1]
foolip: I can talk docs
01:07:11 [MikeSmith]
agenda: https://docs.google.com/document/d/1_d2xUBgNn6nmiIXM6m9zSLjzYhvDS4LMrmgoxMQJKuU/edit#
01:08:10 [JohnJansen]
https://wpt.fyi/interop/IndexedDB/idbobjectstore_createIndex15-autoincrement.htm?label=master&label=experimental&product=chrome&product=edge&aligned&diff&filter=ADC
01:08:21 [foolip]
jugglinmike1: we've been working to improve the docs over the past year
01:09:33 [Orphis]
Orphis has joined #testing
01:10:06 [foolip]
jugglinmike1: we've switched for GitHub pages (Jekyll) to Sphinx, a Python project
01:11:02 [yigu]
See Mike's screen for the figure here: https://meet.google.com/xyo-wzse-nss
01:11:50 [zcorpan]
example search result https://web-platform-tests.org/search.html?q=assert_throws&check_keywords=yes&area=default
01:12:21 [foolip]
jgraham: thanks for improving the docs, it's a big improvement compared to what we had
01:12:27 [jugglinmike1]
Docs: "2019 WPT Documentation Improvements" https://docs.google.com/document/d/16KJbWVRtIjZQX80CQhOvwG2RHlIqVM1ADQTG1Q-QdlE/edit
01:12:34 [MikeSmith]
RRSAgent, make minutes
01:12:34 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
01:13:05 [foolip]
jgraham: Another update. We've now moved from Travis to GitHub Actions. Somewhat mixed results, some spurious failures. Mostly things that require a github token, because the secrets handling is easier
01:13:40 [plh]
plh has joined #testing
01:13:58 [foolip]
JohnJansen: some differences we see between Chrome and Edge might be because we run on Windows too.
01:14:18 [foolip]
lukebjerring: let's break for food
01:14:30 [foolip]
jgraham: let's be back here at 10:45
01:14:47 [Hexcles]
(i.e. in 30min)
01:15:05 [Hexcles]
Hexcles has joined #testing
01:17:56 [denis]
denis has joined #testing
01:22:23 [plh_]
plh_ has joined #testing
01:23:06 [flackr]
flackr has joined #testing
01:24:00 [twisniewski]
present+
01:24:12 [karl]
present+
01:24:22 [miketaylr]
present+ Mike Taylor
01:24:55 [miketaylr]
RRSAgent, make minutes
01:24:55 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html miketaylr
01:25:20 [miketaylr]
present- Mike
01:25:22 [miketaylr]
present- Taylor
01:25:24 [miketaylr]
present+
01:25:26 [miketaylr]
RRSAgent, make minutes
01:25:26 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html miketaylr
01:29:33 [Orphis]
Orphis has joined #testing
01:30:49 [plh]
plh has joined #testing
01:33:54 [tidoust]
tidoust has joined #testing
01:34:11 [Orphis]
Orphis has joined #testing
01:36:23 [Orphis]
Orphis has joined #testing
01:39:32 [karl]
karl has joined #testing
01:43:22 [estes]
estes has joined #testing
01:44:38 [Orphis]
Orphis has joined #testing
01:46:10 [Hexcles]
Hexcles has joined #testing
01:47:26 [kevers]
kevers has joined #testing
01:51:21 [JohnJansen]
Meeting: Web-Platform-Tests TPAC 2019
01:51:36 [JohnJansen]
RRSAgent, make minutes
01:51:36 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen
01:55:47 [zghadyali]
zghadyali has joined #testing
01:56:00 [JohnChen]
JohnChen has joined #testing
01:57:32 [zcorpan]
zcorpan has joined #testing
01:58:26 [JohnJansen]
back. intros...
02:01:42 [foolip]
JohnJansen: if you add `(edge:!pass&edge:!ok)` to the diff query you can see the stuff that's more likely a problem
02:02:37 [karlcow_]
karlcow_ has joined #testing
02:03:42 [jorydotcom]
jorydotcom has joined #testing
02:03:52 [jorydotcom]
+present Jory Burson, Bocoup
02:04:04 [spectranaut]
spectranaut has joined #testing
02:04:08 [jorydotcom]
@jugglinmike1 sleep tight :D
02:04:20 [jorydotcom]
lol
02:04:25 [spectranaut]
present+ Valerie Young, Bocoup
02:04:47 [jorydotcom]
present+ Jory Burson, Bocoup
02:05:59 [marcosc]
marcosc has joined #testing
02:06:54 [gsnedders]
so who have we agreed CSS WG joint meeting with on the CSS WG side?
02:13:05 [MikeSmith]
RRSAgent, make minutes
02:13:05 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
02:13:50 [MikeSmith]
present- Bocoup
02:14:03 [MikeSmith]
RRSAgent, make minutes
02:14:03 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
02:14:29 [foolip]
We're now doing Agenda smithing
02:15:20 [BitBot]
(14wpt) [PR] chromium-wpt-export-bot requested 13#19071 merge into 07master: Port two webkit-xxx-interpolation.html to wpt/ - https://git.io/JeOJr
02:15:53 [BitBot]
(14wpt) [PR] chromium-wpt-export-bot requested 13#19072 merge into 07master: Delete three webkit-xxx-interpolation.html - https://git.io/JeOJK
02:16:00 [jugglinmike]
jugglinmike has joined #testing
02:17:03 [gsnedders]
because https://wiki.csswg.org/planning/tpac-2019#tuesday doesn't have any joint meeting with us there?
02:21:58 [plh_]
plh_ has joined #testing
02:23:46 [JayZhou]
JayZhou has joined #testing
02:31:44 [yigu_]
yigu_ has joined #testing
02:34:41 [xiaoqian]
xiaoqian has joined #testing
02:36:22 [MikeSmith]
https://www.w3.org/wiki/TPAC/2019/SessionIdeas
02:36:34 [JohnJansen]
https://w3c.github.io/tpac-breakouts/sessions.html
02:39:38 [foolip]
scribenick: foolip
02:39:43 [foolip]
Review of 2019 priorities from last TPAC starting
02:39:57 [foolip]
https://docs.google.com/document/d/1UE2KB7gvaEw5gvp4aAQNS9TrFYmnzgDzh5Sr4LQqaQo/edit?usp=sharing
02:40:38 [JohnJansen]
TOPIC: Review of 2019 priorities from last TPAC starting
02:40:44 [zcorpan]
scribenick: zcorpan
02:42:48 [zcorpan]
zcorpan: documentation has been worked on
02:42:56 [zcorpan]
lukebjerring: i like the improved docs!
02:43:14 [zcorpan]
AutomatedTester: improve debugging, this has to do with reftests
02:43:24 [zcorpan]
AutomatedTester: improving tooling about getting debugging info out
02:43:31 [zcorpan]
jgraham: duplicate with the later bullet point
02:44:06 [zcorpan]
JohnJansen: last year i couldn't figure out how to debug a test in python
02:45:25 [zcorpan]
ato: multiprocess debugging in python is fundamental limitation
02:45:37 [zcorpan]
JohnJansen: nobody knows how to debug a test running through wptrunner?
02:45:43 [zcorpan]
jgraham: in wdspec test?
02:45:45 [spectranaut]
spectranaut has joined #testing
02:45:46 [zcorpan]
JohnJansen: yes
02:45:51 [zcorpan]
jgraham: i don' tknow how to do that
02:46:01 [zcorpan]
foolip: printf()
02:46:17 [zcorpan]
jgraham: dunno if something spins up a new python process
02:46:19 [zcorpan]
gsnedders: yes
02:46:40 [zcorpan]
jgraham: maybe pause before the process starts and get the process id
02:46:44 [MikeSmith]
RRSAgent, make minutes
02:46:44 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
02:47:09 [zcorpan]
JohnJansen: reftests have clearly improved in wpt.fyi
02:47:29 [zcorpan]
ato: pytest subprocess
02:47:40 [zcorpan]
ato: command line flag to something something
02:48:12 [zcorpan]
jgraham: the wdspec case is really one process, other things is waiting on it
02:48:26 [zcorpan]
jgraham: if waiting time is indefinite, which i think we can, then it could work
02:48:46 [zcorpan]
ato: also webdriver related timeouts you need to tweak
02:48:56 [zcorpan]
jgraham: for other test types we try to do that
02:49:11 [zcorpan]
jgraham: disabling timeouts when running gdb debugger
02:49:32 [zcorpan]
ato: we've set timeout multiplier to something very high
02:49:43 [zcorpan]
lukebjerring: feature request?
02:49:48 [zcorpan]
JohnJansen: i'm logging one
02:50:04 [JohnJansen]
RRSAgent, make minutes
02:50:04 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen
02:50:07 [zcorpan]
foolip: async_cleanup is major thing that happened
02:50:29 [zcorpan]
foolip: how many have used reftests?
02:50:33 [zcorpan]
(show of hands)
02:51:42 [BitBot]
(14wpt) [issue] thejohnjansen opened 13#19073: We need a way to attach a debugger to the executing test when run from wptrunner - https://git.io/JeOUk
02:51:58 [zcorpan]
gsnedders: "has the assignee reviewed" is a metric we can check
02:52:01 [JohnJansen]
I logged an issue for debugging tests: https://github.com/web-platform-tests/wpt/issues/19073
02:52:14 [zcorpan]
foolip: i set a filter to prio PRs where i'm assigned
02:52:32 [Hexcles]
https://github-health.appspot.com/
02:52:49 [zcorpan]
lukebjerring: for people who use github-health this helps
02:53:15 [zcorpan]
jgraham: interesting to see if it's working for ppl who are not on infrastructure
02:53:37 [ato]
What Mozilla uses: https://addons.mozilla.org/en-US/firefox/addon/myqonly/
02:53:52 [zcorpan]
foolip: i look at things i'm assigned to
02:53:58 [zcorpan]
zcorpan: me too, or reassign
02:54:01 [ato]
Shows a notification in your browser for GitHub items you are assigned to.
02:54:08 [zcorpan]
foolip: ideas for wpt-pr-bot?
02:54:21 [zcorpan]
jgraham: improved a bit, but hasn't shifted a lot
02:55:02 [zcorpan]
gsnedders: how many of the 250 PRs from 2017 are still open?
02:55:41 [zcorpan]
gsnedders: we need people who work on this
02:55:46 [zcorpan]
gsnedders: we've tried for years
02:56:54 [zcorpan]
foolip: teaching people how to do email filtering works
02:57:35 [foolip]
zcorpan: we know that new contributors don't have a good time
02:57:42 [AutomatedTester]
scribenick: foolip
02:57:42 [foolip]
zcorpan: their PRs are stuck for years
02:57:55 [foolip]
zcorpan: do we want their contributions from people who don't already work on browser engines?
02:58:20 [foolip]
zcorpan: if the answer is no we shouldn't pretend that we accept PRs, but if we do we should allocate time to do the review
02:59:09 [gsnedders]
s/work on this/work on the web features in question to review them/
02:59:20 [foolip]
jgraham: the problem is who are the we who should do the reviews?
02:59:43 [foolip]
jgraham: ownership is fine-grained and we don't have a way to enforce that people do their job
03:00:12 [foolip]
zcorpan: we can explore incentives to do reviews, like celebrate those who do the most reviews
03:00:22 [jorydotcom]
+1
03:02:40 [tidoust]
tidoust has joined #testing
03:04:17 [zcorpan]
jgraham: it might be not worthwhile in the moment, but in the long run it may be
03:04:42 [zcorpan]
jgraham: choice between possible impact over a long period vs definite impact over short period
03:05:02 [zcorpan]
ato: are we sure that a stale review has been considered
03:05:12 [zcorpan]
ato: if nobody has looked at it, how do we know if it's important
03:05:24 [zcorpan]
miketaylr: has anyone retriaged?
03:05:39 [zcorpan]
miketaylr: might be 50 out of the 440 that are amazing
03:05:59 [zcorpan]
jgraham: i think we classify things in terms of their status (waiting for review, waiting for OP)
03:06:14 [zcorpan]
jgraham: don't classify by importance, size
03:06:47 [zcorpan]
miketaylr: if i say to other person at moz that 50 tests are important, it can be prioritized
03:06:58 [zcorpan]
jgraham: if we can sort things by stuff that show browser issues
03:07:08 [zcorpan]
jgraham: that is valuable
03:07:14 [zcorpan]
foolip: come back to this?
03:07:17 [zcorpan]
jgraham: yes
03:07:24 [tidoust]
tidoust has joined #testing
03:08:03 [zcorpan]
jgraham: next bullet point: beginners onboarding
03:08:14 [zcorpan]
foolip: more to say that docs?
03:08:38 [zcorpan]
jgraham: mdn survey
03:09:08 [zcorpan]
foolip: i think that survey can be helpful to prioritize our work
03:09:27 [zcorpan]
jgraham: "make CI more robust"
03:09:30 [zcorpan]
foolip: i think we did
03:09:48 [zcorpan]
foolip: reliability question, can talk about PRs in next session
03:09:59 [zcorpan]
foolip: running more, getting more complete results
03:10:04 [zcorpan]
jgraham: problems with safari
03:10:05 [BitBot]
(14wpt) [PR] chromium-wpt-export-bot 04closed 13#18999: Revert "Reland "Started implementing the STAPIT algorithm"" - https://git.io/Jemrw
03:10:27 [zcorpan]
foolip: github actoins acting up, problems for PRs
03:10:45 [zcorpan]
foolip: PR results and full run are better now than 1 year ago
03:10:58 [zcorpan]
jgraham: "give web platform engineers the tools they need to prio"
03:11:21 [zcorpan]
jgraham: is done or in progress
03:11:28 [zcorpan]
gsnedders: don't have bug links
03:11:32 [zcorpan]
jgraham: so that's in progress
03:12:17 [zcorpan]
lukebjerring: every bug that i triage that has a crbug is fixed or explanation why it won't be in the short term
03:12:28 [zcorpan]
lukebjerring: made some progress there
03:12:54 [zcorpan]
foolip: would you encourage others to do the same?
03:12:56 [zcorpan]
lukebjerring: yes
03:13:13 [foolip]
lukebjerring: https://wpt.fyi/insights
03:13:23 [zcorpan]
lukebjerring: double down on what is an important failure, not yet really solved
03:13:37 [Hexcles]
lukebjerring: RFC https://github.com/web-platform-tests/rfcs/pull/16 will move the needle more significantly
03:13:50 [zcorpan]
jgraham: we've looked at firefox-only failures
03:14:08 [zcorpan]
jgraham: triage that but resistance is always "ok these things fail but are they relevant to compat?"
03:14:21 [zcorpan]
jgraham: we know if we fix compat bugs they often end up with new tests
03:14:32 [zcorpan]
jgraham: don't have the reverse: does fixing a test fix web compat
03:15:10 [zcorpan]
jgraham: data exists that building up internal understanding of when it's important
03:15:27 [zcorpan]
jgraham: things that are different between firefox and safari removes some objections
03:15:46 [zcorpan]
jgraham: or chrome and safari. firefox-specific failures
03:16:13 [zcorpan]
jgraham: have better ways to tell if a failure is important, is valuable to us
03:16:35 [zcorpan]
jgraham: we still don't understand how given a test failure impacts web compat
03:17:12 [zcorpan]
JohnJansen: if we reduce a site bug and fix that, then track back to which tests now pass
03:17:16 [zcorpan]
JohnJansen: but not reverse
03:17:35 [zcorpan]
JohnJansen: interaction between features also
03:17:57 [zcorpan]
jgraham: we could test that, but don't test all possible intersections
03:18:33 [zcorpan]
jgraham: reducing things that go into the suite
03:18:51 [zcorpan]
foolip: beyond harness errors, can we talk about flaky tests?
03:18:56 [zcorpan]
jgraham: infra has improved
03:19:17 [denis_]
denis_ has joined #testing
03:20:23 [zcorpan]
jgraham: some ideas that we havent' followed up
03:20:58 [zcorpan]
jgraham: if we can get to a situation where a library works around a browser bug, if they also file a browser bug
03:21:03 [zcorpan]
jgraham: would be impactful
03:21:18 [zcorpan]
foolip: triage metadata and bug linking, if there's labeling and start counts
03:21:52 [zcorpan]
foolip: test to spec linking. bikeshed and respec have something!
03:23:40 [zcorpan]
jgraham: progress, but relevant people aren't in the room
03:23:47 [zcorpan]
jgraham: test coverage
03:24:13 [zcorpan]
Hexcles: in blink we have coverage comparison between wpt and legacy layout tests
03:24:20 [zcorpan]
Hexcles: results are pretty good overall
03:24:41 [zcorpan]
lukebjerring: 10% difference between running all tests (wpt+layout) and only running wpt
03:25:06 [zcorpan]
lukebjerring: identify low hanging fruit to upstream tests to wpt
03:25:08 [Hexcles]
https://storage.googleapis.com/blink-wpt-coverage/201812/index.html
03:25:34 [zcorpan]
Hexcles: we can easily measure coverage data
03:25:49 [zcorpan]
Hexcles: i think there's effort to collect data on ongoing basis
03:25:59 [whsieh]
whsieh has joined #testing
03:26:03 [zcorpan]
JohnJansen: region coverage?
03:26:09 [zcorpan]
JohnJansen: what does it mean?
03:26:16 [zcorpan]
gsnedders: basic blocks
03:27:07 [zcorpan]
jgraham: for gecko we have a coverage metric
03:27:12 [foolip]
Tootip is "Region coverage is the percentage of code regions which have been executed at least once. A code region may span multiple lines (e.g in a large function body with no control flow). However, it's also possible for a single line to contain multiple code regions (e.g in 'return x || y && z')."
03:27:18 [zcorpan]
jgraham: daily coverage runs maybe
03:27:34 [zcorpan]
jgraham: can see per testsuite
03:27:41 [ato]
RRSAgent: make minutes, please
03:27:41 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
03:27:44 [zcorpan]
jgraham: the UI doesn't yet display diffs between testsuites
03:27:53 [zcorpan]
jgraham: I have some of that data for gecko
03:28:04 [ato]
ScribeNick: zcorpan
03:28:05 [zcorpan]
jgraham: some bits are better covered by mochitests, like gamepad
03:28:18 [zcorpan]
jgraham: some areas where wpt has better coverage
03:28:19 [ato]
RRSAgent: make minutes
03:28:19 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
03:28:37 [zcorpan]
Hexcles: one thing stood out
03:28:44 [zcorpan]
Hexcles: accessibility is poorly tested in wpt
03:28:48 [zcorpan]
gsnedders: not possible to test
03:29:09 [zcorpan]
jgraham: coverage is a way to identify places where wpt is weak
03:29:20 [zcorpan]
jgraham: might not always be fixable (like maybe GC)
03:29:35 [zcorpan]
jgraham: gamepad or accessibility should be testable
03:29:52 [zcorpan]
foolip: that's test automation
03:30:07 [zcorpan]
foolip: has coverage improved?
03:30:17 [_11]
_11 has joined #testing
03:30:33 [zcorpan]
MikeSmith: how many tests do we have that are using ???
03:30:44 [zcorpan]
MikeSmith: cases manual tests are converted
03:31:05 [zcorpan]
MikeSmith: incentive to groups to automate manual tests
03:31:16 [zcorpan]
MikeSmith: awareness, people don't know this is available
03:31:23 [Hexcles]
s/???/testdriver.js
03:31:28 [zcorpan]
foolip: 400 files testdriver.js
03:32:08 [zcorpan]
jgraham: automating stuff that we can't currently automate... some success
03:32:19 [zcorpan]
ato: in terms of testability extensions of webdriver api
03:32:30 [zcorpan]
ato: we've seen other specs e.g. permissions, write extensions
03:32:43 [zcorpan]
ato: but no implementations
03:32:58 [zcorpan]
foolip: generate test report has been implemented
03:33:29 [zcorpan]
jgraham: gecko hasn't implemented that yet
03:33:51 [zcorpan]
foolip: ~200 files testdriver.js a year ago
03:34:14 [zcorpan]
foolip: pointer events, painful?
03:34:34 [zcorpan]
NavidZ_: let anyone add switch to testdriver protocol
03:34:42 [zcorpan]
NavidZ_: gives user activation to the page
03:34:51 [zcorpan]
NavidZ_: nobody can do that
03:35:05 [zcorpan]
NavidZ_: one way to ask the test author to click or something
03:35:19 [zcorpan]
NavidZ_: question is, how much of this do we want to expose to the testers
03:35:32 [zcorpan]
NavidZ_: not testing parts of the browser
03:35:42 [zcorpan]
NavidZ_: some apis already that can expose that
03:35:53 [zcorpan]
NavidZ_: if we want to go down that path, do it all the way
03:36:03 [zcorpan]
NavidZ_: on windows, behaves differently than linux on mac
03:36:22 [zcorpan]
NavidZ_: chromium tests only, test can choose based on platform
03:36:32 [zcorpan]
NavidZ_: do we want to expose that in testdriver?
03:36:51 [zcorpan]
NavidZ_: focus next: can't get that without user interaction. difference between platforms
03:37:31 [zcorpan]
NavidZ_: adding automation for specific things, expose more and more of inner workings of the browser
03:37:49 [zcorpan]
foolip: do we want to add API for create user interaction defined in html?
03:37:58 [zcorpan]
ato: big discussion in browser testing and tools
03:38:14 [Hexcles]
Hexcles has joined #testing
03:38:18 [zcorpan]
<lunch>
03:38:31 [zcorpan]
s/<lunch>/<std-lunch>/
03:40:03 [zcorpan]
zcorpan has joined #testing
04:01:30 [cybai]
cybai has joined #testing
04:02:46 [spectranaut]
spectranaut has joined #testing
04:06:13 [denis]
denis has joined #testing
04:07:07 [cybai_]
cybai_ has joined #testing
04:12:36 [cybai]
cybai has joined #testing
04:14:50 [Hexcles]
Hexcles has joined #testing
04:20:21 [kevers]
kevers has joined #testing
04:24:08 [BitBot]
(14wpt) [PR] dirkschulze requested 13#19074 merge into 07master: Add smfr as reviewer for CSS Transforms - https://git.io/JeOTN
04:24:11 [denis]
denis has joined #testing
04:24:30 [prushforth]
prushforth has joined #testing
04:25:03 [prushforth]
present+ Peter Rushforth, Natural Resources Canada, Observer
04:30:56 [whsieh]
whsieh has joined #testing
04:31:02 [tidoust]
tidoust has joined #testing
04:34:28 [zcorpan]
zcorpan has joined #testing
04:40:23 [BitBot]
(14wpt) [PR] dirkschulze 03merged 13#19074 into 07master: Add smfr as reviewer for CSS Transforms - https://git.io/JeOTN
04:42:45 [zcorpan]
RRSAgent: make minutes
04:42:45 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html zcorpan
04:45:11 [BitBot]
(14wpt) [PR] moz-wptsync-bot requested 13#19075 merge into 07master: [Gecko Bug 1579993] Add WPT subgrid tests and a few regular Grid baseline alignment tests. - https://git.io/JeOkz
04:45:34 [marcosc]
marcosc has joined #testing
04:45:57 [zcorpan]
participants & their position to help with scribing https://docs.google.com/spreadsheets/d/1cqPK6ze2OCLsho4twJHNLZUPktfejIiiDlMwv0TaZBg/edit#gid=0
04:46:46 [Hexcles]
Hexcles has joined #testing
04:51:20 [zghadyali]
zghadyali has joined #Testing
04:56:20 [plh]
plh has joined #testing
05:00:41 [cybai]
cybai has joined #testing
05:01:54 [Hexcles]
Hexcles has joined #testing
05:04:52 [saschanaz]
saschanaz has joined #testing
05:07:40 [ato]
RRSAgent: make minutes, please
05:07:40 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
05:07:41 [jorydotcom]
jorydotcom has joined #testing
05:08:25 [marcosc]
marcosc has joined #testing
05:09:29 [foolip]
Just added https://www.w3.org/wiki/TPAC/2019/SessionIdeas#web-platform-tests_update_.26_discussion
05:09:29 [denis_]
denis_ has joined #testing
05:10:11 [CalebRouleau]
CalebRouleau has joined #testing
05:10:23 [CalebRouleau]
present+ Caleb Rouleau, Google
05:10:55 [JohnJansen]
present+
05:10:57 [ato]
ScribeNick: ato
05:11:15 [cb]
cb has joined #testing
05:11:34 [zcorpan]
present- Google
05:11:40 [ato]
Topic: Infra: making full use of the test results on PRs
05:11:55 [ato]
foolip: PR checks are a bit noisy, so at the moment you want to ignore them. Fourteen in total.
05:12:11 [ato]
... Flaky tests also. Should we do something about them?
05:12:25 [ato]
jgraham: Let's talk about what is there already.
05:12:44 [foolip]
Example: https://github.com/web-platform-tests/wpt/pull/19067
05:12:57 [MikeSmith]
RRSAgent, make minutes
05:12:57 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
05:13:12 [whsieh]
whsieh has joined #testing
05:13:19 [MikeSmith]
present- Natural_Resources_Canada
05:13:28 [MikeSmith]
present- Observer
05:14:37 [BitBot]
(14wpt) [PR] dirkschulze 03merged 13#11169 into 07master: Remove tests for SVG transform with CSS syntax - part 1 - https://git.io/JeOk6
05:14:44 [ato]
foolip: The Azure pipeline is noisy, just produces results for Safari.
05:14:53 [ato]
foolip: This is a request for the Azure team, to make this less noisy.
05:15:08 [ato]
JohnJansen: You'd like the report to be merged into one? All the pipelines into one?
05:15:26 [ato]
gsnedders: In GitHub checks it makes sense for them to be separate?
05:15:38 [ato]
foolip: Talking to Microsoft about this makes sense I think.
05:16:15 [ato]
jgraham: With TaskCluster you have to click through to get the specific URL for the job.
05:16:26 [ato]
... This is not natural to everyone.
05:16:43 [ato]
... I think switching that to checks will make it noisy, but may make it easier to figure out what's failing.
05:17:07 [ato]
... A decision task would help also, because it would just run the tasks that are relevant.
05:17:08 [JohnJansen]
ACTION: JohnJansen follow up with Azure Pipelines team for this
05:17:14 [ato]
... Rather than the tasks we have decided to run for you.
05:17:31 [ato]
[talk about priorities]
05:18:17 [ato]
jgraham: With a decision task [on Taskcluster] it will run only the things that are dependencies, such as lints when source file changes, but essentially just the test jobs related.
05:18:27 [cybai_]
cybai_ has joined #testing
05:18:40 [ato]
jgraham: We've run up quite hard against the limits of GitHub.
05:18:51 [ato]
jgraham: Checks vs. non-blocking checks
05:19:40 [ato]
foolip: Final part, wpt.fyi.
05:20:04 [ato]
foolip: Sometimes there are more wpt.fyi non-blocking checks, due to deployment etc.
05:20:07 [cb_]
cb_ has joined #testing
05:20:19 [ato]
... Could we merge this into a single check somehow?
05:20:35 [ato]
lukebjerring: If it's desirable we could reuse the same name and aggregate the information.
05:21:00 [ato]
lukebjerring: Pending results will tie into these, so we will have pending checks whilst the process [?] is still ongoing.
05:21:19 [ato]
foolip: Sometimes I see the same problem in all three.
05:21:29 [ato]
jgraham: I've never clicked on these that have neutral status.
05:21:38 [ato]
jgraham: So I don't think it’s an effective signal to give users.
05:22:19 [Hexcles]
s/process [?]/result processor/
05:22:36 [ato]
... Being able to look at the Firefox results specifically has been useful, but I don't think that the neutral results provide any value.
05:22:48 [ato]
foolip: How do we indicate this in a better way? Make it fail?
05:23:07 [ato]
jgraham: Failing with a button to "un-fail" it might be OK, but confusing.
05:23:15 [ato]
... Maybe submit an issue on the code, somewhere?
05:23:19 [ato]
foolip: Review comment?
05:23:21 [karl]
karl has joined #testing
05:23:39 [ato]
jgraham: "This test appears to be erroring in this browser, if that’s fine you should dismiss this review and accept the PR."
05:24:19 [foolip]
https://developer.github.com/v3/checks/runs/ has screenshot at top of possible outcomes
05:24:29 [ato]
lukebjerring: If you click on the details for the wpt.fyi job, there’s a recompute button right now, and for the case that is neutral which I would hope to upgrade to blocking, it would add a comment saying which user clicked the button.
05:24:39 [ato]
... "Luke marked this as passing using the ignore button"
05:25:20 [ato]
foolip: There are a few different options, a red triangle and provide text.
05:25:38 [ato]
foolip: I suppose it wouldn’t be blocking then. We could decide not to make it blocking, I mean.
05:26:05 [ato]
jgraham: I’m happy to experiment with stuff here, but I’m very cynical about how much people writing these tests are going to care to dig into the issues.
05:26:20 [ato]
... At least initially people will be whining because their PRs are stuck and don’t bother investigating.
05:26:28 [rego]
rego has joined #testing
05:26:36 [ato]
foolip: Low-frequency but serious solution we’re willing to experiment with?
05:26:48 [ato]
jgraham: No apss everywhere.
05:27:27 [ato]
foolip: Let's go with "consistent error" everywhere.
05:27:31 [ato]
s/apss/pass/
05:27:53 [ato]
foolip: A lot of tests that Chrome adds are harness errors in Safari.
05:27:54 [jesopo]
jesopo has joined #testing
05:28:11 [ato]
jgraham: Maybe it should be more cynical about things that come from browser syncs.
05:28:29 [ato]
... A test for an API that only exists in Chrome might be problematic.
05:28:34 [Hexcles]
ACTION: lukebjerring Hexcles: explore setting wpt.fyi check results to failures when e.g. tests error everywhere
05:29:21 [ato]
lukebjerring: It has to previously been failing and now passing at the moment, and I think it would be more useful to make it blocking than neutral.
05:30:01 [ato]
lukebjerring: We have merged PRs that failed tests, but they were neutral in the PR and didn't check. We had to unroll a big list of changes.
05:30:17 [zcorpan]
zcorpan has joined #testing
05:30:26 [ato]
jgraham: There needs to be some mechanism whereby the author has to say that “this is what I intended”.
05:30:41 [ato]
... Pass going to fail happens often. The test could’ve been wrong.
05:30:58 [CalebRouleau]
q+
05:31:05 [ato]
... The test might’ve been passing in all browsers, but is now always failing.
05:31:43 [ato]
lukebjerring: If you fix a test that was passing incorrectly and becomes failing, you’ll be blocking becaus ethe statistics will be wrong [?]
05:32:04 [ato]
gsnedders: It’s not the job failing within the Chromium CI system.
05:32:28 [ato]
jgraham: People are less invested in gettnig their change landed in Chromium than they are in WPT.
05:32:38 [ato]
jgraham: Or is your CL blocked until it lands in WPT?
05:32:46 [ato]
lukebjerring: That’s a discussion we want to have today.
05:32:48 [JohnJansen]
q?
05:33:00 [CalebRouleau]
q-
05:33:11 [ato]
lukebjerring: We want to propagate that into the original CL before it lands, before the export happens. This reduces human manual intervention.
05:33:31 [ato]
jgraham: [explains Mozilla process]
05:34:29 [ato]
lukebjerring: Forcing someone to click an extra button is worth it balanced against the pain it puts on us later.
05:34:38 [xiaoqian]
xiaoqian has joined #testing
05:34:55 [ato]
... Because of the privilege that WPT is given, it is important for WPT not to land into a bad state.
05:35:16 [ato]
jgraham: Occasionally we land something that fails all our tests.
05:35:29 [ato]
... Coming up with good heuristics is hard, but I agree it’s a problem.
05:35:40 [ato]
... The problem is that this is a system platform developrs don’t interact with often.
05:35:50 [CalebRouleau]
q+
05:36:37 [foolip]
q+
05:36:37 [ato]
ato: Should upstream PRs be scrutinised harder?
05:37:04 [ato]
jgraham: Maybe that case should have different heuristics.
05:37:29 [gsnedders]
ack CalebRouleau
05:37:49 [ato]
CalebRouleau: Whitelist of things that is supposed to be passing?
05:37:59 [ato]
CalebRouleau: This would be a change in the code.
05:38:15 [ato]
jgraham: The expectation metadata could be put in WPT perhaps.
05:38:22 [ato]
... But it would meet a lot of resistance.
05:38:26 [ato]
Hexcles: I agree.
05:38:50 [ato]
jgraham: It would be like for the infra tests. You would have to go and update the expectations when you mean for something to change.
05:39:00 [ato]
jgraham: It would work, because that’s how we do it for Gecko.
05:39:09 [estes]
estes has joined #testing
05:39:32 [ato]
CalebRouleau: The proposal was _whitelist_, not expectation data for all.
05:39:55 [foolip]
q-
05:40:06 [ato]
lukebjerring: You’re going to end up exhaustively listing the metadata anyway, and you’ll want this per-browser basis.
05:40:31 [ato]
jgraham: It would also increase the workload on authors because they would have to update the metadata also for other web browsers.
05:41:01 [ato]
lukebjerring: Someone can submit something to the codebase and suddenly everything fails, without any warning.
05:41:37 [Hexcles]
q?
05:41:39 [ato]
jgraham: Human intervention needed at some level.
05:41:52 [ato]
... Different rules for different directories?
05:43:11 [ato]
ato: It’s not unprecedented that tests are wrong and we make them go from pass to fail expecedtly.
05:43:39 [ato]
jgraham: [explains a recent case]
05:45:10 [Hexcles]
foolip: if you are heavily involved you can become a reviewer or even codeowner on github
05:45:57 [ato]
lukebjerring: I think it’s reasonable to ask people to explain why they are making a test go from pass to fail.
05:46:48 [ato]
... It’s easy to demonstrate why we would impose a blocking check for it that people would understand.
05:47:01 [ato]
jgraham: I would need to see statistics to see that it’s usually problematic [?]
05:47:51 [ato]
... Another case is where someone adds a test that passes only in a single browser, but times out or fails in all others.
05:48:06 [ato]
lukebjerring: Do we make these a failing action on GitHub and allow it to be ignored?
05:48:19 [ato]
... And when a regression happens, they are propagated into the Firefox CI?
05:48:53 [ato]
jgraham: We’re starting up a project so that we can hopefully surface this stuff as feedback in the code review, as opposed to finding out after a change lands.
05:49:03 [CalebRouleau]
CalebRouleau has joined #testing
05:49:25 [ato]
... I think we can surface this stuff to developers earlier and get them to look at it more readily.
05:49:49 [ato]
... It’s unclear what the best mechanism for the feedback in WPT on GH should be.
05:50:37 [ato]
... If you choose to ignore you have to give a reason I think.
05:50:41 [ato]
... With checks you can't maybe.
05:51:15 [ato]
foolip: The GH checks are made for this, they have a desired outcome and you have to give a reason.
05:51:21 [ato]
foolip: That flow is sort of built for this.
05:51:33 [ato]
jgraham: I’m unconvinced they have got the UI right.
05:51:38 [ato]
foolip: Well we haven’t tried it out.
05:52:23 [ato]
jgraham: OK, maybe we should try the checks thing first, otherwise try the code review thing.
05:52:38 [ato]
Hexcles: We could have different rules for different directories also.
05:53:10 [ato]
jgraham: If we could initially roll it out for directories with developers we know are engaged.
05:53:43 [ato]
foolip: If we see the flow in a PR, maybe this is going to be easier to assess.
05:54:45 [ato]
lukebjerring: I do a deliberate regression PR to wpt.fyi for this
05:55:04 [ato]
RRSAgent: make minutes
05:55:04 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
05:56:25 [ato]
RRSAgent: Make minutes
05:56:25 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
05:57:28 [Hexcles]
ACTION foolip: write an RFC for making regressions detected by wpt.fyi require actions
05:57:40 [ato]
foolip: Understanding the flakiness is super-hard.
05:58:07 [ato]
foolip: We could make the logs less verbose, but then we’d have to increase it to find out what’s wrong.
05:58:22 [ato]
jgraham: We have a log handler that picks out the things it thinks is important.
05:58:49 [hyojin]
hyojin has joined #testing
05:59:26 [ato]
lukebjerring: Custom interpretation jobs is what wpt.fyi is doing by definition.
05:59:55 [cb]
cb has joined #testing
05:59:57 [Hexcles]
RRSAgent make minutes
06:00:31 [ato]
lukebjerring: Instead of designing TaskCluster to have custom log interpretation, we should do this in wpt.fyi.
06:01:07 [ato]
jgraham: But if we had this in TaskCluster, we would use this consistently also for other things. It could produce an artifact we could reuse elsewhere.
06:02:14 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html gsnedders
06:03:53 [Hexcles]
ACTION: Hexcles: switch Taskcluster to GitHub Checks
06:04:51 [ato]
jgraham: [explains TaskCluster]
06:06:46 [ato]
RRSAgent, draft minutes
06:06:46 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
06:07:47 [ato]
jgraham: Regarding flakey tests, recently the expected test status at Mozilla have support for multiple test statuses.
06:07:59 [ato]
... For example, this test can either pass or fail.
06:08:19 [zghadyali]
zghadyali has joined #testing
06:08:27 [ato]
lukebjerring: What is the main reason you’re against having expectation data upstream in WPT?
06:08:48 [ato]
jgraham: On WPT we’re running things mostly on one platform.
06:08:53 [ato]
... On Gecko we’re running a vast number of platforms which is specific.
06:09:25 [ato]
... This could cause double work because you would have to care for other browsers’ expectation data in addition to your own.
06:10:25 [ato]
Hexcles: There's a more fundamental problem: browser versions matter.
06:10:43 [ato]
Hexcles: If you have an upstream expectation that applies to a specific version, it will be even more difficult.
06:11:12 [JohnJansen]
present+
06:13:02 [ato]
lukebjerring: There's an implicit falkiness if you have multiple acceptable statuses, and having a totally separate place to say soemthing is flaky seems kind of bad because it duplicates information.
06:15:36 [ato]
jgraham: If you had the WPT flakiness data in tree you could build some interesting tooling.
06:16:02 [ato]
jgraham: "You’ve marked this test as not-flaky in Chrome, but it’s still flaky elsewhere.”
06:16:34 [ato]
foolip: Flakiness on master vs. flakiness on PRs?
06:17:20 [cb]
cb has joined #testing
06:17:26 [ato]
lukebjerring: We run a cron job that looks at the last ten runs and checks for flaky tests and lets some person know about it.
06:18:08 [ato]
jgraham: Can we have a checks page?
06:18:14 [ato]
form on*
06:18:22 [ato]
lukebjerring: Recompute, ignore
06:18:54 [ato]
... You could also have "flag as flaky" which would change the metadata and re-run the computation, say.
06:19:42 [ato]
... If you have a flaky test on TC and people aren’t bothered looking into it, they are already ignoring this and force merging.
06:19:49 [ato]
... So it would not make the current situation any worse.
06:20:11 [ato]
foolip: I don’t think we necessarily have anything we disagree on with regards to flaky tests.
06:21:37 [ato]
JohnJansen: Explanation on how to deal with flaky tests.
06:21:57 [cybai]
cybai has joined #testing
06:22:06 [ato]
lukebjerring: Documentation for “so you’ve been told your tests are flaky” sounds like a good idea.
06:22:27 [ato]
Hexcles: A tutorial linked from the GH checks.
06:23:08 [ato]
foolip: If we have a button that marks as flaky, you can be sure people are going to click it.
06:23:17 [ato]
... This will eventually make the system useless if overused.
06:23:31 [ato]
lukebjerring: It’s hard to identify false-positives.
06:23:54 [ato]
jgraham: If you haven’t seen flakiness in the last month, then we probably don’t care.
06:24:07 [ato]
jgraham: You can remove expectation data when the flakiness goes away.
06:24:32 [cb]
cb has joined #testing
06:25:02 [ato]
jgraham: In Chromium you run every test on every commit, you can get backed out if the test becomes flaky as a result of the CL.
06:25:29 [ato]
... In the Gecko case it matters less if the metadata is a little bit inaccurate.
06:26:28 [ato]
Topic: Python 3
06:26:53 [ato]
jgraham: Python Foundation are stop going to stop maintaining Python 2 sometime next year.
06:27:07 [ato]
... There is no need for immediate panic because RedHat will continue providing updates for another four years.
06:27:20 [ato]
... But we should have a plan for migration to Python 3.
06:27:38 [ato]
s/are stop/are/
06:27:58 [ato]
... Gecko is starting to move things to Python 3 slowly, and there is increased need for us to have a roadmap for this as well.
06:28:21 [ato]
Hexcles: There was no conclusion what WebKit is going to do based on the email thread.
06:28:43 [ato]
jgraham: Previously we had assumed that WebKit was a blocking concern.
06:29:12 [ato]
... But now it looks like they are switching to Python 3, or possibly going to stop shipping Python altogether.
06:29:18 [ato]
Hexcles: macOS 10.11
06:29:30 [ato]
Hexcles: We should find someone from the WebKit community.
06:29:33 [Hexcles]
s/10.11/10.15
06:29:43 [ato]
s/.11/10.15/
06:29:57 [ato]
RRSAgent: make minutes
06:29:57 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
06:31:07 [ato]
jgraham: We need support for both Python 2 and 3.
06:31:39 [ato]
... For example making the WPT frontend run in either, then make the commands it despatches to run Python 3.
06:32:10 [ato]
... That seems to be the way Gecko works.
06:32:29 [ato]
Break.
06:33:44 [tidoust]
tidoust has joined #testing
06:35:10 [Hexcles]
Hexcles has joined #testing
06:40:23 [BitBot]
(14wpt) [PR] autofoolip requested 13#19076 merge into 07master: Update interfaces/IndexedDB.idl - https://git.io/JeOIW
06:40:29 [BitBot]
(14wpt) [PR] autofoolip requested 13#19077 merge into 07master: Update interfaces/gamepad.idl - https://git.io/JeOI8
06:40:33 [BitBot]
(14wpt) [PR] autofoolip requested 13#19078 merge into 07master: Update interfaces/geometry.idl - https://git.io/JeOI4
06:40:37 [BitBot]
(14wpt) [PR] autofoolip requested 13#19079 merge into 07master: Update interfaces/webmidi.idl - https://git.io/JeOIB
06:40:41 [BitBot]
(14wpt) [PR] autofoolip requested 13#19080 merge into 07master: Update interfaces/webrtc-stats.idl - https://git.io/JeOIR
06:42:33 [plh]
plh has joined #testing
06:43:45 [xiaoqian]
xiaoqian has joined #testing
06:44:50 [spectranaut]
spectranaut has joined #testing
06:46:06 [zcorpan]
zcorpan has joined #testing
06:56:06 [cybai]
cybai has joined #testing
07:02:10 [Manishearth]
Manishearth has joined #testing
07:02:41 [Hexcles]
Hexcles has joined #testing
07:04:13 [ato]
Topic: WebXR
07:04:23 [ato]
RRSAgent: make minutes
07:04:23 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
07:05:05 [foolip]
A testing API exists: https://github.com/immersive-web/webxr-test-api
07:05:14 [CalebRouleau]
CalebRouleau has joined #testing
07:05:57 [ato]
RRSAgent: this meeting spans midnight
07:05:58 [karl]
karl has joined #testing
07:05:59 [BitBot]
(14wpt) [PR] foolip 03merged 13#19076 into 07master: Update interfaces/IndexedDB.idl - https://git.io/JeOIW
07:05:59 [Manishearth]
present+
07:06:06 [ato]
RRSAgent: listen
07:06:07 [JohnJansen]
present+
07:06:10 [ato]
RRSAgent: make minutes
07:06:10 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
07:06:12 [ato]
present+
07:06:13 [JohnChen]
JohnChen has joined #testing
07:06:17 [mounir]
mounir has joined #testing
07:06:20 [Manishearth]
present+
07:06:21 [mounir]
Present+
07:06:41 [ato]
jgraham: How to test things that you can't model to simple interaction.
07:07:03 [ato]
... WebXR is an example of a spec implemented in more than just Chrome, that the tests won't work in other browsers because they have this Mojo crap in them.
07:07:07 [Manishearth]
https://github.com/immersive-web/webxr-test-api/blob/master/explainer.md
07:07:13 [ato]
... The question is what is the testing strategy for WebXR.
07:07:31 [ato]
Present+ mounir
07:07:39 [ato]
Present+ Manishearth
07:07:45 [BitBot]
(14wpt) [PR] foolip 03merged 13#19078 into 07master: Update interfaces/geometry.idl - https://git.io/JeOI4
07:07:50 [ato]
mounir: There is a testing API in Chrome.
07:08:01 [ato]
... The backend of that in Chrome is using Mojo.
07:08:08 [ato]
... So this is not directly exposed to test code.
07:08:23 [ato]
... The solution we have is to have an internal API or something.
07:08:36 [ato]
Manishearth: There was a testing API proposal that was out of date, and no one implemented it.
07:08:37 [jorydotcom]
jorydotcom has joined #testing
07:08:45 [BitBot]
(14wpt) [PR] foolip 03merged 13#19080 into 07master: Update interfaces/webrtc-stats.idl - https://git.io/JeOIR
07:08:50 [JohnJansen]
RRSAgent, make minutes
07:08:50 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen
07:08:52 [ato]
... There were WPT that had a utils folder that did include specific things, but the tests were written on a shared API.
07:08:56 [ato]
... It was backed by some Mojo sstuff.
07:09:19 [ato]
... We implemented a new API for testing that has a backend in Chrome.
07:09:28 [ato]
... So that's why there's still Mojo in there.
07:09:44 [ato]
... We're able to run the tests just fine [in Firefox] because we have a native implementation of the API.
07:10:14 [ato]
jgraham: I heard an expectation that Mojo had to load, and the tests would fail irrespective of what the browser did.
07:10:31 [ato]
Manishearth: I've certainly been able to run the tests without Mojo on Servo with success.
07:11:46 [ato]
... In Servo it's a regular WebIDL that we expose with a pref when needed.
07:12:04 [ato]
foolip: We haven't figured out how to get this to work in Chrome on regular Chrome build.
07:12:24 [ato]
jgraham: Do the WebXR people have any needs from us?
07:13:15 [ato]
mounir: We are challenged about where to put the Mojo bits [?]
07:13:34 [ato]
jgraham: If you want us to install a magic extension we could facilitate that.
07:14:04 [ato]
Hexcles: This seems like a very Chrome specific problem.
07:14:05 [BitBot]
(14wpt) [PR] chromium-wpt-export-bot requested 13#19081 merge into 07master: [LayoutNG] Allow overflow-/word-wrap to work with keep-all - https://git.io/JeOIp
07:14:20 [ato]
... We're discussing archiving and fetching Mojo for testing.
07:14:38 [ato]
jgraham: If you want to set a pref, put a file on the filesystem, or install an extension we could do that.
07:14:59 [ato]
... For Gecko tests you can use internal APIs by loading a web extension first.
07:15:42 [ato]
Hexcles: There are some challenges lining up the right Mojo version with the right Chrome version.
07:15:53 [ato]
... We would need to map it on a revision by revision basis.
07:16:10 [ato]
jgraham: It sounds like there are no fundamental WPT issues here.
07:16:40 [ato]
lukebjerring: We build the chromedriver binary [unsigned?] in [some Google system].
07:17:09 [ato]
Hexcles: We will need to add some logic to WPT to figure out the URL to fetch Mojo from.
07:17:14 [ato]
... I suppose there's no objection to that.
07:17:24 [ato]
jgraham: We're happy to do browser specific stuff for some tests.
07:18:00 [ato]
mounir: Why don't you guys use a content shell for testing?
07:18:17 [ato]
jgraham: The question is how representative it is of the user experience.
07:18:26 [ato]
Hexcles: Internally we're moving towads running the full browser.
07:18:39 [ato]
jgraham: It used to be the case that you couldn't run chromedriver against content shell.
07:18:43 [ato]
Hexcles: It's supposed to work.
07:18:46 [marcosc]
marcosc has joined #testing
07:19:03 [ato]
JohnChen: I can't speak to whether it works today, but it's meant to.
07:20:09 [ato]
ato: The complication with Firefox is that it reads a bunch of prefs at startup time.
07:20:23 [ato]
Manishearth: Is this the first time such a testing API is implemented?
07:20:39 [ato]
jgraham: I think it is for something that is tested cross-browser.
07:20:49 [ato]
Manishearth: It's not that we've made a grave mistake?
07:20:51 [ato]
jgraham: No.
07:22:11 [ato]
mounir: What is the time line for moving to wptrunner for the Chrome infrastructure, so we run full builds?
07:22:25 [ato]
foolip: I don't know the timeline, but we're working on it and have for some time.
07:23:39 [ato]
[discussion about how to fetch the latest Chrome]
07:24:22 [ato]
gsnedders: It would be useful to have Chromium nightly builds running in WPT.
07:24:50 [Ms2ger]
Ms2ger has joined #testing
07:25:07 [Hexcles]
^ That's unsigned Chromium, where EME etc. does not work
07:25:22 [ato]
https://download-chromium.appspot.com/ provides chromium builds
07:25:58 [ato]
foolip: Just to fix the Mojo problem, there has to be some JS to inject the Mojo files...
07:26:21 [ato]
jgraham: We could mark certain directories to require the Mojo stuff or something.
07:27:29 [ato]
... Unfortunately for the prefs stuff, this is encoded in the Mozilla metadata and this is not upstreamed to WPT.
07:27:52 [ato]
... But there is an argument that this could be upstreamed, because at the moment you will occassionally see differences when running Firefox tests upstream.
07:28:48 [ato]
RRSAgent: make minutes
07:28:48 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
07:31:24 [ato]
Topic: Python 3
07:31:33 [whsieh]
whsieh has joined #testing
07:31:38 [CalebRouleau]
CalebRouleau has joined #testing
07:32:23 [ato]
whsieh: Older versions of macOS might not have Python 3 installed.
07:32:54 [ato]
... It might still be years before we can drop Python 2.
07:33:12 [ato]
jgraham: We need a strategy for us to move to Python 3 in a finite amount of time.
07:33:35 [ato]
... But we can't move to Python 2.
07:33:49 [ato]
... Does the manifest generation work in 3 now?
07:33:55 [ato]
gsnedders: It generates a completely different manifest in 2 and 3.
07:34:09 [ato]
gsnedders: Is Python 2 going to be maintained past 2024?
07:34:30 [ato]
jgraham: Gecko is moving away from Python 2, but it's going to be years.
07:34:44 [ato]
... There are for example also dependencies on wptrunner.
07:35:15 [ato]
foolip: So wptrunner can support only 3?
07:35:26 [ato]
jgraham: I think it needs to support 2 and 3 for some time.
07:35:57 [ato]
foolip: It would be nice if wptrunner keeps working on Python 2 on older Macs.
07:36:05 [ato]
... So you don't have to download anything special.
07:36:13 [ato]
jgraham: That has historically been a requirement.
07:36:24 [ato]
Hexcles: It sounds like this is not an urgent matter.
07:36:32 [zghadyali]
zghadyali has joined #testing
07:36:33 [ato]
gsnedders: Until Apple stops shipping Python 2.
07:37:57 [ato]
jgraham: In Gecko, all new commands has to be Python 3.
07:38:05 [ato]
... In practice I'm not sure if it matters.
07:38:14 [ato]
... But there's a push to move to Python 3.
07:38:45 [ato]
... One first step would be to run the infrastructure tests in both versions.
07:38:58 [ato]
... We should maybe start writing new code in Python 3?
07:39:11 [ato]
... For example, require entry-points to be Python 2+3 compatible.
07:39:21 [ato]
Hexcles: There's currently no incentive.
07:39:41 [ato]
... Every vendor seems to be postponing the migration indefinitely.
07:39:47 [ato]
jgraham: Not sure that is the case for Mozilla.
07:41:02 [ato]
... wptrunner is Python 2 only.
07:41:24 [ato]
... The web server stuff works on 3, but the handler scripts might be fine but no one has checked.
07:41:28 [ato]
Hexcles: They are definitely not fine!
07:42:10 [BitBot]
(14wpt) [PR] chromium-wpt-export-bot requested 13#19082 merge into 07master: [webnfc] Add tests for NFCPushOptions.ignoreRead - https://git.io/JeOL8
07:42:31 [ato]
jgraham: For new entry-points we could require 3.
07:42:45 [ato]
... That's not a big ask, because people add these relatively seldom.
07:43:13 [ato]
... Once we get the manifest generator to generate the same results in 2 and 3, there should be a unit test for the behaviour.
07:43:25 [ato]
Hexcles: Doesn't sound like there's a modular approach.
07:43:57 [ato]
gsnedders: Manifests does a lot of string manipulation, but there is less string conversion happening in wptrunner.
07:44:14 [ato]
jgraham: I've started getting random patches for this.
07:44:41 [ato]
... Often I'm scared of accepting these because testing is hard.
07:46:05 [ato]
Hexcles: It's hard to modularise wptrunner.
07:46:25 [ato]
... Someone needs to spend time to make it work on Python 3, then have integration tests for the Python 3 fixes.
07:46:47 [ato]
jgraham: One first step would be to get it to import cleanly in Python 3 without SyntaxErrors.
07:46:56 [ato]
Hexcles: Do you have an estimate?
07:47:08 [ato]
jgraham: It's not a small amount of work.
07:47:12 [ato]
gsnedders: There's a long tail of work.
07:47:25 [ato]
jgraham: I can imagine some team at Mozilla might get an intern to do this.
07:47:59 [ato]
... There seems to be some agreement that we need to do this work, and that it's acceptable to stand up tests for the Python 3 behaviour.
07:48:19 [ato]
... I'm saying there are people working on this, and we should support the people doing work on this. Not that we should do the work right now.
07:48:27 [MikeSmith]
RRSAgent, make minutes
07:48:27 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith
07:48:38 [ato]
... And maybe in two years macOS might be more of a force to dictate further progress.
07:48:49 [Manishearth]
Manishearth has left #testing
07:49:27 [ato]
MikeSmith: On macOS, homebrew installs Python 3 by default.
07:49:41 [ato]
... This is a huge hurdle for contributors to WPT.
07:49:49 [ato]
... Because it overrides the system default Python 2.
07:50:08 [BitBot]
(14wpt) [PR] chromium-wpt-export-bot 03merged 13#19081 into 07master: [LayoutNG] Allow overflow-/word-wrap to work with keep-all - https://git.io/JeOIp
07:50:33 [foolip]
Doubt in the room of whether this is correct.
07:53:06 [ato]
gsnedders: The constraint comes from the WebKit community, who are opposed to installing any other software on the system.
07:53:19 [ato]
MikeSmith: Increasingly there are more and more brew packages relying on Python 3.
07:54:33 [ato]
[discussion about misguided Linux distributions about how they are shipping Python]
07:57:09 [gsnedders]
MikeSmith: https://docs.brew.sh/Homebrew-and-Python#python-3x-or-python-2x says it souldn't
07:58:20 [ato]
gsnedders: The manifest migration is easily a month's work.
07:59:12 [ato]
... There are performance challenges involved.
07:59:28 [ato]
... And hard to do without making a complete mess of it.
07:59:58 [ato]
[technical discussion about type annotations]
08:00:22 [ato]
RRSAgent: make minutes
08:00:22 [RRSAgent]
I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato
08:00:38 [ato]
RRSAgent: stop