00:17:51 RRSAgent has joined #testing 00:17:51 logging to https://www.w3.org/2019/09/16-testing-irc 00:18:45 Agenda: https://docs.google.com/document/d/1_d2xUBgNn6nmiIXM6m9zSLjzYhvDS4LMrmgoxMQJKuU/edit# 00:19:09 RRSAgent: Make logs public 00:19:13 Is it "present+ "? 00:19:42 Komehyo 00:19:46 Uh, https://www.w3.org/2002/03/RRSAgent 00:20:01 Present+ 00:20:12 Present+ 00:22:42 RRSAgent: make minutes 00:22:42 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html jgraham 00:27:08 https://blog.mozilla.org/opendesign/firefox-the-evolution-of-a-brand/ 00:27:12 estes has joined #testing 00:27:31 (14wpt) [PR] moz-wptsync-bot requested 13#19070 merge into 07master: [Gecko Bug 1315892] text-orientation: upright' forces used 'direction' to LTR. - https://git.io/JeOvA 00:31:17 Hexcles has joined #testing 00:31:46 present+ 00:31:57 present+ 00:31:57 present+ 00:31:59 present+ 00:32:00 present+ 00:32:04 present+ 00:32:06 zcorpan has joined #testing 00:32:10 present+ 00:32:19 present+ 00:32:19 jgraham: that's how you do it. 00:32:24 https://bocoup.com/blog/how-to-scribe-at-tpac 00:32:34 present+ 00:32:44 zghadyali_ has joined #testing 00:33:16 present+ 00:33:20 present+ Simon Pieters, Bocoup 00:33:22 Zakim has joined #testing 00:33:25 dxie_ has joined #testing 00:33:26 present+ David Burns, Mozilla 00:33:39 RRSAgent: halp! 00:33:39 I'm logging. I don't understand 'halp!', Hexcles. Try /msg RRSAgent help 00:33:39 present+ Philip Jägenstedt, Google 00:33:44 present+ Andreas Tolfsen (Mozilla) 00:33:47 present+, Daniel Xie, Google 00:34:07 present+ Robert Ma (Google) 00:34:09 present+ Zoher Ghadyali, Microsoft 00:34:25 scribenick: foolip 00:34:44 lukebjerring: we have an Agenda in a doc, will work on that after first break 00:34:50 present+ John Chen, Google 00:34:52 present+, Qiuyi(Joyee) Zhang, Igalia 00:34:57 lukebjerring: first status updates: https://docs.google.com/presentation/d/10pP5UdurCE3_5YMk6ds8ksz5GIj_Bxixfi9GJP2LOkw/edit?usp=sharing 00:35:28 yigu has joined #testing 00:37:32 lukebjerring: the dents in the Safari graph are infrastructure issues that affected reults 00:37:46 present+ 00:38:41 RRSAgent, make minutes 00:38:41 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 00:38:48 agenda? 00:39:18 jgraham: big change for Firefox is rewrite of encodings 00:39:41 RRSAgent: make minutes 00:39:41 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 00:40:51 lukebjerring: browser-specific failures and passes are interesting for interoperability 00:40:58 Present- Bocoup 00:41:06 Present- Mozilla 00:41:18 Present- Google 00:41:33 Present- Andreas 00:41:39 Present- Tolfsen 00:41:40 yigu has joined #testing 00:41:47 Present- (Mozilla) 00:41:54 Present- (Google) 00:42:01 Present- Microsoft 00:42:09 Present- Igalia 00:42:24 RRSAgent, make minutes 00:42:24 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 00:42:52 jugglinmike1: yes! 00:43:06 jugglinmike1: meet.google.com/xyo-wzse-nss 00:43:15 Orphis has joined #testing 00:43:22 jugglinmike1: I thought you'd be out for the day 00:43:45 present+ gsnedders 00:43:57 example of is:different: https://wpt.fyi/results/?label=master&label=experimental&aligned&q=is%3Adifferent 00:44:03 kevers has joined #testing 00:44:09 Present+ 00:44:17 jugglinmike1, I can webrtc you in 00:44:22 example of the `all` query: https://wpt.fyi/results/?label=master&label=experimental&aligned&q=all%28status%3Aerror%29 00:44:54 present+ Yi Gu, Google 00:45:05 jugglinmike1: can you join? I'll have to click OK 00:45:21 jugglinmike1: good? 00:45:40 s/jugglinmike1:/jugglinmike1,/ 00:45:46 RRSAgent, make minutes 00:45:46 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 00:46:29 present 00:46:55 Hexcles: I'll do a live demo now! 00:47:04 Orphis has joined #testing 00:47:30 present+ Yi_Gu 00:47:38 Present- Google 00:47:42 RRSAgent, make minutes 00:47:42 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 00:48:19 Present- Ma 00:48:24 present+ Kevin Ellis 00:48:39 present- Robert 00:48:49 Title: Web Platform Tests, Day 1, TPAC 2019 00:48:57 present+ Robert_Ma 00:49:09 present- Kevin_Ellis 00:49:15 present+ Kevin_Ellis 00:49:21 present- Kevin 00:49:28 present- Ellis 00:50:23 s/jugglinmike1: can you join? I'll have to click OK/ 00:50:32 s/jugglinmike1, good?/ 00:50:50 RRSAgent, make minutes 00:50:50 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 00:53:01 i/we have an Agenda/Topic: Intro from Luke Bjerring 00:53:19 Meeting: Web Platform Tests, Day 1, TPAC 2019 00:53:27 RRSAgent: make minutes 00:53:27 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 00:53:45 i/live demo/Topic: Live demo from Robert Ma 00:53:48 RRSAgent, make minutes 00:53:48 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 00:55:17 RRSAgent: agenda? 00:55:17 I'm logging. Sorry, nothing found for 'agenda' 00:57:52 jgraham: Taskcluster is going to split into separate instances for Gecko CI and rest of world, will give us new URLs and new UI 00:58:08 jgraham: hope to do Android x86 emulator on Taskcluster by end of year 00:59:00 jgraham: want to run Chrome and Firefox. much more capacity limits than desktop 00:59:07 jgraham: will probably start with daily runs 00:59:35 foolip: also trying to get WebKitGTK running 01:00:20 jgraham: trying to add decision task for Taskcluster, would allow us to schedule dependent tasks 01:00:33 jgraham: for example, we can avoid getting different versions of Firefox 01:00:49 jgraham: or could have a build job 01:02:21 JohnJansen: Edge is now a Chromium browser 01:02:43 JohnJansen: have looked to take advantage of wpt.fyi. 99.9% results are the same a Chrome, so differences are interesting 01:03:04 JohnJansen: we have found bugs (regressions) where Edge is different to Chrome 01:03:18 JohnJansen: we aim to be 24 hours behind Chrome 01:03:32 JohnJansen: but we also have different features that can cause tests to break. usually turns into blocking bugs 01:03:39 JohnJansen: really appreciate the interop view 01:04:01 JohnJansen: when will Edge stable be released? we're bug driven, not date driven 01:04:18 JohnJansen: beta felt very stable to me, very impressed by quality of Chromium out of the box. 01:04:45 karl has joined #testing 01:04:59 JohnJansen: wpt.fyi has helped us immensely. interesting to see the pass rate increase so much, now differences are really important 01:05:20 wanming has joined #testing 01:05:34 https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=edge&aligned&q=is%3Adifferent is a view that might be helpful 01:06:04 foolip: I can talk docs 01:07:11 agenda: https://docs.google.com/document/d/1_d2xUBgNn6nmiIXM6m9zSLjzYhvDS4LMrmgoxMQJKuU/edit# 01:08:10 https://wpt.fyi/interop/IndexedDB/idbobjectstore_createIndex15-autoincrement.htm?label=master&label=experimental&product=chrome&product=edge&aligned&diff&filter=ADC 01:08:21 jugglinmike1: we've been working to improve the docs over the past year 01:09:33 Orphis has joined #testing 01:10:06 jugglinmike1: we've switched for GitHub pages (Jekyll) to Sphinx, a Python project 01:11:02 See Mike's screen for the figure here: https://meet.google.com/xyo-wzse-nss 01:11:50 example search result https://web-platform-tests.org/search.html?q=assert_throws&check_keywords=yes&area=default 01:12:21 jgraham: thanks for improving the docs, it's a big improvement compared to what we had 01:12:27 Docs: "2019 WPT Documentation Improvements" https://docs.google.com/document/d/16KJbWVRtIjZQX80CQhOvwG2RHlIqVM1ADQTG1Q-QdlE/edit 01:12:34 RRSAgent, make minutes 01:12:34 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 01:13:05 jgraham: Another update. We've now moved from Travis to GitHub Actions. Somewhat mixed results, some spurious failures. Mostly things that require a github token, because the secrets handling is easier 01:13:40 plh has joined #testing 01:13:58 JohnJansen: some differences we see between Chrome and Edge might be because we run on Windows too. 01:14:18 lukebjerring: let's break for food 01:14:30 jgraham: let's be back here at 10:45 01:14:47 (i.e. in 30min) 01:15:05 Hexcles has joined #testing 01:17:56 denis has joined #testing 01:22:23 plh_ has joined #testing 01:23:06 flackr has joined #testing 01:24:00 present+ 01:24:12 present+ 01:24:22 present+ Mike Taylor 01:24:55 RRSAgent, make minutes 01:24:55 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html miketaylr 01:25:20 present- Mike 01:25:22 present- Taylor 01:25:24 present+ 01:25:26 RRSAgent, make minutes 01:25:26 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html miketaylr 01:29:33 Orphis has joined #testing 01:30:49 plh has joined #testing 01:33:54 tidoust has joined #testing 01:34:11 Orphis has joined #testing 01:36:23 Orphis has joined #testing 01:39:32 karl has joined #testing 01:43:22 estes has joined #testing 01:44:38 Orphis has joined #testing 01:46:10 Hexcles has joined #testing 01:47:26 kevers has joined #testing 01:51:21 Meeting: Web-Platform-Tests TPAC 2019 01:51:36 RRSAgent, make minutes 01:51:36 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen 01:55:47 zghadyali has joined #testing 01:56:00 JohnChen has joined #testing 01:57:32 zcorpan has joined #testing 01:58:26 back. intros... 02:01:42 JohnJansen: if you add `(edge:!pass&edge:!ok)` to the diff query you can see the stuff that's more likely a problem 02:02:37 karlcow_ has joined #testing 02:03:42 jorydotcom has joined #testing 02:03:52 +present Jory Burson, Bocoup 02:04:04 spectranaut has joined #testing 02:04:08 @jugglinmike1 sleep tight :D 02:04:20 lol 02:04:25 present+ Valerie Young, Bocoup 02:04:47 present+ Jory Burson, Bocoup 02:05:59 marcosc has joined #testing 02:06:54 so who have we agreed CSS WG joint meeting with on the CSS WG side? 02:13:05 RRSAgent, make minutes 02:13:05 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 02:13:50 present- Bocoup 02:14:03 RRSAgent, make minutes 02:14:03 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 02:14:29 We're now doing Agenda smithing 02:15:20 (14wpt) [PR] chromium-wpt-export-bot requested 13#19071 merge into 07master: Port two webkit-xxx-interpolation.html to wpt/ - https://git.io/JeOJr 02:15:53 (14wpt) [PR] chromium-wpt-export-bot requested 13#19072 merge into 07master: Delete three webkit-xxx-interpolation.html - https://git.io/JeOJK 02:16:00 jugglinmike has joined #testing 02:17:03 because https://wiki.csswg.org/planning/tpac-2019#tuesday doesn't have any joint meeting with us there? 02:21:58 plh_ has joined #testing 02:23:46 JayZhou has joined #testing 02:31:44 yigu_ has joined #testing 02:34:41 xiaoqian has joined #testing 02:36:22 https://www.w3.org/wiki/TPAC/2019/SessionIdeas 02:36:34 https://w3c.github.io/tpac-breakouts/sessions.html 02:39:38 scribenick: foolip 02:39:43 Review of 2019 priorities from last TPAC starting 02:39:57 https://docs.google.com/document/d/1UE2KB7gvaEw5gvp4aAQNS9TrFYmnzgDzh5Sr4LQqaQo/edit?usp=sharing 02:40:38 TOPIC: Review of 2019 priorities from last TPAC starting 02:40:44 scribenick: zcorpan 02:42:48 zcorpan: documentation has been worked on 02:42:56 lukebjerring: i like the improved docs! 02:43:14 AutomatedTester: improve debugging, this has to do with reftests 02:43:24 AutomatedTester: improving tooling about getting debugging info out 02:43:31 jgraham: duplicate with the later bullet point 02:44:06 JohnJansen: last year i couldn't figure out how to debug a test in python 02:45:25 ato: multiprocess debugging in python is fundamental limitation 02:45:37 JohnJansen: nobody knows how to debug a test running through wptrunner? 02:45:43 jgraham: in wdspec test? 02:45:45 spectranaut has joined #testing 02:45:46 JohnJansen: yes 02:45:51 jgraham: i don' tknow how to do that 02:46:01 foolip: printf() 02:46:17 jgraham: dunno if something spins up a new python process 02:46:19 gsnedders: yes 02:46:40 jgraham: maybe pause before the process starts and get the process id 02:46:44 RRSAgent, make minutes 02:46:44 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 02:47:09 JohnJansen: reftests have clearly improved in wpt.fyi 02:47:29 ato: pytest subprocess 02:47:40 ato: command line flag to something something 02:48:12 jgraham: the wdspec case is really one process, other things is waiting on it 02:48:26 jgraham: if waiting time is indefinite, which i think we can, then it could work 02:48:46 ato: also webdriver related timeouts you need to tweak 02:48:56 jgraham: for other test types we try to do that 02:49:11 jgraham: disabling timeouts when running gdb debugger 02:49:32 ato: we've set timeout multiplier to something very high 02:49:43 lukebjerring: feature request? 02:49:48 JohnJansen: i'm logging one 02:50:04 RRSAgent, make minutes 02:50:04 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen 02:50:07 foolip: async_cleanup is major thing that happened 02:50:29 foolip: how many have used reftests? 02:50:33 (show of hands) 02:51:42 (14wpt) [issue] thejohnjansen opened 13#19073: We need a way to attach a debugger to the executing test when run from wptrunner - https://git.io/JeOUk 02:51:58 gsnedders: "has the assignee reviewed" is a metric we can check 02:52:01 I logged an issue for debugging tests: https://github.com/web-platform-tests/wpt/issues/19073 02:52:14 foolip: i set a filter to prio PRs where i'm assigned 02:52:32 https://github-health.appspot.com/ 02:52:49 lukebjerring: for people who use github-health this helps 02:53:15 jgraham: interesting to see if it's working for ppl who are not on infrastructure 02:53:37 What Mozilla uses: https://addons.mozilla.org/en-US/firefox/addon/myqonly/ 02:53:52 foolip: i look at things i'm assigned to 02:53:58 zcorpan: me too, or reassign 02:54:01 Shows a notification in your browser for GitHub items you are assigned to. 02:54:08 foolip: ideas for wpt-pr-bot? 02:54:21 jgraham: improved a bit, but hasn't shifted a lot 02:55:02 gsnedders: how many of the 250 PRs from 2017 are still open? 02:55:41 gsnedders: we need people who work on this 02:55:46 gsnedders: we've tried for years 02:56:54 foolip: teaching people how to do email filtering works 02:57:35 zcorpan: we know that new contributors don't have a good time 02:57:42 scribenick: foolip 02:57:42 zcorpan: their PRs are stuck for years 02:57:55 zcorpan: do we want their contributions from people who don't already work on browser engines? 02:58:20 zcorpan: if the answer is no we shouldn't pretend that we accept PRs, but if we do we should allocate time to do the review 02:59:09 s/work on this/work on the web features in question to review them/ 02:59:20 jgraham: the problem is who are the we who should do the reviews? 02:59:43 jgraham: ownership is fine-grained and we don't have a way to enforce that people do their job 03:00:12 zcorpan: we can explore incentives to do reviews, like celebrate those who do the most reviews 03:00:22 +1 03:02:40 tidoust has joined #testing 03:04:17 jgraham: it might be not worthwhile in the moment, but in the long run it may be 03:04:42 jgraham: choice between possible impact over a long period vs definite impact over short period 03:05:02 ato: are we sure that a stale review has been considered 03:05:12 ato: if nobody has looked at it, how do we know if it's important 03:05:24 miketaylr: has anyone retriaged? 03:05:39 miketaylr: might be 50 out of the 440 that are amazing 03:05:59 jgraham: i think we classify things in terms of their status (waiting for review, waiting for OP) 03:06:14 jgraham: don't classify by importance, size 03:06:47 miketaylr: if i say to other person at moz that 50 tests are important, it can be prioritized 03:06:58 jgraham: if we can sort things by stuff that show browser issues 03:07:08 jgraham: that is valuable 03:07:14 foolip: come back to this? 03:07:17 jgraham: yes 03:07:24 tidoust has joined #testing 03:08:03 jgraham: next bullet point: beginners onboarding 03:08:14 foolip: more to say that docs? 03:08:38 jgraham: mdn survey 03:09:08 foolip: i think that survey can be helpful to prioritize our work 03:09:27 jgraham: "make CI more robust" 03:09:30 foolip: i think we did 03:09:48 foolip: reliability question, can talk about PRs in next session 03:09:59 foolip: running more, getting more complete results 03:10:04 jgraham: problems with safari 03:10:05 (14wpt) [PR] chromium-wpt-export-bot 04closed 13#18999: Revert "Reland "Started implementing the STAPIT algorithm"" - https://git.io/Jemrw 03:10:27 foolip: github actoins acting up, problems for PRs 03:10:45 foolip: PR results and full run are better now than 1 year ago 03:10:58 jgraham: "give web platform engineers the tools they need to prio" 03:11:21 jgraham: is done or in progress 03:11:28 gsnedders: don't have bug links 03:11:32 jgraham: so that's in progress 03:12:17 lukebjerring: every bug that i triage that has a crbug is fixed or explanation why it won't be in the short term 03:12:28 lukebjerring: made some progress there 03:12:54 foolip: would you encourage others to do the same? 03:12:56 lukebjerring: yes 03:13:13 lukebjerring: https://wpt.fyi/insights 03:13:23 lukebjerring: double down on what is an important failure, not yet really solved 03:13:37 lukebjerring: RFC https://github.com/web-platform-tests/rfcs/pull/16 will move the needle more significantly 03:13:50 jgraham: we've looked at firefox-only failures 03:14:08 jgraham: triage that but resistance is always "ok these things fail but are they relevant to compat?" 03:14:21 jgraham: we know if we fix compat bugs they often end up with new tests 03:14:32 jgraham: don't have the reverse: does fixing a test fix web compat 03:15:10 jgraham: data exists that building up internal understanding of when it's important 03:15:27 jgraham: things that are different between firefox and safari removes some objections 03:15:46 jgraham: or chrome and safari. firefox-specific failures 03:16:13 jgraham: have better ways to tell if a failure is important, is valuable to us 03:16:35 jgraham: we still don't understand how given a test failure impacts web compat 03:17:12 JohnJansen: if we reduce a site bug and fix that, then track back to which tests now pass 03:17:16 JohnJansen: but not reverse 03:17:35 JohnJansen: interaction between features also 03:17:57 jgraham: we could test that, but don't test all possible intersections 03:18:33 jgraham: reducing things that go into the suite 03:18:51 foolip: beyond harness errors, can we talk about flaky tests? 03:18:56 jgraham: infra has improved 03:19:17 denis_ has joined #testing 03:20:23 jgraham: some ideas that we havent' followed up 03:20:58 jgraham: if we can get to a situation where a library works around a browser bug, if they also file a browser bug 03:21:03 jgraham: would be impactful 03:21:18 foolip: triage metadata and bug linking, if there's labeling and start counts 03:21:52 foolip: test to spec linking. bikeshed and respec have something! 03:23:40 jgraham: progress, but relevant people aren't in the room 03:23:47 jgraham: test coverage 03:24:13 Hexcles: in blink we have coverage comparison between wpt and legacy layout tests 03:24:20 Hexcles: results are pretty good overall 03:24:41 lukebjerring: 10% difference between running all tests (wpt+layout) and only running wpt 03:25:06 lukebjerring: identify low hanging fruit to upstream tests to wpt 03:25:08 https://storage.googleapis.com/blink-wpt-coverage/201812/index.html 03:25:34 Hexcles: we can easily measure coverage data 03:25:49 Hexcles: i think there's effort to collect data on ongoing basis 03:25:59 whsieh has joined #testing 03:26:03 JohnJansen: region coverage? 03:26:09 JohnJansen: what does it mean? 03:26:16 gsnedders: basic blocks 03:27:07 jgraham: for gecko we have a coverage metric 03:27:12 Tootip is "Region coverage is the percentage of code regions which have been executed at least once. A code region may span multiple lines (e.g in a large function body with no control flow). However, it's also possible for a single line to contain multiple code regions (e.g in 'return x || y && z')." 03:27:18 jgraham: daily coverage runs maybe 03:27:34 jgraham: can see per testsuite 03:27:41 RRSAgent: make minutes, please 03:27:41 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 03:27:44 jgraham: the UI doesn't yet display diffs between testsuites 03:27:53 jgraham: I have some of that data for gecko 03:28:04 ScribeNick: zcorpan 03:28:05 jgraham: some bits are better covered by mochitests, like gamepad 03:28:18 jgraham: some areas where wpt has better coverage 03:28:19 RRSAgent: make minutes 03:28:19 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 03:28:37 Hexcles: one thing stood out 03:28:44 Hexcles: accessibility is poorly tested in wpt 03:28:48 gsnedders: not possible to test 03:29:09 jgraham: coverage is a way to identify places where wpt is weak 03:29:20 jgraham: might not always be fixable (like maybe GC) 03:29:35 jgraham: gamepad or accessibility should be testable 03:29:52 foolip: that's test automation 03:30:07 foolip: has coverage improved? 03:30:17 <_11> _11 has joined #testing 03:30:33 MikeSmith: how many tests do we have that are using ??? 03:30:44 MikeSmith: cases manual tests are converted 03:31:05 MikeSmith: incentive to groups to automate manual tests 03:31:16 MikeSmith: awareness, people don't know this is available 03:31:23 s/???/testdriver.js 03:31:28 foolip: 400 files testdriver.js 03:32:08 jgraham: automating stuff that we can't currently automate... some success 03:32:19 ato: in terms of testability extensions of webdriver api 03:32:30 ato: we've seen other specs e.g. permissions, write extensions 03:32:43 ato: but no implementations 03:32:58 foolip: generate test report has been implemented 03:33:29 jgraham: gecko hasn't implemented that yet 03:33:51 foolip: ~200 files testdriver.js a year ago 03:34:14 foolip: pointer events, painful? 03:34:34 NavidZ_: let anyone add switch to testdriver protocol 03:34:42 NavidZ_: gives user activation to the page 03:34:51 NavidZ_: nobody can do that 03:35:05 NavidZ_: one way to ask the test author to click or something 03:35:19 NavidZ_: question is, how much of this do we want to expose to the testers 03:35:32 NavidZ_: not testing parts of the browser 03:35:42 NavidZ_: some apis already that can expose that 03:35:53 NavidZ_: if we want to go down that path, do it all the way 03:36:03 NavidZ_: on windows, behaves differently than linux on mac 03:36:22 NavidZ_: chromium tests only, test can choose based on platform 03:36:32 NavidZ_: do we want to expose that in testdriver? 03:36:51 NavidZ_: focus next: can't get that without user interaction. difference between platforms 03:37:31 NavidZ_: adding automation for specific things, expose more and more of inner workings of the browser 03:37:49 foolip: do we want to add API for create user interaction defined in html? 03:37:58 ato: big discussion in browser testing and tools 03:38:14 Hexcles has joined #testing 03:38:18 03:38:31 s/// 03:40:03 zcorpan has joined #testing 04:01:30 cybai has joined #testing 04:02:46 spectranaut has joined #testing 04:06:13 denis has joined #testing 04:07:07 cybai_ has joined #testing 04:12:36 cybai has joined #testing 04:14:50 Hexcles has joined #testing 04:20:21 kevers has joined #testing 04:24:08 (14wpt) [PR] dirkschulze requested 13#19074 merge into 07master: Add smfr as reviewer for CSS Transforms - https://git.io/JeOTN 04:24:11 denis has joined #testing 04:24:30 prushforth has joined #testing 04:25:03 present+ Peter Rushforth, Natural Resources Canada, Observer 04:30:56 whsieh has joined #testing 04:31:02 tidoust has joined #testing 04:34:28 zcorpan has joined #testing 04:40:23 (14wpt) [PR] dirkschulze 03merged 13#19074 into 07master: Add smfr as reviewer for CSS Transforms - https://git.io/JeOTN 04:42:45 RRSAgent: make minutes 04:42:45 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html zcorpan 04:45:11 (14wpt) [PR] moz-wptsync-bot requested 13#19075 merge into 07master: [Gecko Bug 1579993] Add WPT subgrid tests and a few regular Grid baseline alignment tests. - https://git.io/JeOkz 04:45:34 marcosc has joined #testing 04:45:57 participants & their position to help with scribing https://docs.google.com/spreadsheets/d/1cqPK6ze2OCLsho4twJHNLZUPktfejIiiDlMwv0TaZBg/edit#gid=0 04:46:46 Hexcles has joined #testing 04:51:20 zghadyali has joined #Testing 04:56:20 plh has joined #testing 05:00:41 cybai has joined #testing 05:01:54 Hexcles has joined #testing 05:04:52 saschanaz has joined #testing 05:07:40 RRSAgent: make minutes, please 05:07:40 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 05:07:41 jorydotcom has joined #testing 05:08:25 marcosc has joined #testing 05:09:29 Just added https://www.w3.org/wiki/TPAC/2019/SessionIdeas#web-platform-tests_update_.26_discussion 05:09:29 denis_ has joined #testing 05:10:11 CalebRouleau has joined #testing 05:10:23 present+ Caleb Rouleau, Google 05:10:55 present+ 05:10:57 ScribeNick: ato 05:11:15 cb has joined #testing 05:11:34 present- Google 05:11:40 Topic: Infra: making full use of the test results on PRs 05:11:55 foolip: PR checks are a bit noisy, so at the moment you want to ignore them. Fourteen in total. 05:12:11 ... Flaky tests also. Should we do something about them? 05:12:25 jgraham: Let's talk about what is there already. 05:12:44 Example: https://github.com/web-platform-tests/wpt/pull/19067 05:12:57 RRSAgent, make minutes 05:12:57 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 05:13:12 whsieh has joined #testing 05:13:19 present- Natural_Resources_Canada 05:13:28 present- Observer 05:14:37 (14wpt) [PR] dirkschulze 03merged 13#11169 into 07master: Remove tests for SVG transform with CSS syntax - part 1 - https://git.io/JeOk6 05:14:44 foolip: The Azure pipeline is noisy, just produces results for Safari. 05:14:53 foolip: This is a request for the Azure team, to make this less noisy. 05:15:08 JohnJansen: You'd like the report to be merged into one? All the pipelines into one? 05:15:26 gsnedders: In GitHub checks it makes sense for them to be separate? 05:15:38 foolip: Talking to Microsoft about this makes sense I think. 05:16:15 jgraham: With TaskCluster you have to click through to get the specific URL for the job. 05:16:26 ... This is not natural to everyone. 05:16:43 ... I think switching that to checks will make it noisy, but may make it easier to figure out what's failing. 05:17:07 ... A decision task would help also, because it would just run the tasks that are relevant. 05:17:08 ACTION: JohnJansen follow up with Azure Pipelines team for this 05:17:14 ... Rather than the tasks we have decided to run for you. 05:17:31 [talk about priorities] 05:18:17 jgraham: With a decision task [on Taskcluster] it will run only the things that are dependencies, such as lints when source file changes, but essentially just the test jobs related. 05:18:27 cybai_ has joined #testing 05:18:40 jgraham: We've run up quite hard against the limits of GitHub. 05:18:51 jgraham: Checks vs. non-blocking checks 05:19:40 foolip: Final part, wpt.fyi. 05:20:04 foolip: Sometimes there are more wpt.fyi non-blocking checks, due to deployment etc. 05:20:07 cb_ has joined #testing 05:20:19 ... Could we merge this into a single check somehow? 05:20:35 lukebjerring: If it's desirable we could reuse the same name and aggregate the information. 05:21:00 lukebjerring: Pending results will tie into these, so we will have pending checks whilst the process [?] is still ongoing. 05:21:19 foolip: Sometimes I see the same problem in all three. 05:21:29 jgraham: I've never clicked on these that have neutral status. 05:21:38 jgraham: So I don't think it’s an effective signal to give users. 05:22:19 s/process [?]/result processor/ 05:22:36 ... Being able to look at the Firefox results specifically has been useful, but I don't think that the neutral results provide any value. 05:22:48 foolip: How do we indicate this in a better way? Make it fail? 05:23:07 jgraham: Failing with a button to "un-fail" it might be OK, but confusing. 05:23:15 ... Maybe submit an issue on the code, somewhere? 05:23:19 foolip: Review comment? 05:23:21 karl has joined #testing 05:23:39 jgraham: "This test appears to be erroring in this browser, if that’s fine you should dismiss this review and accept the PR." 05:24:19 https://developer.github.com/v3/checks/runs/ has screenshot at top of possible outcomes 05:24:29 lukebjerring: If you click on the details for the wpt.fyi job, there’s a recompute button right now, and for the case that is neutral which I would hope to upgrade to blocking, it would add a comment saying which user clicked the button. 05:24:39 ... "Luke marked this as passing using the ignore button" 05:25:20 foolip: There are a few different options, a red triangle and provide text. 05:25:38 foolip: I suppose it wouldn’t be blocking then. We could decide not to make it blocking, I mean. 05:26:05 jgraham: I’m happy to experiment with stuff here, but I’m very cynical about how much people writing these tests are going to care to dig into the issues. 05:26:20 ... At least initially people will be whining because their PRs are stuck and don’t bother investigating. 05:26:28 rego has joined #testing 05:26:36 foolip: Low-frequency but serious solution we’re willing to experiment with? 05:26:48 jgraham: No apss everywhere. 05:27:27 foolip: Let's go with "consistent error" everywhere. 05:27:31 s/apss/pass/ 05:27:53 foolip: A lot of tests that Chrome adds are harness errors in Safari. 05:27:54 jesopo has joined #testing 05:28:11 jgraham: Maybe it should be more cynical about things that come from browser syncs. 05:28:29 ... A test for an API that only exists in Chrome might be problematic. 05:28:34 ACTION: lukebjerring Hexcles: explore setting wpt.fyi check results to failures when e.g. tests error everywhere 05:29:21 lukebjerring: It has to previously been failing and now passing at the moment, and I think it would be more useful to make it blocking than neutral. 05:30:01 lukebjerring: We have merged PRs that failed tests, but they were neutral in the PR and didn't check. We had to unroll a big list of changes. 05:30:17 zcorpan has joined #testing 05:30:26 jgraham: There needs to be some mechanism whereby the author has to say that “this is what I intended”. 05:30:41 ... Pass going to fail happens often. The test could’ve been wrong. 05:30:58 q+ 05:31:05 ... The test might’ve been passing in all browsers, but is now always failing. 05:31:43 lukebjerring: If you fix a test that was passing incorrectly and becomes failing, you’ll be blocking becaus ethe statistics will be wrong [?] 05:32:04 gsnedders: It’s not the job failing within the Chromium CI system. 05:32:28 jgraham: People are less invested in gettnig their change landed in Chromium than they are in WPT. 05:32:38 jgraham: Or is your CL blocked until it lands in WPT? 05:32:46 lukebjerring: That’s a discussion we want to have today. 05:32:48 q? 05:33:00 q- 05:33:11 lukebjerring: We want to propagate that into the original CL before it lands, before the export happens. This reduces human manual intervention. 05:33:31 jgraham: [explains Mozilla process] 05:34:29 lukebjerring: Forcing someone to click an extra button is worth it balanced against the pain it puts on us later. 05:34:38 xiaoqian has joined #testing 05:34:55 ... Because of the privilege that WPT is given, it is important for WPT not to land into a bad state. 05:35:16 jgraham: Occasionally we land something that fails all our tests. 05:35:29 ... Coming up with good heuristics is hard, but I agree it’s a problem. 05:35:40 ... The problem is that this is a system platform developrs don’t interact with often. 05:35:50 q+ 05:36:37 q+ 05:36:37 ato: Should upstream PRs be scrutinised harder? 05:37:04 jgraham: Maybe that case should have different heuristics. 05:37:29 ack CalebRouleau 05:37:49 CalebRouleau: Whitelist of things that is supposed to be passing? 05:37:59 CalebRouleau: This would be a change in the code. 05:38:15 jgraham: The expectation metadata could be put in WPT perhaps. 05:38:22 ... But it would meet a lot of resistance. 05:38:26 Hexcles: I agree. 05:38:50 jgraham: It would be like for the infra tests. You would have to go and update the expectations when you mean for something to change. 05:39:00 jgraham: It would work, because that’s how we do it for Gecko. 05:39:09 estes has joined #testing 05:39:32 CalebRouleau: The proposal was _whitelist_, not expectation data for all. 05:39:55 q- 05:40:06 lukebjerring: You’re going to end up exhaustively listing the metadata anyway, and you’ll want this per-browser basis. 05:40:31 jgraham: It would also increase the workload on authors because they would have to update the metadata also for other web browsers. 05:41:01 lukebjerring: Someone can submit something to the codebase and suddenly everything fails, without any warning. 05:41:37 q? 05:41:39 jgraham: Human intervention needed at some level. 05:41:52 ... Different rules for different directories? 05:43:11 ato: It’s not unprecedented that tests are wrong and we make them go from pass to fail expecedtly. 05:43:39 jgraham: [explains a recent case] 05:45:10 foolip: if you are heavily involved you can become a reviewer or even codeowner on github 05:45:57 lukebjerring: I think it’s reasonable to ask people to explain why they are making a test go from pass to fail. 05:46:48 ... It’s easy to demonstrate why we would impose a blocking check for it that people would understand. 05:47:01 jgraham: I would need to see statistics to see that it’s usually problematic [?] 05:47:51 ... Another case is where someone adds a test that passes only in a single browser, but times out or fails in all others. 05:48:06 lukebjerring: Do we make these a failing action on GitHub and allow it to be ignored? 05:48:19 ... And when a regression happens, they are propagated into the Firefox CI? 05:48:53 jgraham: We’re starting up a project so that we can hopefully surface this stuff as feedback in the code review, as opposed to finding out after a change lands. 05:49:03 CalebRouleau has joined #testing 05:49:25 ... I think we can surface this stuff to developers earlier and get them to look at it more readily. 05:49:49 ... It’s unclear what the best mechanism for the feedback in WPT on GH should be. 05:50:37 ... If you choose to ignore you have to give a reason I think. 05:50:41 ... With checks you can't maybe. 05:51:15 foolip: The GH checks are made for this, they have a desired outcome and you have to give a reason. 05:51:21 foolip: That flow is sort of built for this. 05:51:33 jgraham: I’m unconvinced they have got the UI right. 05:51:38 foolip: Well we haven’t tried it out. 05:52:23 jgraham: OK, maybe we should try the checks thing first, otherwise try the code review thing. 05:52:38 Hexcles: We could have different rules for different directories also. 05:53:10 jgraham: If we could initially roll it out for directories with developers we know are engaged. 05:53:43 foolip: If we see the flow in a PR, maybe this is going to be easier to assess. 05:54:45 lukebjerring: I do a deliberate regression PR to wpt.fyi for this 05:55:04 RRSAgent: make minutes 05:55:04 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 05:56:25 RRSAgent: Make minutes 05:56:25 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 05:57:28 ACTION foolip: write an RFC for making regressions detected by wpt.fyi require actions 05:57:40 foolip: Understanding the flakiness is super-hard. 05:58:07 foolip: We could make the logs less verbose, but then we’d have to increase it to find out what’s wrong. 05:58:22 jgraham: We have a log handler that picks out the things it thinks is important. 05:58:49 hyojin has joined #testing 05:59:26 lukebjerring: Custom interpretation jobs is what wpt.fyi is doing by definition. 05:59:55 cb has joined #testing 05:59:57 RRSAgent make minutes 06:00:31 lukebjerring: Instead of designing TaskCluster to have custom log interpretation, we should do this in wpt.fyi. 06:01:07 jgraham: But if we had this in TaskCluster, we would use this consistently also for other things. It could produce an artifact we could reuse elsewhere. 06:02:14 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html gsnedders 06:03:53 ACTION: Hexcles: switch Taskcluster to GitHub Checks 06:04:51 jgraham: [explains TaskCluster] 06:06:46 RRSAgent, draft minutes 06:06:46 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 06:07:47 jgraham: Regarding flakey tests, recently the expected test status at Mozilla have support for multiple test statuses. 06:07:59 ... For example, this test can either pass or fail. 06:08:19 zghadyali has joined #testing 06:08:27 lukebjerring: What is the main reason you’re against having expectation data upstream in WPT? 06:08:48 jgraham: On WPT we’re running things mostly on one platform. 06:08:53 ... On Gecko we’re running a vast number of platforms which is specific. 06:09:25 ... This could cause double work because you would have to care for other browsers’ expectation data in addition to your own. 06:10:25 Hexcles: There's a more fundamental problem: browser versions matter. 06:10:43 Hexcles: If you have an upstream expectation that applies to a specific version, it will be even more difficult. 06:11:12 present+ 06:13:02 lukebjerring: There's an implicit falkiness if you have multiple acceptable statuses, and having a totally separate place to say soemthing is flaky seems kind of bad because it duplicates information. 06:15:36 jgraham: If you had the WPT flakiness data in tree you could build some interesting tooling. 06:16:02 jgraham: "You’ve marked this test as not-flaky in Chrome, but it’s still flaky elsewhere.” 06:16:34 foolip: Flakiness on master vs. flakiness on PRs? 06:17:20 cb has joined #testing 06:17:26 lukebjerring: We run a cron job that looks at the last ten runs and checks for flaky tests and lets some person know about it. 06:18:08 jgraham: Can we have a checks page? 06:18:14 form on* 06:18:22 lukebjerring: Recompute, ignore 06:18:54 ... You could also have "flag as flaky" which would change the metadata and re-run the computation, say. 06:19:42 ... If you have a flaky test on TC and people aren’t bothered looking into it, they are already ignoring this and force merging. 06:19:49 ... So it would not make the current situation any worse. 06:20:11 foolip: I don’t think we necessarily have anything we disagree on with regards to flaky tests. 06:21:37 JohnJansen: Explanation on how to deal with flaky tests. 06:21:57 cybai has joined #testing 06:22:06 lukebjerring: Documentation for “so you’ve been told your tests are flaky” sounds like a good idea. 06:22:27 Hexcles: A tutorial linked from the GH checks. 06:23:08 foolip: If we have a button that marks as flaky, you can be sure people are going to click it. 06:23:17 ... This will eventually make the system useless if overused. 06:23:31 lukebjerring: It’s hard to identify false-positives. 06:23:54 jgraham: If you haven’t seen flakiness in the last month, then we probably don’t care. 06:24:07 jgraham: You can remove expectation data when the flakiness goes away. 06:24:32 cb has joined #testing 06:25:02 jgraham: In Chromium you run every test on every commit, you can get backed out if the test becomes flaky as a result of the CL. 06:25:29 ... In the Gecko case it matters less if the metadata is a little bit inaccurate. 06:26:28 Topic: Python 3 06:26:53 jgraham: Python Foundation are stop going to stop maintaining Python 2 sometime next year. 06:27:07 ... There is no need for immediate panic because RedHat will continue providing updates for another four years. 06:27:20 ... But we should have a plan for migration to Python 3. 06:27:38 s/are stop/are/ 06:27:58 ... Gecko is starting to move things to Python 3 slowly, and there is increased need for us to have a roadmap for this as well. 06:28:21 Hexcles: There was no conclusion what WebKit is going to do based on the email thread. 06:28:43 jgraham: Previously we had assumed that WebKit was a blocking concern. 06:29:12 ... But now it looks like they are switching to Python 3, or possibly going to stop shipping Python altogether. 06:29:18 Hexcles: macOS 10.11 06:29:30 Hexcles: We should find someone from the WebKit community. 06:29:33 s/10.11/10.15 06:29:43 s/.11/10.15/ 06:29:57 RRSAgent: make minutes 06:29:57 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 06:31:07 jgraham: We need support for both Python 2 and 3. 06:31:39 ... For example making the WPT frontend run in either, then make the commands it despatches to run Python 3. 06:32:10 ... That seems to be the way Gecko works. 06:32:29 Break. 06:33:44 tidoust has joined #testing 06:35:10 Hexcles has joined #testing 06:40:23 (14wpt) [PR] autofoolip requested 13#19076 merge into 07master: Update interfaces/IndexedDB.idl - https://git.io/JeOIW 06:40:29 (14wpt) [PR] autofoolip requested 13#19077 merge into 07master: Update interfaces/gamepad.idl - https://git.io/JeOI8 06:40:33 (14wpt) [PR] autofoolip requested 13#19078 merge into 07master: Update interfaces/geometry.idl - https://git.io/JeOI4 06:40:37 (14wpt) [PR] autofoolip requested 13#19079 merge into 07master: Update interfaces/webmidi.idl - https://git.io/JeOIB 06:40:41 (14wpt) [PR] autofoolip requested 13#19080 merge into 07master: Update interfaces/webrtc-stats.idl - https://git.io/JeOIR 06:42:33 plh has joined #testing 06:43:45 xiaoqian has joined #testing 06:44:50 spectranaut has joined #testing 06:46:06 zcorpan has joined #testing 06:56:06 cybai has joined #testing 07:02:10 Manishearth has joined #testing 07:02:41 Hexcles has joined #testing 07:04:13 Topic: WebXR 07:04:23 RRSAgent: make minutes 07:04:23 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 07:05:05 A testing API exists: https://github.com/immersive-web/webxr-test-api 07:05:14 CalebRouleau has joined #testing 07:05:57 RRSAgent: this meeting spans midnight 07:05:58 karl has joined #testing 07:05:59 (14wpt) [PR] foolip 03merged 13#19076 into 07master: Update interfaces/IndexedDB.idl - https://git.io/JeOIW 07:05:59 present+ 07:06:06 RRSAgent: listen 07:06:07 present+ 07:06:10 RRSAgent: make minutes 07:06:10 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 07:06:12 present+ 07:06:13 JohnChen has joined #testing 07:06:17 mounir has joined #testing 07:06:20 present+ 07:06:21 Present+ 07:06:41 jgraham: How to test things that you can't model to simple interaction. 07:07:03 ... WebXR is an example of a spec implemented in more than just Chrome, that the tests won't work in other browsers because they have this Mojo crap in them. 07:07:07 https://github.com/immersive-web/webxr-test-api/blob/master/explainer.md 07:07:13 ... The question is what is the testing strategy for WebXR. 07:07:31 Present+ mounir 07:07:39 Present+ Manishearth 07:07:45 (14wpt) [PR] foolip 03merged 13#19078 into 07master: Update interfaces/geometry.idl - https://git.io/JeOI4 07:07:50 mounir: There is a testing API in Chrome. 07:08:01 ... The backend of that in Chrome is using Mojo. 07:08:08 ... So this is not directly exposed to test code. 07:08:23 ... The solution we have is to have an internal API or something. 07:08:36 Manishearth: There was a testing API proposal that was out of date, and no one implemented it. 07:08:37 jorydotcom has joined #testing 07:08:45 (14wpt) [PR] foolip 03merged 13#19080 into 07master: Update interfaces/webrtc-stats.idl - https://git.io/JeOIR 07:08:50 RRSAgent, make minutes 07:08:50 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html JohnJansen 07:08:52 ... There were WPT that had a utils folder that did include specific things, but the tests were written on a shared API. 07:08:56 ... It was backed by some Mojo sstuff. 07:09:19 ... We implemented a new API for testing that has a backend in Chrome. 07:09:28 ... So that's why there's still Mojo in there. 07:09:44 ... We're able to run the tests just fine [in Firefox] because we have a native implementation of the API. 07:10:14 jgraham: I heard an expectation that Mojo had to load, and the tests would fail irrespective of what the browser did. 07:10:31 Manishearth: I've certainly been able to run the tests without Mojo on Servo with success. 07:11:46 ... In Servo it's a regular WebIDL that we expose with a pref when needed. 07:12:04 foolip: We haven't figured out how to get this to work in Chrome on regular Chrome build. 07:12:24 jgraham: Do the WebXR people have any needs from us? 07:13:15 mounir: We are challenged about where to put the Mojo bits [?] 07:13:34 jgraham: If you want us to install a magic extension we could facilitate that. 07:14:04 Hexcles: This seems like a very Chrome specific problem. 07:14:05 (14wpt) [PR] chromium-wpt-export-bot requested 13#19081 merge into 07master: [LayoutNG] Allow overflow-/word-wrap to work with keep-all - https://git.io/JeOIp 07:14:20 ... We're discussing archiving and fetching Mojo for testing. 07:14:38 jgraham: If you want to set a pref, put a file on the filesystem, or install an extension we could do that. 07:14:59 ... For Gecko tests you can use internal APIs by loading a web extension first. 07:15:42 Hexcles: There are some challenges lining up the right Mojo version with the right Chrome version. 07:15:53 ... We would need to map it on a revision by revision basis. 07:16:10 jgraham: It sounds like there are no fundamental WPT issues here. 07:16:40 lukebjerring: We build the chromedriver binary [unsigned?] in [some Google system]. 07:17:09 Hexcles: We will need to add some logic to WPT to figure out the URL to fetch Mojo from. 07:17:14 ... I suppose there's no objection to that. 07:17:24 jgraham: We're happy to do browser specific stuff for some tests. 07:18:00 mounir: Why don't you guys use a content shell for testing? 07:18:17 jgraham: The question is how representative it is of the user experience. 07:18:26 Hexcles: Internally we're moving towads running the full browser. 07:18:39 jgraham: It used to be the case that you couldn't run chromedriver against content shell. 07:18:43 Hexcles: It's supposed to work. 07:18:46 marcosc has joined #testing 07:19:03 JohnChen: I can't speak to whether it works today, but it's meant to. 07:20:09 ato: The complication with Firefox is that it reads a bunch of prefs at startup time. 07:20:23 Manishearth: Is this the first time such a testing API is implemented? 07:20:39 jgraham: I think it is for something that is tested cross-browser. 07:20:49 Manishearth: It's not that we've made a grave mistake? 07:20:51 jgraham: No. 07:22:11 mounir: What is the time line for moving to wptrunner for the Chrome infrastructure, so we run full builds? 07:22:25 foolip: I don't know the timeline, but we're working on it and have for some time. 07:23:39 [discussion about how to fetch the latest Chrome] 07:24:22 gsnedders: It would be useful to have Chromium nightly builds running in WPT. 07:24:50 Ms2ger has joined #testing 07:25:07 ^ That's unsigned Chromium, where EME etc. does not work 07:25:22 https://download-chromium.appspot.com/ provides chromium builds 07:25:58 foolip: Just to fix the Mojo problem, there has to be some JS to inject the Mojo files... 07:26:21 jgraham: We could mark certain directories to require the Mojo stuff or something. 07:27:29 ... Unfortunately for the prefs stuff, this is encoded in the Mozilla metadata and this is not upstreamed to WPT. 07:27:52 ... But there is an argument that this could be upstreamed, because at the moment you will occassionally see differences when running Firefox tests upstream. 07:28:48 RRSAgent: make minutes 07:28:48 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 07:31:24 Topic: Python 3 07:31:33 whsieh has joined #testing 07:31:38 CalebRouleau has joined #testing 07:32:23 whsieh: Older versions of macOS might not have Python 3 installed. 07:32:54 ... It might still be years before we can drop Python 2. 07:33:12 jgraham: We need a strategy for us to move to Python 3 in a finite amount of time. 07:33:35 ... But we can't move to Python 2. 07:33:49 ... Does the manifest generation work in 3 now? 07:33:55 gsnedders: It generates a completely different manifest in 2 and 3. 07:34:09 gsnedders: Is Python 2 going to be maintained past 2024? 07:34:30 jgraham: Gecko is moving away from Python 2, but it's going to be years. 07:34:44 ... There are for example also dependencies on wptrunner. 07:35:15 foolip: So wptrunner can support only 3? 07:35:26 jgraham: I think it needs to support 2 and 3 for some time. 07:35:57 foolip: It would be nice if wptrunner keeps working on Python 2 on older Macs. 07:36:05 ... So you don't have to download anything special. 07:36:13 jgraham: That has historically been a requirement. 07:36:24 Hexcles: It sounds like this is not an urgent matter. 07:36:32 zghadyali has joined #testing 07:36:33 gsnedders: Until Apple stops shipping Python 2. 07:37:57 jgraham: In Gecko, all new commands has to be Python 3. 07:38:05 ... In practice I'm not sure if it matters. 07:38:14 ... But there's a push to move to Python 3. 07:38:45 ... One first step would be to run the infrastructure tests in both versions. 07:38:58 ... We should maybe start writing new code in Python 3? 07:39:11 ... For example, require entry-points to be Python 2+3 compatible. 07:39:21 Hexcles: There's currently no incentive. 07:39:41 ... Every vendor seems to be postponing the migration indefinitely. 07:39:47 jgraham: Not sure that is the case for Mozilla. 07:41:02 ... wptrunner is Python 2 only. 07:41:24 ... The web server stuff works on 3, but the handler scripts might be fine but no one has checked. 07:41:28 Hexcles: They are definitely not fine! 07:42:10 (14wpt) [PR] chromium-wpt-export-bot requested 13#19082 merge into 07master: [webnfc] Add tests for NFCPushOptions.ignoreRead - https://git.io/JeOL8 07:42:31 jgraham: For new entry-points we could require 3. 07:42:45 ... That's not a big ask, because people add these relatively seldom. 07:43:13 ... Once we get the manifest generator to generate the same results in 2 and 3, there should be a unit test for the behaviour. 07:43:25 Hexcles: Doesn't sound like there's a modular approach. 07:43:57 gsnedders: Manifests does a lot of string manipulation, but there is less string conversion happening in wptrunner. 07:44:14 jgraham: I've started getting random patches for this. 07:44:41 ... Often I'm scared of accepting these because testing is hard. 07:46:05 Hexcles: It's hard to modularise wptrunner. 07:46:25 ... Someone needs to spend time to make it work on Python 3, then have integration tests for the Python 3 fixes. 07:46:47 jgraham: One first step would be to get it to import cleanly in Python 3 without SyntaxErrors. 07:46:56 Hexcles: Do you have an estimate? 07:47:08 jgraham: It's not a small amount of work. 07:47:12 gsnedders: There's a long tail of work. 07:47:25 jgraham: I can imagine some team at Mozilla might get an intern to do this. 07:47:59 ... There seems to be some agreement that we need to do this work, and that it's acceptable to stand up tests for the Python 3 behaviour. 07:48:19 ... I'm saying there are people working on this, and we should support the people doing work on this. Not that we should do the work right now. 07:48:27 RRSAgent, make minutes 07:48:27 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html MikeSmith 07:48:38 ... And maybe in two years macOS might be more of a force to dictate further progress. 07:48:49 Manishearth has left #testing 07:49:27 MikeSmith: On macOS, homebrew installs Python 3 by default. 07:49:41 ... This is a huge hurdle for contributors to WPT. 07:49:49 ... Because it overrides the system default Python 2. 07:50:08 (14wpt) [PR] chromium-wpt-export-bot 03merged 13#19081 into 07master: [LayoutNG] Allow overflow-/word-wrap to work with keep-all - https://git.io/JeOIp 07:50:33 Doubt in the room of whether this is correct. 07:53:06 gsnedders: The constraint comes from the WebKit community, who are opposed to installing any other software on the system. 07:53:19 MikeSmith: Increasingly there are more and more brew packages relying on Python 3. 07:54:33 [discussion about misguided Linux distributions about how they are shipping Python] 07:57:09 MikeSmith: https://docs.brew.sh/Homebrew-and-Python#python-3x-or-python-2x says it souldn't 07:58:20 gsnedders: The manifest migration is easily a month's work. 07:59:12 ... There are performance challenges involved. 07:59:28 ... And hard to do without making a complete mess of it. 07:59:58 [technical discussion about type annotations] 08:00:22 RRSAgent: make minutes 08:00:22 I have made the request to generate https://www.w3.org/2019/09/16-testing-minutes.html ato 08:00:38 RRSAgent: stop