IRC log of testing on 2019-09-17
Timestamps are in UTC.
- 00:43:47 [RRSAgent]
- RRSAgent has joined #testing
- 00:43:47 [RRSAgent]
- logging to https://www.w3.org/2019/09/17-testing-irc
- 00:44:01 [zcorpan]
- RRSAgent: this meeting spans midnight
- 00:44:09 [JohnChen]
- present+
- 00:44:15 [boaz]
- boaz has joined #testing
- 00:44:15 [Hexcles]
- present+
- 00:44:22 [boaz]
- present+
- 00:44:29 [AutomatedTester]
- present+ David Burns, Mozilla
- 00:44:36 [jorydotcom]
- present+ Jory Burson
- 00:44:38 [zcorpan]
- present+
- 00:44:42 [zcorpan]
- present- Mozilla
- 00:44:47 [boaz]
- present+ Boaz Sender, Bocoup
- 00:44:54 [JohnJansen]
- present+
- 00:45:01 [Lan]
- Lan has joined #testing
- 00:45:01 [zcorpan]
- present- Bocoup
- 00:45:05 [jorydotcom]
- jugglinmike are you dialed in?
- 00:45:07 [MikeSmith]
- present+
- 00:45:10 [NavidZ_]
- present+ NavidZ
- 00:45:11 [Hexcles]
- RRSAgent, draft minutes
- 00:45:11 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html Hexcles
- 00:45:13 [foolip]
- jugglinmike: you can try joining now
- 00:45:18 [Lan]
- present+
- 00:45:28 [MikeSmith]
- RRSAgent, make logs public
- 00:45:40 [JohnJansen]
- RRSAgent, make minutes
- 00:45:40 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 00:46:05 [JohnJansen]
- RRSAgent, Title: Browser Testing and Tools, TPAC 2019
- 00:46:05 [RRSAgent]
- I'm logging. I don't understand 'Title: Browser Testing and Tools, TPAC 2019', JohnJansen. Try /msg RRSAgent help
- 00:46:15 [MikeSmith]
- Present+ JoryBurson
- 00:46:25 [MikeSmith]
- Present- Burson
- 00:46:32 [MikeSmith]
- Present- Jory
- 00:46:32 [JohnJansen]
- Topic: Test Automation
- 00:46:42 [MikeSmith]
- RRSAgent, make minutes
- 00:46:42 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html MikeSmith
- 00:46:50 [JohnJansen]
- Title: Browser Testing and Tools, TPAC 2019
- 00:47:05 [foolip]
- jgraham: talked with WebXR people yesterday, they're seemingly the first to define a test-only API which has multiple implementations
- 00:47:08 [MikeSmith]
- RRSAgent, make minutes
- 00:47:08 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html MikeSmith
- 00:47:13 [AutomatedTester]
- scribenick: foolip
- 00:47:36 [foolip]
- jgraham: there might also be tests that break for the wrong reason, like assuming that the mojo stuff is there
- 00:48:02 [foolip]
- jgraham: our intent, depending on foolip et al, is to make the mojo part transparent so that you don't have to explictly included
- 00:48:18 [foolip]
- jgraham: but maybe some metadata somewhere to say for which directories that injection needs to happen
- 00:48:43 [foolip]
- jgraham: in the end, it'd work as if you set a pref and get a test-only API
- 00:49:11 [JohnJansen]
- Meeting: Web-Platform-Tests TPAC 2019
- 00:49:23 [foolip]
- jgraham: that probably covers it for mojo
- 00:49:37 [reillyg]
- present+
- 00:49:47 [JohnJansen]
- RRSAgent, make minutes
- 00:49:47 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 00:49:50 [odejesush]
- present*
- 00:50:20 [foolip]
- reillyg: want to say sorry, my work was the first to use mojo, and other have copied that. some haven't hidden their use of mojo as much
- 00:50:34 [foolip]
- reillyg: question of how such tests should fail if the browser doesn't support the API or mojo
- 00:51:03 [foolip]
- reillyg: if we can get infra improvements to get a guide for how this should work, then that's the right way forward
- 00:51:29 [foolip]
- reillyg: more long term, APIs like this should probably be webdriver extensions, but open questions about capabilities of it
- 00:51:44 [foolip]
- reillyg: but as long as we have general agreement about how to structure to not cause issues, that's fine
- 00:52:00 [MikeSmith]
- Present+ odejesush
- 00:52:07 [ato]
- Present+
- 00:52:14 [foolip]
- jgraham: if we can get to the point where we don't need an explicit mojo include, we'd just lint to check that you don't
- 00:52:34 [JohnJansen]
- RRSAgent, Title: Browser Testing and Tools, TPAC 2019
- 00:52:34 [RRSAgent]
- I'm logging. I don't understand 'Title: Browser Testing and Tools, TPAC 2019', JohnJansen. Try /msg RRSAgent help
- 00:52:42 [ato]
- Meeting: Web Platform Tests, Day 2, TPAC 2019
- 00:52:45 [foolip]
- reillyg: the pattern you can see in wpt isn't the way it should be
- 00:53:01 [foolip]
- reillyg: the most contentious was scripts just included mojo using <script>
- 00:53:09 [ato]
- RRSAgent: make minutes
- 00:53:09 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 00:53:32 [foolip]
- reillyg: but for scripts that detect if mojo is available, that then loads mojo stubs, is that acceptable, or should impl and files be moved to elsewhere?
- 00:53:42 [foolip]
- q?
- 00:53:45 [foolip]
- q+ boaz
- 00:53:47 [boaz]
- q+
- 00:54:02 [JohnJansen]
- q?
- 00:54:04 [wanming]
- wanming has joined #testing
- 00:54:15 [foolip]
- jgraham: it should be possible to make it almost completely transparent, so that you can just use the test API
- 00:54:25 [Zakim]
- Zakim has joined #testing
- 00:54:29 [foolip]
- jgraham: in the short term the pattern of check if mojo is available seems reasonable
- 00:54:31 [zcorpan]
- q+ boaz
- 00:55:01 [foolip]
- jgraham: a pattern that we can point to as "this is what you should do" then that's a good step on the way
- 00:55:24 [reillyg]
- q+
- 00:55:24 [foolip]
- Hexcles: sounds about right
- 00:55:50 [foolip]
- Hexcles: general principle is that /gen/ shouldn't be checked into wpt, and we're looking at ways at archiving and releases those somewhere, along with Chromium
- 00:55:57 [ato]
- RRSAgent: make minutes
- 00:55:57 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 00:56:03 [foolip]
- Hexcles: as for impl of test APIs, impl itself can be in wpt I think
- 00:56:03 [BitBot]
- (14wpt) [PR] lukebjerring 03merged 13#19106 into 07master: Update interfaces/geometry.idl - https://git.io/JeOBX
- 00:56:05 [jorydotcom]
- JohnJansen did you also have a question
- 00:56:08 [MikeSmith]
- q?
- 00:56:15 [jorydotcom]
- ack boaz
- 00:56:15 [MikeSmith]
- ack boaz
- 00:56:23 [foolip]
- boaz: hi Mike
- 00:56:25 [zcorpan]
- q+
- 00:56:29 [ato]
- RRSAgent: This meeting spans midnight
- 00:56:37 [zcorpan]
- q?
- 00:56:41 [MikeSmith]
- ack reillyg
- 00:56:43 [boaz]
- q?
- 00:56:51 [plh]
- plh has joined #testing
- 00:57:17 [foolip]
- reillyg: only concern is I want to avoid it being too magical, figuring out how and why they got loaded was a bit of a journey
- 00:57:24 [foolip]
- reillyg: needs to be clearly at the top of a test
- 00:57:50 [yigu]
- yigu has joined #testing
- 00:58:09 [foolip]
- jgraham: in other browsers that don't do the same thing, it wouldn't be accurate
- 00:58:24 [foolip]
- reillyg: but everyone would have it implemented somehow, and pointing out how it works would be good
- 00:58:33 [foolip]
- jgraham: maybe a meta flag for the manifest
- 00:58:41 [foolip]
- jgraham: for gecko it'd be behind a pref
- 00:58:57 [foolip]
- reillyg: if I'm developing in gecko, I'd like to know what pref I have to enable
- 00:59:02 [foolip]
- q?
- 00:59:30 [jugglinmike]
- q+
- 00:59:31 [foolip]
- jgraham: we could do a <meta name="something"> to communicate
- 00:59:40 [ella]
- ella has joined #testing
- 00:59:53 [Hexcles]
- ack zcorpan
- 01:00:04 [reillyg]
- ack jugglinmike
- 01:00:26 [foolip]
- jugglinmike: last year part of the discussion was whether APIs would be exposed to web devs, a compelling reason to use webdriver
- 01:00:37 [foolip]
- jugglinmike: if in Chromium it's done by injecting code that won't happen
- 01:00:43 [foolip]
- jugglinmike: wonder if that's still a goal
- 01:00:56 [reillyg]
- q+
- 01:01:06 [foolip]
- jugglinmike: can we reach a state where we just run the browser with a flag?
- 01:01:22 [foolip]
- jgraham: agreement that where we can use webdriver that is better
- 01:01:29 [karl]
- karl has joined #testing
- 01:01:38 [foolip]
- jgraham: but some of these APIs can add significant binary bloat, increasing download size
- 01:02:14 [foolip]
- jgraham: adding to webdriver is always good, historically we have cases tested internally that should have been tested using webdriver
- 01:02:39 [foolip]
- jgraham: but there will always be cases where it's not possible or expedient to figure out the webdriver API for a specific thing
- 01:02:42 [foolip]
- q+
- 01:03:03 [foolip]
- jgraham: probably makes sense to have some test-only APIs in some cases long term
- 01:03:31 [foolip]
- reillyg: the use of mojo is out of expedience and development speed, adding webdriver surface is a lot of work
- 01:03:34 [boaz]
- q+
- 01:03:54 [foolip]
- reillyg: first cut using mojo will be easier. would like to see webdriver for some of these things, but value isn't there for all of them
- 01:04:05 [james]
- james has joined #testing
- 01:04:18 [foolip]
- reillyg: for webusb, what I need to test in depth is very different to what a web developer needs to test their code
- 01:04:34 [foolip]
- reillyg: I want to mock the underlying OS, they want do mock the web platform
- 01:05:07 [reillyg]
- s/mock the web platform/mock user interaction with permissions UI/
- 01:05:10 [foolip]
- jgraham: for webxr, in the long term probably want some automation solution, but hard to get that done quickly
- 01:05:29 [foolip]
- q?
- 01:05:36 [foolip]
- q- reillyg
- 01:05:40 [xiaoqian]
- xiaoqian has joined #testing
- 01:05:43 [Hexcles]
- ack foolip
- 01:06:13 [foolip]
- foolip: what guidance should we give to people who want to know what path to take: webdriver of test-only API?
- 01:06:32 [foolip]
- jgraham: I think, prefer webdriver, and if you're doing test-only API it needs to have a spec
- 01:06:49 [foolip]
- jgraham: it's not an invitation to expose internal guts that can't be implemented by anyone else
- 01:07:10 [foolip]
- jgraham: haven't had cases of multi-vendor interest until now, webxr being the first.
- 01:07:23 [zcorpan]
- ato: we can use the same https://docs.google.com/spreadsheets/d/1cqPK6ze2OCLsho4twJHNLZUPktfejIiiDlMwv0TaZBg/edit?usp=sharing
- 01:07:38 [foolip]
- jgraham: when we do get more interest, I'm sure we'll get feedback on test-only APIs accidentally relying on internal details
- 01:07:58 [Hexcles]
- ack boaz
- 01:07:59 [foolip]
- jgraham: you should look to get as much feedback on your test-only API as the rest of the spec
- 01:08:30 [foolip]
- boaz: in Chrome, how do you decide whether to pick webdriver or test-only API?
- 01:08:40 [foolip]
- foolip: we don't have guidance and don't know, that's why I ask :)
- 01:09:33 [foolip]
- reillyg: writing the test-only API was informative for crystalizing the behavior I wanted, recommend doing that
- 01:09:36 [foolip]
- q?
- 01:09:41 [jugglinmike]
- q
- 01:09:55 [jugglinmike]
- rather
- 01:09:56 [jugglinmike]
- q+
- 01:10:56 [foolip]
- jgraham: it would be *nice* to know we don't depend on insufficiently defined API
- 01:10:59 [Hexcles]
- s/rather//
- 01:11:08 [foolip]
- jgraham: when the second impl comes along turns out you have to make a lot of changes
- 01:11:21 [JohnJansen]
- q?
- 01:11:24 [foolip]
- jgraham: I imagine we'll have the same problem with test-only APIs
- 01:11:34 [foolip]
- boaz: anything we can do to avoid that?
- 01:11:34 [ato]
- s/Web-Platform-Tests TPAC 2019/Web Platform Tests, Day 2, TPAC 2019/
- 01:11:44 [foolip]
- boaz: like not contributing tests that can only run in one browser?
- 01:11:55 [zcorpan]
- q?
- 01:11:56 [MikeSmith]
- q?
- 01:12:03 [foolip]
- jgraham: think that's difficult, deeper question is how valuable are tests for features with only one impl?
- 01:12:04 [Hexcles]
- ack jugglinmike
- 01:12:04 [MikeSmith]
- ack jugglinmike
- 01:12:10 [reillyg]
- q+
- 01:12:17 [foolip]
- jugglinmike: that's the real interesting thing
- 01:12:31 [foolip]
- jugglinmike: whether webdriver or test-only best discussed with [scribe not sure]
- 01:12:48 [reillyg]
- s/[scribe not sure]/Browser Testing and Tools WG/
- 01:13:00 [foolip]
- jugglinmike: I'm not sure how much to push for/against test-only APIs
- 01:13:16 [foolip]
- jugglinmike: should we require when we add a new dir in wpt that the spec says something about automation?
- 01:13:27 [ato]
- q+
- 01:13:28 [foolip]
- jugglinmike: that may be too drastic, but would be good with more clarity for reviewers
- 01:13:52 [CalebRouleau]
- CalebRouleau has joined #testing
- 01:13:54 [foolip]
- jgraham: people probably agree that it's OK to complain about tests using non-std APIs
- 01:14:13 [CalebRouleau]
- present+
- 01:14:16 [foolip]
- jgraham: put people determined to write tests for their experimental feature, hard to stop them from doing that
- 01:14:23 [foolip]
- jgraham: difficult to do that gatekeeping
- 01:14:29 [boaz]
- q?
- 01:14:36 [boaz]
- q+
- 01:14:39 [foolip]
- jgraham: if there are cases that have a good spec for the web-exposed part but not for the testing we should push back
- 01:14:51 [Hexcles]
- ack reillyg
- 01:15:11 [foolip]
- reillyg: I'd be sad to this group reject tests for features that aren't standardized yet
- 01:15:42 [MikeSmith]
- q?
- 01:15:48 [foolip]
- jgraham: by "standardized" I mean just "spec exists", as opposed to just an explainer. this isn't about the standards track, it's about making an attempt to make a standard
- 01:16:24 [foolip]
- reillyg: in early stages of development you'll write some code, some tests, some spec, seems reasonable to have tests before there's a formal spec
- 01:16:35 [foolip]
- reillyg: whether those are in wpt or kept private, is a reasonable question
- 01:16:49 [foolip]
- q?
- 01:17:06 [jorydotcom]
- jorydotcom has joined #testing
- 01:17:13 [foolip]
- reillyg: I would like to see more linting, testing that we aren't depending on random things
- 01:17:28 [foolip]
- reillyg: some level of linting to check you aren't assuming things that aren't well-defined
- 01:17:34 [zcorpan]
- q?
- 01:17:52 [foolip]
- foolip: tentative tests are a thing
- 01:17:59 [foolip]
- jgraham: but not tentative directories
- 01:18:19 [foolip]
- ato: the webdriver group has engaged with other groups, like permissions
- 01:18:30 [foolip]
- ato: returning problem is lack of motivation for implementation
- 01:18:57 [foolip]
- ato: webdriver covers an enormous part of the browser, so people on webdriver don't have expertise to implement things specific to say webxr
- 01:19:12 [foolip]
- ato: should encourage tests to think hard about testing, but some things are very hard with current webdriver protocol
- 01:19:22 [foolip]
- ato: some things also might not make sense to test using webdriver
- 01:19:49 [foolip]
- ato: jugglinmike mentioned what would be useful for web devs also, part of our responsibility to think about that
- 01:19:49 [zcorpan]
- ack ato
- 01:19:54 [zcorpan]
- ack boaz
- 01:20:24 [foolip]
- boaz: agree that writing tests early is great, and that the material should be shared, last thing we want to do is to keep that from happening
- 01:20:34 [foolip]
- boaz: for some of these experiments, would another place make sense?
- 01:20:38 [foolip]
- q+
- 01:20:52 [Hexcles]
- q+
- 01:20:58 [foolip]
- boaz: for tests that depend on things that are proprietay
- 01:21:19 [foolip]
- jgraham: two cases. if you depend on proprietary APIs don't upstream.
- 01:21:35 [foolip]
- jgraham: ... if you're depending on something you have no intent to standardize
- 01:21:44 [rniwa]
- rniwa has joined #testing
- 01:21:46 [foolip]
- jgraham: but for something like std-toast the tentative test mechanism works
- 01:22:02 [foolip]
- boaz: but isn't there material that can't be run in all browser?
- 01:22:29 [foolip]
- jgraham: unclear if there are tests that will never work even if they implement the feature. that's the line: would this be testable if the browser implemented the spec as written?
- 01:22:32 [foolip]
- q?
- 01:22:36 [foolip]
- q-
- 01:23:43 [foolip]
- jgraham: there have been cases where changes make the tests wrong and not per spec, browser-specific by accident
- 01:24:02 [foolip]
- Hexcles: a bit concerned about adding yet another place for staging
- 01:24:10 [foolip]
- Hexcles: in chromium we already have wpt_internal
- 01:24:20 [jorydotcom]
- ack Hexcles
- 01:24:21 [marcosc]
- marcosc has joined #testing
- 01:24:22 [foolip]
- Hexcles: this is for things that are very early stage or unlikely to be testable
- 01:24:37 [foolip]
- Hexcles: a level between that and wpt is too much to organize
- 01:24:45 [foolip]
- Hexcles: but tentative directories is reasonable
- 01:24:59 [foolip]
- Hexcles: like if you don't yet have a spec draft, mark the whole directory as tentative
- 01:25:02 [ato]
- Hexcles: ++
- 01:25:02 [jgraham]
- q+
- 01:25:19 [foolip]
- Hexcles: agree with jgraham that the bar is if it could be tested in other browsers, that's good enough for wpt
- 01:25:23 [zcorpan]
- q+
- 01:25:40 [foolip]
- Hexcles: need not require that it can already be tested in multiple impls
- 01:25:44 [foolip]
- https://cs.chromium.org/chromium/src/third_party/blink/web_tests/wpt_internal/
- 01:25:58 [foolip]
- jgraham: we have the same thing, a directory for things we can't export for some reason
- 01:26:12 [foolip]
- q?
- 01:26:23 [JohnJansen]
- q?
- 01:26:28 [foolip]
- jgraham: there may also be a wpt.fyi feature request to search for tentative tests, other than in the test name
- 01:26:30 [Hexcles]
- ack jgraham
- 01:26:59 [foolip]
- jgraham: might be a wpt.fyi feature request to filter by tentative
- 01:27:01 [JohnJansen]
- q+
- 01:27:04 [foolip]
- `!tentatve` should work
- 01:27:05 [jgraham]
- ack zcorpan
- 01:27:21 [foolip]
- zcorpan: what's the effect of having tentative tests?
- 01:27:24 [foolip]
- jgraham: informative only
- 01:27:32 [foolip]
- zcorpan: so not different in wpt.fyi
- 01:27:53 [foolip]
- zcorpan: wonder what tentative directories would be for?
- 01:27:59 [foolip]
- q?
- 01:28:11 [foolip]
- JohnJansen: hello jugglinmike
- 01:28:21 [foolip]
- JohnJansen: a long time ago we had vendor folders in wpt
- 01:28:23 [Hexcles]
- ack JohnJansen
- 01:28:24 [foolip]
- jgraham: *laugh*
- 01:28:45 [foolip]
- JohnJansen: the idea itself was sound, an indication they weren't meant to run in other browsers
- 01:28:51 [ato]
- q+
- 01:28:58 [foolip]
- JohnJansen: sounds similar to tentative folder or another staging repository
- 01:29:19 [foolip]
- https://wpt.fyi/results/old-tests/submission/Microsoft
- 01:30:24 [zcorpan]
- q+
- 01:30:31 [foolip]
- JohnJansen: coming up with ideas here for how to store non-all-browser tests is irrelevant if we can't enforce that all tests follow that pattern
- 01:30:37 [boaz]
- q?
- 01:30:53 [foolip]
- jgraham: rule for wpt is that in theory all tests should run in all browsers
- 01:31:06 [JohnJansen]
- q?
- 01:31:11 [JohnJansen]
- +1
- 01:31:16 [foolip]
- jgraham: let's clear the queue and break
- 01:31:48 [NavidZ_]
- q+
- 01:32:11 [Hexcles]
- ack ato
- 01:32:14 [foolip]
- ato: I'm not aware of the history of vendor directories, but in principle sounds like a good idea to run other's tests
- 01:32:45 [foolip]
- ato: in Mozilla have tests in mozilla-central where we might want to reuse the wpt harness
- 01:32:53 [tidoust]
- tidoust has joined #testing
- 01:33:02 [foolip]
- ato: and turns out some of the wpt tooling is quite good
- 01:33:07 [foolip]
- q?
- 01:33:08 [JohnJansen]
- q?
- 01:33:21 [foolip]
- zcorpan: in response to JohnJansen, how to enforce?
- 01:33:30 [foolip]
- zcorpan: one way is lint checker, which jugglinmike already did for mojo tests
- 01:33:33 [JohnJansen]
- RRSAgent, make minutes
- 01:33:33 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 01:33:36 [Hexcles]
- ack zcorpan
- 01:33:44 [jgraham]
- Zakim: close the queue
- 01:33:49 [foolip]
- zcorpan: think jgraham reviewed that, so we have a lint already for mojo tests
- 01:33:54 [jgraham]
- close the queue
- 01:34:31 [foolip]
- zcorpan: we may need a policy written down
- 01:34:48 [foolip]
- foolip: we also have a lint for LayoutTests-specific APIs
- 01:35:10 [zcorpan]
- q?
- 01:35:10 [Hexcles]
- ack NavidZ_
- 01:35:15 [foolip]
- boaz: can we have an action item for that?
- 01:35:22 [foolip]
- foolip: you have edit access now :)
- 01:35:53 [foolip]
- NavidZ_: webdriver pointer actions, we have that now.
- 01:36:09 [foolip]
- NavidZ_: but some things depend on the browser or the platform, like disabling flinging
- 01:36:24 [foolip]
- NavidZ_: because I don't want the page to scroll along some random curve
- 01:36:49 [foolip]
- NavidZ_: second is platform-specific things like #ifdefs in the code
- 01:37:06 [foolip]
- NavidZ_: browsers match platform conventions, e.g. with how stylus works
- 01:37:24 [foolip]
- NavidZ_: context menus are also different on different platforms
- 01:37:33 [foolip]
- NavidZ_: should we expose those differences to tests?
- 01:38:05 [zcorpan]
- https://github.com/web-platform-tests/wpt/pull/18509 is the lint check PR
- 01:38:20 [shimazu]
- shimazu has joined #testing
- 01:38:24 [zcorpan]
- break till 11
- 01:38:32 [Hexcles]
- Hexcles has joined #testing
- 01:44:01 [reillyg]
- reillyg has joined #testing
- 01:44:35 [Orphis]
- Orphis has joined #testing
- 01:48:13 [odejesush]
- odejesush has joined #testing
- 01:58:02 [shimazu]
- shimazu has joined #testing
- 01:59:27 [spectranaut]
- spectranaut has joined #testing
- 02:01:18 [Hexcles]
- Hexcles has joined #testing
- 02:03:02 [zcorpan]
- zcorpan has joined #testing
- 02:03:12 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19107 merge into 07master: Port perspective and perspective-origin interpolation tests to wpt - https://git.io/JeORo
- 02:03:36 [james]
- james has joined #testing
- 02:04:12 [shimazu]
- shimazu has joined #testing
- 02:06:12 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot 03merged 13#19016 into 07master: Expand 'autofocus' attribute support to all of HTML and SVG elements. - https://git.io/JemyF
- 02:09:04 [AutomatedTester]
- scribenick: AutomatedTester
- 02:09:05 [kevers]
- kevers has joined #testing
- 02:09:12 [JohnJansen]
- https://github.com/web-platform-tests/rfcs/pull/24
- 02:09:15 [kevers]
- present+
- 02:09:15 [JohnJansen]
- present+
- 02:09:32 [JohnJansen]
- Topic: RFC: Test Editor
- 02:09:46 [foolip]
- q?
- 02:10:08 [AutomatedTester]
- foolip: this session is for the rfc for the test editor
- 02:10:12 [jorydotcom]
- jorydotcom has joined #testing
- 02:10:17 [foolip]
- q?
- 02:10:55 [fremy]
- fremy has joined #testing
- 02:11:01 [jgraham]
- scribe AutomatedTester
- 02:11:27 [marcosc]
- marcosc has joined #testing
- 02:11:30 [ato]
- Reminder, this is the current seat map: https://docs.google.com/spreadsheets/d/1cqPK6ze2OCLsho4twJHNLZUPktfejIiiDlMwv0TaZBg/edit#gid=0
- 02:11:39 [AutomatedTester]
- fremy: let's talk about the history of this rfc. Part of my work there was a need to quickly make wpt. This would be from reducing a webcompat issues
- 02:12:04 [AutomatedTester]
- ... one of my ideas was to use something like jsfiddle and would be quick and easy to make wpt
- 02:12:14 [AutomatedTester]
- ... there would just have the basics for the doc
- 02:12:30 [AutomatedTester]
- ... and then you can add simple assertions.
- 02:12:54 [AutomatedTester]
- fremy: people do not want to have to fork wpt/ github repo. They just want to quickly make a test.
- 02:13:05 [ato]
- RRSAgent: please make the minutes
- 02:13:05 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 02:13:35 [AutomatedTester]
- fremy: you can go from "i need to create a quick test" to "here is a PR" and make it seemless as possible
- 02:13:57 [ato]
- ScribeNick: AutomatedTester
- 02:14:04 [AutomatedTester]
- ... this is how it started and as part of CSSWG we have expanded it
- 02:14:40 [AutomatedTester]
- JohnJansen: there was a request from several people outside of MS to move it away from MS to wpt
- 02:15:02 [AutomatedTester]
- ... I have renamed the repo to editor
- 02:15:15 [AutomatedTester]
- ... and we have been getting a bunch of questions on how we can use it
- 02:15:21 [ato]
- q?
- 02:15:32 [foolip]
- q?
- 02:15:42 [AutomatedTester]
- ... the request then to see if we can move forward and what is required to close the rfc
- 02:16:06 [AutomatedTester]
- foolip: is there someone to willing to deploy and maintain it? including infra?
- 02:16:11 [CalebRouleau]
- CalebRouleau has joined #testing
- 02:16:21 [rniwa]
- rniwa has joined #testing
- 02:16:35 [AutomatedTester]
- JohnJansen: we have modified the azure instance to do all of this
- 02:16:36 [fremy]
- https://wptest.center
- 02:16:43 [foolip]
- q?
- 02:16:47 [AutomatedTester]
- foolip: is there someone to work on this?
- 02:16:54 [JohnJansen]
- https://github.com/web-platform-tests/editor
- 02:16:54 [AutomatedTester]
- JohnJansen: yes
- 02:17:10 [JohnJansen]
- https://wptest.center/#/new
- 02:18:00 [ato]
- Present + fremy
- 02:18:05 [AutomatedTester]
- fremy: what does it take to make this project useful for people at large? Workflow changes? Is there value for this group? How can we manage the contribution story
- 02:18:26 [AutomatedTester]
- ... I am happy to continue to contributing this and make the changes to make it work
- 02:18:39 [jorydotcom]
- q+
- 02:18:44 [AutomatedTester]
- ... but it would be good to make sure that there are other vendors willing to participate
- 02:18:53 [JohnJansen]
- q?
- 02:19:23 [AutomatedTester]
- jorydotcom: This moved out of MS github to wpt github.
- 02:19:37 [foolip]
- ack jorydotcom
- 02:19:39 [foolip]
- q?
- 02:19:43 [AutomatedTester]
- ... have we done all the due diligence on the move
- 02:19:49 [AutomatedTester]
- JohnJansen: yes, MS legal agreed
- 02:20:23 [AutomatedTester]
- JohnJansen: I have been using it to test reductions that I have made on web compat issues
- 02:20:55 [AutomatedTester]
- JohnJansen: we were in a state where reductions were more important and therefore no one was doing wpt for these issues
- 02:21:56 [AutomatedTester]
- jgraham: the people in this group are less likely to use this project and this might be the wrong group to figure out what is missing and what features could be missing
- 02:22:38 [foolip]
- q?
- 02:22:42 [AutomatedTester]
- ... having said that speaking to the webcompat or MDN people might mean the feature requests would come from people adjacent to this group
- 02:23:07 [MikeSmith]
- q?
- 02:23:23 [AutomatedTester]
- JohnJansen: the request in the RFC is to make editor available from this group for other people to use
- 02:23:37 [zghadyali]
- zghadyali has joined #testing
- 02:23:44 [Lan]
- Lan has joined #testing
- 02:24:16 [AutomatedTester]
- ... we could use take it and then see if in the year that is great then we support it or if it doesnt then just archive the repo
- 02:25:40 [AutomatedTester]
- fremy: the CSSWG like and we have changed the process to start changing a spec then it needed a test. We used this process to remove the friction in getting new tests. TabAtkins has used this in their workflow and seems happy so far
- 02:26:10 [AutomatedTester]
- ... we need to make tools to allow editors that they dont need to change their workflow to make more wpt
- 02:26:25 [BitBot]
- (14rfcs) [issue] boazsender opened 13#30: Create an RFC for what material is and is not allowed in WPT - https://git.io/JeOR5
- 02:26:26 [xiaoqian]
- xiaoqian has joined #testing
- 02:26:37 [AutomatedTester]
- ... so if we can make this more official then people are likely to use it more
- 02:27:16 [jgraham]
- q+
- 02:29:13 [foolip]
- q?
- 02:29:17 [foolip]
- ack jgraham
- 02:29:47 [AutomatedTester]
- jgraham: there is a community building thing here. people could use but there are "things" that need fixing. But if we can build a community and see how it goes. We can link to the docs but not say it is the prefered way
- 02:30:09 [AutomatedTester]
- foolip: is going to need someone to support it?
- 02:30:22 [AutomatedTester]
- fremy: is MS going to support it?
- 02:31:07 [AutomatedTester]
- JohnJansen: it depends on how much work but we can have a look to see how it goes. If MS fails in the contribution then we can re-evaluate it
- 02:31:36 [AutomatedTester]
- foolip: if it is going to be called a editor then it needs to fix tests are already in the repo
- 02:32:22 [AutomatedTester]
- fremy: this is currently being hosted on its own environment to manipulate it. E.g. slow network.
- 02:32:33 [AutomatedTester]
- foolip: so edit and then run on the CI.
- 02:32:37 [AutomatedTester]
- fremy: yes
- 02:33:08 [AutomatedTester]
- jgraham: if you are writing a test then you can use wpt-test.live to host it
- 02:33:37 [AutomatedTester]
- jgraham: you can reuse server side features this way
- 02:34:12 [AutomatedTester]
- foolip: unless you are changing the resources in the backend you can do a lot of tests now
- 02:35:13 [AutomatedTester]
- jgraham: [describes how the resources could work as an example and use the wpt]
- 02:35:43 [AutomatedTester]
- fremy: for a new test it is nice to have the test resouces (css, js) split and easy to use
- 02:35:56 [AutomatedTester]
- q?
- 02:36:54 [AutomatedTester]
- foolip: to resolve this RFC, we do have some questions that need answering
- 02:36:58 [TabAtkins]
- While there are lots of cool things the editor could do as well, I can day with confidence that its current functionality is extremely useful for writing tests for myself. It's surprisingly hard to do the test-commit dance by hand.
- 02:37:09 [AutomatedTester]
- ... like CoC.
- 02:37:28 [AutomatedTester]
- JohnJansen: there was a question about the reference to mongodb
- 02:37:53 [AutomatedTester]
- Hexcles: there is an issue with google people needing to have mongo and we cant
- 02:38:08 [plh]
- plh has joined #testing
- 02:38:30 [AutomatedTester]
- fremy: you could have it locally but you don't really use it. in azure we use a doc storage
- 02:38:55 [AutomatedTester]
- fremy: you just need the mongodb protocol
- 02:39:16 [AutomatedTester]
- Hexcles: is there other alternatives that use mongodb protocol
- 02:39:25 [AutomatedTester]
- fremy: yes
- 02:39:44 [AutomatedTester]
- foolip: so this could be abstracted away
- 02:40:05 [AutomatedTester]
- foolip: but it uses mongodb npm package?
- 02:40:16 [AutomatedTester]
- fremy: thats the compat issue
- 02:40:22 [AutomatedTester]
- jgraham: who implemented it?
- 02:41:02 [AutomatedTester]
- Hexcles: we can just double check with legal but we can't install the database
- 02:41:31 [AutomatedTester]
- foolip: there are details that we dont but we need to figure this out
- 02:42:14 [AutomatedTester]
- [discussion about what database needs to be installed locally]
- 02:43:13 [AutomatedTester]
- fremy: we dont need a database locally. We just needed something that was able to store documents and to make sure that it doesn't limit you to the MS cloud document storage
- 02:43:20 [xiaoqian]
- xiaoqian has joined #testing
- 02:44:11 [AutomatedTester]
- foolip: if you run the tests locally with no mongodb it just errors so we can talk to legal
- 02:44:21 [AutomatedTester]
- foolip: so it jkust needs a kv storage
- 02:44:29 [AutomatedTester]
- fremy: yes, its really simple
- 02:45:39 [AutomatedTester]
- fremy: the use of the database is for login and storing the tests during dev
- 02:45:50 [AutomatedTester]
- AutomatedTester: could we move to sqlite?
- 02:45:56 [AutomatedTester]
- jgraham: yes but the model is different
- 02:46:40 [AutomatedTester]
- fremy: for no database we could just write to files, thats how it used to work. It's really really simple
- 02:47:02 [foolip]
- q?
- 02:47:12 [AutomatedTester]
- foolip: there is the CLA question
- 02:47:28 [AutomatedTester]
- fremy: we should delete it
- 02:47:32 [AutomatedTester]
- JohnJansen: agreed
- 02:48:03 [AutomatedTester]
- JohnJansen: we can create individual issues on the repo and approve this rfc
- 02:48:22 [shimazu]
- shimazu has joined #testing
- 02:48:25 [AutomatedTester]
- foolip: we just to make sure that some things are done in order
- 02:48:35 [zcorpan]
- zcorpan has joined #testing
- 02:49:21 [AutomatedTester]
- foolip: things that need to happen. 1) change the wpt CoC 2) delete the cla 3) pick a URL
- 02:49:32 [AutomatedTester]
- 3) will not block the RFC
- 02:50:06 [AutomatedTester]
- foolip: is there a hostname we should use
- 02:50:35 [AutomatedTester]
- jgraham: we can have it under web-platform-tests.org
- 02:51:07 [AutomatedTester]
- foolip: so editor.web-platform-tests.org
- 02:52:00 [JohnJansen]
- rrsagent, make minutes
- 02:52:00 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 02:52:24 [AutomatedTester]
- fremy: MS is going continue to do hosting
- 02:52:42 [Boaz]
- Boaz has joined #Testing
- 02:52:44 [AutomatedTester]
- ... and then TabAtkins, Anton and I will continue to look at it
- 02:52:46 [Boaz]
- Present+ Boaz Sender, Bocoup
- 02:53:10 [AutomatedTester]
- foolip: is there going to be a big change to make it an editor and not just a reduction tool
- 02:53:14 [ato]
- RRSAgent: make minutes
- 02:53:14 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 02:53:45 [ato]
- Present- Bocoup
- 02:54:05 [ato]
- Present- boaz
- 02:54:10 [AutomatedTester]
- fremy: you would need to have a file list
- 02:54:19 [ato]
- RRSAgent: make minutes
- 02:54:19 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 02:54:29 [AutomatedTester]
- [technical discussion on editor and how that would work]
- 02:55:00 [AutomatedTester]
- ACTION JohnJansen to file issues for 1) change the wpt CoC 2) delete the cla
- 02:55:23 [ato]
- RRSAgent: Please make the minutes
- 02:55:23 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 02:55:25 [foolip]
- q?
- 02:55:27 [jgraham]
- q?
- 02:55:30 [AutomatedTester]
- ACTION JohnJansen to paste links to the minutes
- 02:56:19 [Boaz]
- http://web-platform-tests.live/
- 02:56:25 [AutomatedTester]
- lukebjerring: one thing to mention, there is an implicit flow and how it copies it to web-platform-tests.live
- 02:56:27 [Boaz]
- will be http://wpt.live
- 02:56:35 [ato]
- ACTION: JohnJansen to file issues for (1) change the WPT CoC (2) delete the CLA
- 02:57:31 [foolip]
- q?
- 02:57:31 [ato]
- s/ACTION JohnJansen to paste links to the minutes//
- 02:57:59 [AutomatedTester]
- ...the flow for http://web-platform-tests.live/ will allow resources to work for people
- 02:58:01 [ato]
- RRSAgent: Please draft the minutes
- 02:58:01 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 02:58:10 [AutomatedTester]
- fremy: I was unaware of this process
- 02:58:35 [AutomatedTester]
- ... we can discuss how this could work and make/fix features in the editor
- 02:58:48 [foolip]
- q?
- 03:01:00 [AutomatedTester]
- Topic: CoC
- 03:02:10 [zcorpan]
- github: https://github.com/web-platform-tests/rfcs/pull/17
- 03:02:19 [AutomatedTester]
- Boaz: there was a PR for the CoC that was opened a year ago. I and jorydotcom are happy to be points of contact
- 03:02:25 [marcosc]
- marcosc has joined #testing
- 03:02:32 [AutomatedTester]
- ... and we need to get consesus on the RFC
- 03:02:54 [foolip]
- q?
- 03:03:11 [AutomatedTester]
- ... and make wpt a good place to contribute
- 03:03:42 [foolip]
- q+
- 03:03:43 [foolip]
- q?
- 03:03:43 [AutomatedTester]
- jorydotcom: I spoke to gsnedders about some of the process. They had concerns about 2 people from the same company
- 03:04:31 [AutomatedTester]
- jgraham: and I have spoke to gsnedders about this and one of the concerns was that for a CoC enforcing group would need to reflect the diversity that we want and have a large diverse group
- 03:05:15 [foolip]
- q?
- 03:05:26 [AutomatedTester]
- ... we need to make sure that we cover the diversity groups covered and while the people offering are more diverse than this group but it doesnt reflect where we want to be
- 03:05:56 [AutomatedTester]
- jorydotcom: absent of the diversity we want the people on the group need to make sure that the relevant people have the relevant training
- 03:06:01 [foolip]
- q?
- 03:06:25 [AutomatedTester]
- Boaz: the right thing to do is to also add the process for people to voluteer on the CoC group
- 03:06:53 [AutomatedTester]
- foolip: is the RFC process for adding/removing people
- 03:07:06 [AutomatedTester]
- jgraham: that is unclear. We need to have that properly documented
- 03:07:34 [AutomatedTester]
- ... and not everything needs to be documented in the public
- 03:08:15 [AutomatedTester]
- ... e.g. if there are concerns for the moderating being in the team that doesnt need to be in the public sphere.
- 03:08:40 [AutomatedTester]
- plh: we have moderates that feel supported to jump in and make the relevant comments to protect the group
- 03:09:04 [AutomatedTester]
- plh: and it shouldnt really be from the core team
- 03:10:25 [AutomatedTester]
- jorydotcom: I can share the TC39 CoC how that works. People can email saying that would like to join and then the CoC team will discuss and then CoC team emails a private list for TC39 only to say who is being added
- 03:11:01 [AutomatedTester]
- jgraham: that process works for TC39 since they are closed membership where we dont have that process and tooling for that
- 03:11:59 [AutomatedTester]
- jgraham: we dont have a closed mailing list
- 03:12:06 [zcorpan]
- zcorpan has joined #testing
- 03:13:08 [AutomatedTester]
- jgraham: we would need to have a process that someone is nominated, self nominated is fine, and return feedback would come back. The announce needs to be public but the feedback is private
- 03:13:14 [MikeSmith]
- q+ to comment
- 03:13:20 [foolip]
- q?
- 03:13:28 [AutomatedTester]
- ack foolip
- 03:13:33 [foolip]
- ack foolip
- 03:13:34 [foolip]
- ack MikeSmith
- 03:13:34 [Zakim]
- MikeSmith, you wanted to comment
- 03:13:37 [Boaz]
- q+
- 03:13:37 [foolip]
- q?
- 03:13:51 [zcorpan]
- zcorpan has joined #testing
- 03:13:58 [AutomatedTester]
- MikeSmith: I want to ask if the moderates responding is private?
- 03:13:59 [zcorpan]
- q?
- 03:14:04 [foolip]
- ack Boaz
- 03:14:11 [AutomatedTester]
- Boaz: no this is for the process for joinging the moderates
- 03:14:12 [zcorpan]
- q+
- 03:14:22 [JohnJansen]
- RRSAgent, make minuts
- 03:14:22 [RRSAgent]
- I'm logging. I don't understand 'make minuts', JohnJansen. Try /msg RRSAgent help
- 03:14:25 [JohnJansen]
- RRSAgent, make minutes
- 03:14:25 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 03:14:30 [AutomatedTester]
- Boaz: this sounds like we need to make the process and move it forward
- 03:14:59 [AutomatedTester]
- Boaz: since we are from the same company we would like to get more people to join so it is not just us
- 03:15:37 [AutomatedTester]
- jgraham: I can't speak for gsnedders as they have the concerns and they would need to give it
- 03:16:10 [AutomatedTester]
- jgraham: I think that if we dont have it perfect and iterate with the idea of where we want to be then it is a good start
- 03:16:20 [foolip]
- ack zcorpan
- 03:16:59 [spectranaut_]
- spectranaut_ has joined #testing
- 03:17:23 [AutomatedTester]
- zcorpan: I want to discuss something with the moderates. What is the consequence if someone violates the CoC. Ban seems harsh and warnings is not harsh enough. We need to know how this could work
- 03:18:51 [AutomatedTester]
- jorydotcom: There is a limited set of things that we can do as a group of volunteers. You need to build a ladder from "you broke our norms and here is some docs on what we agree and we want you in the community" to a full ban. In TC39 we do have temp bans for X months and keep that in a private file.
- 03:19:14 [AutomatedTester]
- zcorpan: that is the thing that we need to record a history for a specific person on each violation
- 03:19:31 [AutomatedTester]
- ... and what was done and said to that person
- 03:19:52 [AutomatedTester]
- Boaz: we can also explicitly add addedums on what we need to do
- 03:19:53 [zcorpan]
- zcorpan has joined #testing
- 03:19:55 [zcorpan]
- zcorpan has joined #testing
- 03:20:10 [AutomatedTester]
- jgraham: the Rust community has a moderation policy on top of the CoC
- 03:20:43 [AutomatedTester]
- ... and what the ladder looks like and describes what the valid responses can be
- 03:21:06 [jgraham]
- https://www.rust-lang.org/policies/code-of-conduct#moderation
- 03:21:09 [AutomatedTester]
- ... e.g. keep responses private if you have been moderated
- 03:21:41 [AutomatedTester]
- fremy: how would you appeal?
- 03:21:56 [Boaz]
- q+
- 03:23:06 [AutomatedTester]
- fremy: if the group is small then the appeals might cause issues
- 03:23:14 [AutomatedTester]
- Boaz: next steps?
- 03:23:30 [AutomatedTester]
- AutomatedTester: we need to get gsnedders opinion on this
- 03:23:52 [AutomatedTester]
- jgraham: I am not sure the RFC process is valid for this...
- 03:25:03 [AutomatedTester]
- jgraham: the consesus is that it is not perfect but it is a good start and we need iterate on this to make sure things arent blocked
- 03:25:42 [AutomatedTester]
- ACTION: create process for adding moderators via RFC
- 03:26:52 [Hexcles]
- Hexcles has joined #testing
- 03:26:53 [AutomatedTester]
- RRSAgent: make notes
- 03:26:53 [RRSAgent]
- I'm logging. I don't understand 'make notes', AutomatedTester. Try /msg RRSAgent help
- 03:35:32 [plh]
- plh has joined #testing
- 04:02:04 [Hexcles]
- Hexcles has joined #testing
- 04:02:28 [rniwa]
- rniwa has joined #testing
- 04:02:53 [cybai]
- cybai has joined #testing
- 04:06:20 [zcorpan]
- zcorpan has joined #testing
- 04:06:31 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19108 merge into 07master: Move 'autofocus' IDL attribute to HTMLOrSVGElement - https://git.io/JeO0z
- 04:07:20 [denis]
- denis has joined #testing
- 04:10:00 [spectranaut_]
- presentation: https://bocoup.github.io/presentation-aria-and-webdriver/#/
- 04:10:01 [shimazu]
- shimazu has joined #testing
- 04:10:04 [wanming]
- wanming has joined #testing
- 04:10:33 [spectranaut_]
- whoops wrong channel ^
- 04:23:59 [shimazu]
- shimazu has joined #testing
- 04:25:49 [marcosc]
- marcosc has joined #testing
- 04:28:25 [Hexcles]
- Hexcles has joined #testing
- 04:31:01 [denis]
- denis has joined #testing
- 04:34:51 [shimazu]
- shimazu has joined #testing
- 04:36:00 [shimazu]
- shimazu has joined #testing
- 04:36:49 [tidoust]
- tidoust has joined #testing
- 04:38:58 [zcorpan]
- zcorpan has joined #testing
- 04:40:34 [cb]
- thanks for sharing anyway !
- 04:41:49 [shimazu]
- shimazu has joined #testing
- 04:46:16 [shimazu]
- shimazu has joined #testing
- 04:49:35 [jorydotcom]
- jorydotcom has joined #testing
- 04:52:25 [zcorpan]
- zcorpan has joined #testing
- 04:53:13 [BitBot]
- (14wpt) [PR] moz-wptsync-bot 03merged 13#19093 into 07master: [Gecko Bug 1488530] Add more tests for clip-path:path(). - https://git.io/JeOch
- 04:54:24 [marcosc]
- marcosc has joined #testing
- 04:54:54 [Hexcles]
- Hexcles has joined #testing
- 04:55:07 [marcosc]
- marcosc has joined #testing
- 04:55:44 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19109 merge into 07master: Add expression representation to InterpolableLength - https://git.io/JeO0A
- 04:55:50 [Hexcles]
- RRSAgent, make minutes
- 04:55:50 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html Hexcles
- 04:57:05 [plh]
- plh has joined #testing
- 04:57:36 [zcorpan_]
- zcorpan_ has joined #testing
- 05:06:15 [marcosc]
- marcosc has joined #testing
- 05:06:49 [foolip]
- Alright, we should start the wpt.fyi session now
- 05:07:27 [ato]
- We should just start.
- 05:08:18 [AutomatedTester]
- RRSAgent: make minutes
- 05:08:18 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html AutomatedTester
- 05:08:47 [foolip]
- q?
- 05:08:52 [foolip]
- q- botie
- 05:08:55 [foolip]
- q- Boaz
- 05:09:19 [foolip]
- scribenick: lukebjerring
- 05:09:20 [AutomatedTester]
- scribenick lukebjerring
- 05:09:27 [JohnJansen]
- Topic: WPT.FYI features, etc
- 05:09:33 [zcorpan]
- zcorpan has joined #testing
- 05:09:39 [kevers]
- kevers has joined #testing
- 05:09:53 [Hexcles]
- RRSAgent, who's the scriber?
- 05:09:53 [RRSAgent]
- I'm logging. Sorry, nothing found for 'who's the scriber'
- 05:09:55 [diemol]
- diemol has joined #testing
- 05:10:04 [JohnJansen]
- RRSAgent, scribe?
- 05:10:04 [RRSAgent]
- I'm logging. Sorry, nothing found for 'scribe'
- 05:10:06 [ato]
- RRSAgent: Please draft the minutes
- 05:10:06 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 05:10:28 [foolip]
- scribenick foolip
- 05:10:34 [ricea]
- ricea has joined #testing
- 05:10:41 [foolip]
- lukebjerring: overview of the triage metadata project:
- 05:10:47 [ato]
- ScribeNick: foolip
- 05:10:53 [foolip]
- lukebjerring: we've created https://github.com/web-platform-tests/wpt-metadata parallel to wpt
- 05:11:03 [JohnJansen]
- TOPIC: WPT.FYI features, etc
- 05:11:18 [foolip]
- lukebjerring: it's out-of-tree metadata about the results, which wpt.fyi consumes to let people triage their searches
- 05:11:31 [foolip]
- lukebjerring: there's a `link:something` atom that you can filter by
- 05:11:59 [foolip]
- lukebjerring: work also underway to create UI for adding links when you spot the failures
- 05:12:25 [foolip]
- lukebjerring: had some feedback which made us revise, new plan is to use GitHub login and have cookie to do stuff on your behalf
- 05:12:40 [CalebRouleau]
- CalebRouleau has joined #testing
- 05:12:42 [foolip]
- foolip: why is it important to do anything on the user's behalf?
- 05:13:18 [foolip]
- jgraham: don't necessarily need to use GitHub API call with user's credentials, but it is a requirement that we know who the user is
- 05:13:31 [foolip]
- jgraham: wpt.fyi wants to validate with GitHub who you are
- 05:13:53 [foolip]
- lukebjerring: feature requests in CSS working group...
- 05:14:02 [foolip]
- lukebjerring: they'd like to be able to search by flags in a test
- 05:14:23 [foolip]
- lukebjerring: I've filed https://github.com/web-platform-tests/wpt.fyi/issues/1491 for labeling of results
- 05:14:48 [foolip]
- lukebjerring: then a bot could extract the flags from the wpt repo's tests
- 05:14:54 [foolip]
- lukebjerring: then search by label on wpt.fyi
- 05:15:11 [foolip]
- jgraham: similar to what we do for manifest
- 05:15:18 [denis]
- denis has joined #testing
- 05:15:33 [foolip]
- jgraham: one way would be to put it into the manifest and update from that, but we'd worry about bloating it
- 05:15:45 [foolip]
- lukebjerring: question of whether we need a separate repo
- 05:16:00 [foolip]
- lukebjerring: main reason is we expect a lot of bloat that's only for wpt.fyi
- 05:16:03 [foolip]
- q?
- 05:16:31 [foolip]
- lukebjerring: also have requests to filter by test type, and another for `<link rel="help">` links
- 05:16:56 [foolip]
- jgraham: historically the only information in the manifest is what you need to run the tests
- 05:17:10 [foolip]
- jgraham: now 18 MB or so, could easy grow much bigger
- 05:17:43 [foolip]
- lukebjerring: another kind of metadata is flakiness
- 05:18:50 [foolip]
- lukebjerring: we want a few different ways to add flaky metadata to the repo, and have that information be used for filtering in wpt.fyi
- 05:19:19 [foolip]
- AutomatedTester: does flakiness apply to all browsers?
- 05:19:30 [foolip]
- lukebjerring: it's like the triage links, you can specify which browser
- 05:20:31 [AutomatedTester]
- scribenick lukebjerring
- 05:20:44 [plh]
- plh has joined #testing
- 05:20:53 [rniwa]
- rniwa has joined #testing
- 05:21:36 [lukebjerring]
- lukebjerring: other suggestions welcome
- 05:21:47 [lukebjerring]
- jgraham: labels should be generic enough to cover most stuff
- 05:21:56 [ato]
- RRSAgent: Please draft the minutes
- 05:21:56 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 05:22:12 [ato]
- ScribeNick: lukebjerring
- 05:22:17 [lukebjerring]
- jgraham: wondering what the tradeoff for keeping it separate to wpt repo justified
- 05:22:43 [lukebjerring]
- ... could have a single job that has the right creds and only worry about the one repo
- 05:22:55 [lukebjerring]
- foolip: it's not too bad to use tokens
- 05:23:11 [lukebjerring]
- jgraham: GitHub actions give you a token more securely
- 05:23:26 [ato]
- s/scribenick lukebjerring//
- 05:23:30 [lukebjerring]
- foolip: there is an API you can call to trigger an action anyway
- 05:23:46 [lukebjerring]
- jgraham: one repo may end up being the right trade-off
- 05:24:01 [lukebjerring]
- foolip: would have to drop the files on import/export
- 05:24:26 [lukebjerring]
- jgraham: could just leave it there. info might be useful. you could atomically add a test and its metadata at the same time
- 05:24:52 [shimazu]
- shimazu has joined #testing
- 05:25:09 [lukebjerring]
- foolip: taskcluster jobs will just have to be restarted, which wouldn't be the case for one repo
- 05:26:14 [lukebjerring]
- lukebjerring: also an issue with file renames, could catch it straight away
- 05:27:10 [lukebjerring]
- jgraham: we could add lint rules etc
- 05:27:41 [lukebjerring]
- lukebjerring: it's not clear whether we would break consumers of the wpt META.yml files
- 05:28:13 [lukebjerring]
- foolip: would we nest another file tree? or re-use the existing META.yml files
- 05:29:43 [lukebjerring]
- lukebjerring: one upside of separation is grabbing the whole repo as an archive without any bloat
- 05:29:54 [JohnJansen]
- RRSAgent, make minutes
- 05:29:54 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 05:30:12 [lukebjerring]
- foolip: deriving labels from the tests themselves, having bots that update the metadata, etc... could get messy
- 05:31:18 [lukebjerring]
- jgraham: initially we were only considering human-generated stuff. now we're trying to crawl data from the tests and exposing it
- 05:31:21 [karl]
- karl has joined #testing
- 05:31:37 [lukebjerring]
- foolip: an extended manifest with derived info jgraham suggested could be an option here
- 05:32:12 [lukebjerring]
- jgraham: --extended manifest could just include extra data. if you need it you use it, if you don't, you don't
- 05:32:45 [lukebjerring]
- ... also potentially gives you a complete list of tests that should have been run, which will help with validation of reports
- 05:34:41 [lukebjerring]
- ... sounds like if we're extracting from the tests, we shouldn't have it in the META.yml files. that'd be... difficult / potentially divergent
- 05:34:55 [lukebjerring]
- ... larger version of the manifest is the better option
- 05:35:08 [lukebjerring]
- JohnJansen: what would be the extra info?
- 05:35:31 [lukebjerring]
- jgraham: spec URL(s), some parsed meta tag info
- 05:35:45 [eeeps]
- eeeps has joined #testing
- 05:36:05 [lukebjerring]
- ... reason its not done is just the sheer bloat. performance is also an issue, but can be solved with some incremental updates
- 05:36:36 [lukebjerring]
- foolip: manifest is huge, but most people don't update it, they download the latest one
- 05:37:23 [lukebjerring]
- jgraham: Windows struggles because of the amount of io
- 05:37:29 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot 03merged 13#19108 into 07master: Move 'autofocus' IDL attribute to HTMLOrSVGElement - https://git.io/JeO0z
- 05:38:24 [lukebjerring]
- jgraham: we'll just silently produce two copies of the manifest, wpt.fyi will be the only extended manifest consumer.
- 05:39:28 [lukebjerring]
- ... sounds like this addresses the needs.
- 05:40:16 [lukebjerring]
- foolip: so having the meta in wpt would solve the issues ? are we agreeing that we should converge the two?
- 05:40:41 [lukebjerring]
- lukebjerring: yep. and reuse the existing META.yml files, no need for parallel dir
- 05:41:07 [lukebjerring]
- foolip: deleting irrelevant META now becomes a bit more of an ask? lots of spam
- 05:42:51 [lukebjerring]
- Hexcles: We can have a GitHub action to archive out all the META.yml in isolation. maybe even the extended manifest in the same repo.
- 05:45:43 [lukebjerring]
- foolip: there's an item here for triaging regressions in major releases
- 05:46:11 [lukebjerring]
- lukebjerring: what's the feature request here? helper for building version-comparison query/diff?
- 05:46:56 [lukebjerring]
- Hexcles: difficulty is that WPT is changing over time, as well as the browsers
- 05:47:10 [lukebjerring]
- jgraham: just compare stable and beta ongoingly.
- 05:47:30 [lukebjerring]
- ... could come up with a deliberate overlap for older stable versions for a while.
- 05:47:53 [lukebjerring]
- Hexcles: we have different flags for beta. what we need is apples to apples comparison
- 05:48:13 [BitBot]
- (14wpt) [PR] fred-wang requested 13#19110 merge into 07master: Add MathML tests to check that legacy mstyle attributes are ignored. - https://git.io/JeOEw
- 05:48:22 [lukebjerring]
- jgraham: We've tried to get beta and stable as close to vanilla as possible
- 05:48:39 [zcorpan_]
- zcorpan_ has joined #testing
- 05:49:01 [lukebjerring]
- ... so far we haven't had anyone requesting this
- 05:49:20 [lukebjerring]
- ... should the bugs be triaged exactly the same? I think probably the should.
- 05:49:59 [shimazu]
- shimazu has joined #testing
- 05:50:04 [lukebjerring]
- foolip: browser specific failures are probably more useful, but the beta comparison is still good.
- 05:51:34 [lukebjerring]
- lukebjerring: let's talk about manual results submission
- 05:52:01 [lukebjerring]
- ... hard part is aggregating conflicting submissions (which was a feature request)
- 05:52:30 [lukebjerring]
- jgraham: wpt.fyi is very run-centric. submitting one result at a time is not
- 05:54:03 [tidoust]
- tidoust has joined #testing
- 05:56:15 [lukebjerring]
- JohnJansen: the model that they're using is running the test, saying pass/fail, maybe wait a long time, run another test, and then aggregate it all next to the automatically collected results
- 05:56:19 [plh]
- plh has joined #testing
- 05:57:00 [lukebjerring]
- jgraham: we can have this convo with them, but, you'll probably find that lots of manual data munging to get it all into one view that doesn't aggregate with the automatic data, they won't use it, they'll use the existing tool
- 05:57:37 [marcosc]
- marcosc has joined #testing
- 05:57:52 [lukebjerring]
- JohnJansen: they used to have a locked test suite (locked to a SHA) and they would not change the rev of the suite
- 05:58:07 [lukebjerring]
- Hexcles: so they locked multiple components to different SHAs?
- 05:58:29 [lukebjerring]
- JohnJansen: yes because they're interested in whether it most recently passed
- 05:59:11 [lukebjerring]
- jgraham: If you can take a SHA, collapse by product (single column), that would fix the way wpt.fyi treats runs separately
- 06:02:43 [BitBot]
- (14wpt) [PR] aluochromium requested 13#19111 merge into 07master: Configure chrome_android and android_webview to use server_host env option. - https://git.io/JeOED
- 06:04:15 [lukebjerring]
- JohnJansen: how would collapsing different runs in the UI handle conflicting results?
- 06:04:22 [lukebjerring]
- lukebjerring: most recent wins?
- 06:04:45 [lukebjerring]
- JohnJansen: I doubt they'd be OK with that, they'd want most authoritative wins
- 06:04:54 [lukebjerring]
- ... they actually have a voting UI
- 06:06:03 [Hexcles]
- Hexcles has joined #testing
- 06:06:06 [BitBot]
- (14wpt) [PR] fred-wang requested 13#19112 merge into 07master: Add MathML test to check that menclose@notation="radical" is not supp… - https://git.io/JeOEQ
- 06:07:53 [fantasai]
- fantasai has joined #testing
- 06:07:57 [fantasai]
- yo
- 06:14:19 [denis]
- denis has joined #testing
- 06:18:03 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot 03merged 13#19082 into 07master: [webnfc] Add tests for NFCPushOptions.ignoreRead - https://git.io/JeOL8
- 06:18:31 [xiaoqian]
- xiaoqian has joined #testing
- 06:19:23 [rniwa]
- rniwa has joined #testing
- 06:32:21 [tidoust]
- tidoust has joined #testing
- 06:33:25 [Hexcles]
- Hexcles has joined #testing
- 06:33:31 [JohnJansen]
- yo
- 06:34:36 [shimazu]
- shimazu has joined #testing
- 06:34:45 [ato]
- RRSAgent: Please draft the minutes
- 06:34:45 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 06:39:10 [gsnedders]
- RRSAgent: make the minutes
- 06:39:10 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html gsnedders
- 06:39:19 [AutomatedTester]
- scribenick AutomatedTester
- 06:40:11 [ato]
- ScribeNick: AutomatedTester
- 06:40:17 [ato]
- s/scribenick AutomatedTester//
- 06:40:36 [AutomatedTester]
- Hexcles: introduction: We have chromium team asking for support for QUIC. The problem is python2 doesnt have cryto issues
- 06:40:55 [JohnJansen]
- s/cryto/crypto
- 06:40:58 [AutomatedTester]
- Hexcles: we want to see if there a way that we can use a Go implementation
- 06:41:11 [AutomatedTester]
- jgraham: what is the difference for the H2?
- 06:41:30 [AutomatedTester]
- Victor: what does it have in it?
- 06:41:46 [AutomatedTester]
- jgraham: [explains details of H2 support]
- 06:42:06 [AutomatedTester]
- Victor: the QUIC and H2 are very different due to tls support
- 06:42:45 [AutomatedTester]
- Victor if you look at the QUIC impl matrix are there are 20 on there and 10 well done
- 06:43:06 [AutomatedTester]
- Victor [describes different example impl]
- 06:43:32 [zcorpan]
- zcorpan has joined #testing
- 06:44:04 [BitBot]
- (14wpt) [PR] fred-wang requested 13#19113 merge into 07master: Add MathML reftest to check that the mode attribute has no effect. - https://git.io/JeOu4
- 06:44:18 [zcorpan]
- zcorpan has joined #testing
- 06:44:39 [AutomatedTester]
- jgraham: 1 question: what are the use cases for testing are and what API are you thinking?
- 06:45:06 [AutomatedTester]
- ... traditionally we have been testing the interaction between protocol and features but not the protocol directly
- 06:46:04 [AutomatedTester]
- ... What would the test look like?
- 06:47:04 [ricea]
- ricea has joined #testing
- 06:47:22 [BitBot]
- (14wpt) [PR] fred-wang 03merged 13#19110 into 07master: Add MathML tests to check that legacy mstyle attributes are ignored. - https://git.io/JeOEw
- 06:47:25 [AutomatedTester]
- yhirano_: we have a python script that ...(websocket mention)
- 06:47:39 [BitBot]
- (14wpt) [PR] fred-wang 03merged 13#19112 into 07master: Add MathML test to check that menclose@notation="radical" is not supp… - https://git.io/JeOEQ
- 06:48:32 [AutomatedTester]
- jgraham: if something that needs to be in the API, what would you do it in
- 06:48:40 [AutomatedTester]
- Victor: that would be in Go or Rust
- 06:49:12 [AutomatedTester]
- jgraham: we currently dont have and compiled code step atm, we can add it but its not there at the moment
- 06:49:26 [AutomatedTester]
- jgraham: there is some impl complexity that needs to be supported
- 06:49:30 [gsnedders]
- https://wicg.github.io/web-transport/ is the specification in question, right?
- 06:49:31 [BitBot]
- (14wpt) [PR] fred-wang 03merged 13#19113 into 07master: Add MathML reftest to check that the mode attribute has no effect. - https://git.io/JeOu4
- 06:49:56 [shimazu]
- shimazu has joined #testing
- 06:50:14 [AutomatedTester]
- Hexcles: could the QUIC team give us a prebuilt binary and compiling tests
- 06:50:58 [AutomatedTester]
- ... and be separate repo for tests
- 06:51:08 [AutomatedTester]
- jgraham: we need to have ideally in a single repo
- 06:51:28 [AutomatedTester]
- yhirano_: we would have to have the handlers somewhere that can be updated regularly
- 06:51:50 [AutomatedTester]
- victor: is the requirement that no compilation or it must be python
- 06:52:01 [AutomatedTester]
- jgraham: there is no precedent for compiled code in wpt
- 06:53:51 [AutomatedTester]
- Hexcles: but the language doesnt matter here if it is rust or go
- 06:54:19 [AutomatedTester]
- jgraham: the complexity is that Firefox CI does Rust not Go
- 06:54:46 [AutomatedTester]
- ... and google will probably have the opposite problem
- 06:55:29 [AutomatedTester]
- ricea: [describes how we could have python over the go and connections to wptserve]
- 06:56:05 [AutomatedTester]
- ricea: or we could do the handlers in JS but it would be horrible
- 06:58:16 [AutomatedTester]
- ricea: my concern is that if we ship it we are limiting who can contribute to fix it while if we ship the build script then more people could add things. if the compiled step then having a handler in a non-compiuled language makes sense
- 06:58:51 [AutomatedTester]
- gsnedders: my concerns if we want webkit more involved they have pushed back if there has been runtime dependencies
- 06:59:07 [Hexcles]
- i/What would the test look like?/Victor: tests are for WebTransport, which is similar to WebSocket but in QUIC
- 07:01:00 [AutomatedTester]
- jgraham: we could have a process for compiling and then uploading to github for upstream, the browser version would be very different
- 07:02:04 [rniwa]
- rniwa has joined #testing
- 07:02:27 [AutomatedTester]
- jgraham: understanding the constraints for everyone would be simplify things.
- 07:03:33 [AutomatedTester]
- youennf: we have multiple servers why should we have it
- 07:03:43 [AutomatedTester]
- ... but at the end of the day the value of tests will win
- 07:04:39 [ricea]
- ricea has joined #testing
- 07:06:02 [zcorpan]
- zcorpan has joined #testing
- 07:06:30 [marcosc]
- marcosc has joined #testing
- 07:06:58 [Hexcles]
- scibenick: Hexcles
- 07:07:08 [Hexcles]
- ScribeNick: Hexcles
- 07:07:36 [Hexcles]
- Topic: ergonomics of test writing
- 07:07:45 [Hexcles]
- RRSAgent, update minutes
- 07:07:45 [RRSAgent]
- I'm logging. I don't understand 'update minutes', Hexcles. Try /msg RRSAgent help
- 07:07:51 [Hexcles]
- RRSAgent, make minutes
- 07:07:51 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html Hexcles
- 07:08:10 [JohnJansen]
- TOPIC: Ergonomics
- 07:08:11 [Hexcles]
- reillyg: I've found it difficult to debug WPT
- 07:08:23 [JohnJansen]
- Topic: ergonomics of test writing
- 07:08:50 [Hexcles]
- foolilp: is it the error message or wpt.fyi that's confusing?
- 07:09:03 [Hexcles]
- reillyg: e.g. the "expand" button on wpt.fyi
- 07:09:11 [Hexcles]
- lukebjerring: [explains the UI]
- 07:09:19 [jorydotcom]
- jorydotcom has joined #testing
- 07:09:41 [Hexcles]
- jgraham: what is the ideal state?
- 07:10:51 [Hexcles]
- reillyg: summary of all the tests failed per browser (referring to Blink layout test viewer: expected, actual, diff)
- 07:11:12 [shimazu]
- shimazu has joined #testing
- 07:11:43 [Hexcles]
- ... make it as obvious as possible -- a link to failing tests, especially in PR
- 07:12:13 [Hexcles]
- lukebjerring: wpt.fyi GitHub Checks are not great (see discussion yesterday)
- 07:13:21 [Hexcles]
- jgraham: the "diff" on wpt.fyi is different from the diff in Blink CI -- we don't have baselines
- 07:13:30 [Ms2ger]
- Ms2ger has joined #testing
- 07:13:58 [Hexcles]
- lukebjerring: perhaps expand the results by default
- 07:15:00 [Hexcles]
- reillyg: want a summary page of all browsers instead of five different links
- 07:15:54 [Hexcles]
- lukebjerring: probably better to build a new view for the PR use case
- 07:17:28 [Hexcles]
- jgraham: collapse everything into one single check
- 07:18:17 [xiaoqian]
- xiaoqian has joined #testing
- 07:18:36 [Hexcles]
- lukebjerring: open issue: attach a list of run IDs to a check run instead of using e.g. "chrome@SHA" to avoid a race condition and to be able to show pending status
- 07:19:50 [ato]
- RRSAgent: Please draft the minutes.
- 07:19:50 [RRSAgent]
- I'm logging. I don't understand 'Please draft the minutes.', ato. Try /msg RRSAgent help
- 07:19:52 [ato]
- RRSAgent: Please draft the minutes
- 07:19:52 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 07:20:00 [Hexcles]
- shimazu asked about sub test time out (one subtest time out makes the whole test time out)
- 07:21:00 [Hexcles]
- jgraham: promise_test is especially problematic because they run sequentially
- 07:21:21 [Hexcles]
- ... if a sub test times out the subsequent tests won't run
- 07:22:49 [Hexcles]
- lukebjerring: "difficult to determine subtest timeout" isn't a great argument. Being able to set subtest timeout is better and more meaningful than harness timoeout
- 07:23:08 [reillyg]
- q+
- 07:24:09 [Hexcles]
- jgraham: allowing authors to define subtest timeout is error-prone and flaky (e.g. a virtual environment can be paused)
- 07:24:48 [Hexcles]
- lukebjerring: but step_timeout takes multiplier into account; we can do the same for subtest timeout
- 07:25:20 [Hexcles]
- ... or we can set "short" "long" etc. timeout instead of concrete numbers
- 07:25:43 [Hexcles]
- q?
- 07:25:52 [Hexcles]
- ack reillyg
- 07:26:14 [Hexcles]
- reillyg: slow test is usually a test with too many subtests
- 07:26:25 [Hexcles]
- ... shall we prefer splitting those
- 07:26:52 [Hexcles]
- ... can the harness monitor the progress of subtests?
- 07:27:27 [Hexcles]
- jgraham: it sounds like a good design and possible, but we don't do that now (harness sends a JSON blob at the end)
- 07:28:43 [Hexcles]
- ... it doesn't solve your promise_test case?
- 07:28:44 [Hexcles]
- reillyg: combine the two: the harness monitors the promise chain
- 07:29:30 [Hexcles]
- jgraham: good idea
- 07:29:39 [zcorpan]
- zcorpan has joined #testing
- 07:30:22 [Hexcles]
- lukebjerring: e.g. similar to Travi (timeout if no output in 10m)
- 07:30:45 [Hexcles]
- jgraham: there's a lot of effort; testdriver.js & wptrunner need to be re-engineered
- 07:31:20 [Hexcles]
- ... testdriver.js already has a queue
- 07:32:41 [Hexcles]
- reillyg: improve the communication between harness and wptrunner. the test can declare the total number of subtests
- 07:32:49 [Hexcles]
- jgraham: there are more complicated cases
- 07:33:04 [Hexcles]
- foolip: for now we can split tests
- 07:33:23 [Hexcles]
- shimazu: yes we are doing that
- 07:33:46 [yhirano_]
- yhirano_ has left #testing
- 07:34:10 [Hexcles]
- shimazu: one more thing: service worker un/registration is expensive, which becomes a big overhead if we split promise tests
- 07:34:20 [xiaoqian]
- xiaoqian has joined #testing
- 07:35:08 [Hexcles]
- ... we use a promise test to do the setup
- 07:35:16 [Hexcles]
- foolip: this is the same as IDL harness test
- 07:35:38 [Hexcles]
- lukebjerring: this is not a great pattern
- 07:36:17 [Hexcles]
- foolip: we don't have an async setup function
- 07:36:28 [Hexcles]
- ... [explains how this breaks async_test]
- 07:37:02 [Hexcles]
- jgraham: yeah the lifecycle is tricky
- 07:37:32 [Hexcles]
- ... if we say "if you use async_setup you can only use promise_test" then we might be fine
- 07:38:44 [Hexcles]
- jgraham: we can have promise_setup!
- 07:39:24 [Hexcles]
- foolip: and emit errors if authors mix wrong setup/tests
- 07:40:20 [Hexcles]
- ... write an RFC!
- 07:41:00 [marcosc]
- marcosc has joined #testing
- 07:41:22 [Hexcles]
- lukebjerring: I imagine it's very common to have lots of promise tests stuffed in a test file to avoid the setup overhead
- 07:42:18 [jorydotcom]
- jorydotcom has joined #testing
- 07:42:46 [Hexcles]
- foolip: can we just call promise_setup setup? if it returns a thenable then it's a setup for promise_test
- 07:43:19 [gsnedders]
- jgraham: everyone loves things that are polymorphic on their return type, right?
- 07:43:30 [Hexcles]
- jgraham: put this as an option in the RFC
- 07:44:18 [Hexcles]
- lukebjerring: I actually sent an RFC earlier for explicit naming / against polymorphic parameters
- 07:45:26 [Hexcles]
- ACTION jgraham file an issue to keep running as long as they’re producing subtest results
- 07:45:39 [Hexcles]
- ACTION foolip: RFC for async setup() (or promise_setup())
- 07:45:48 [Hexcles]
- ACTION jgraham: file an issue to keep running as long as they’re producing subtest results
- 07:45:56 [JohnJansen]
- RRSAgent, make minutes
- 07:45:56 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 07:46:26 [Hexcles]
- foolip: switching topic to single page tests
- 07:46:41 [Hexcles]
- ... [explains single page tests]
- 07:46:50 [kevers]
- kevers has joined #testing
- 07:47:04 [Hexcles]
- ... it was added earlier to reduce boilerplate (similar to mochi tests)
- 07:47:28 [Hexcles]
- ... useful for async tests (saves a layer of wrapping)
- 07:47:39 [Hexcles]
- ... it's become problematic because of accidental opt-in
- 07:48:34 [Hexcles]
- ... ~600 tests that fail strangely because they accidentally trigger single page mode (e.g. they trigger an exception before starting any test)
- 07:49:20 [Hexcles]
- jgraham and foolip debating what result type is / and should be in this case
- 07:49:50 [Hexcles]
- foolip: [explains again when single page mode is triggered -- see RFC / docs]
- 07:50:13 [Hexcles]
- https://github.com/web-platform-tests/rfcs/pull/28
- 07:50:22 [Hexcles]
- lukebjerring: what's the motivation?
- 07:50:54 [Hexcles]
- jgraham: not for async tests, but e.g. something that responds to multiple events, each of which is a callback that needs to be wrapped in t.step
- 07:51:42 [Hexcles]
- ... if you don't wrap the callbacks, it ends up as a harness error
- 07:52:22 [Hexcles]
- ... so the real use case is to avoid wrapping every single callback
- 07:53:05 [Hexcles]
- foolip: my proposal is an explicit opt-in
- 07:53:17 [Hexcles]
- ... the real issue is difficulty to write correct async tests
- 07:53:32 [reillyg]
- q+ odejesush
- 07:53:34 [Hexcles]
- ... if we solve that we can get rid of single page tests altogether
- 07:54:10 [Hexcles]
- jgraham: I'm historically opposed to adding explicit opt-in to a feature which aims to avoid boilerplate
- 07:55:05 [Hexcles]
- lukebjerring: there are so few of single-page tests (~130), so adding a one-line opt-in isn't too bad
- 07:55:53 [Hexcles]
- jgraham: if there are only 130 of these -- although it seems to me this is a useful feature -- people are not using the feature much so it doesn't really matter?
- 07:55:56 [Hexcles]
- q?
- 07:55:59 [foolip]
- q+
- 07:56:11 [AutomatedTester]
- ack odejesush
- 07:56:53 [shimazu]
- shimazu has joined #testing
- 07:57:31 [Hexcles]
- odejesush: another issue: sometimes forget an event listener (e.g. for bluetooth disconnection)
- 07:58:40 [Hexcles]
- foolip: is the problem just forgetting to write an await
- 07:59:30 [Hexcles]
- reillyg: perhaps a linter?
- 07:59:36 [marcosc]
- marcosc has joined #testing
- 07:59:38 [Hexcles]
- ... rewrite all in TypeScript!
- 07:59:41 [Hexcles]
- (laugh)
- 07:59:44 [foolip]
- q?
- 07:59:50 [Hexcles]
- ack foolip
- 08:02:12 [Hexcles]
- foolip: [back to single-page tests] can we require setup.. explict done?
- 08:02:41 [Hexcles]
- jgraham: that sounds like a bad opt-in
- 08:04:14 [Hexcles]
- foolip: to fix 9000(?) tests, we can either add the t.step wrapper or turn them into single-page tests
- 08:04:38 [Hexcles]
- ... 9000 tests are the number of tests that have only one subtest
- 08:05:23 [shimazu]
- shimazu has joined #testing
- 08:05:27 [Hexcles]
- ... they are not necessarily broken
- 08:06:37 [JohnJansen]
- rrsagent, make minutes
- 08:06:37 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html JohnJansen
- 08:06:37 [Hexcles]
- ... but if there's a harness error in such case we could fix it by adding the t.step wrapper or turning them into single-page tests
- 08:06:48 [Hexcles]
- Topic: 2020 priorities
- 08:07:12 [Hexcles]
- https://docs.google.com/document/u/1/d/1gie7LFb6cAUfabY86MYuWM7m7ux_FaKhDkLdpz0zWkg/edit?usp=sharing
- 08:07:22 [Hexcles]
- foolip: last year we wrote lots of things
- 08:07:35 [Hexcles]
- ... but nobody looked at it over the year
- 08:07:45 [Hexcles]
- ... though we managed to get lots of them done
- 08:08:47 [MikeSmith]
- denis: https://www.w3.org/2019/09/17-testing-minutes.html has not been getting (re)generated for most of the day
- 08:09:51 [Hexcles]
- jgraham: this is not going to be something that has teeth, but something to align on
- 08:09:58 [MikeSmith]
- s/denis: https:\/\/www.w3.org\/2019\/09\/17-testing-minutes.html has not been getting (re)generated for most of the day/
- 08:10:08 [Hexcles]
- lukebjerring: let's enumerate stuff that's important
- 08:11:02 [Hexcles]
- Hexcles: can we start by going over action items?
- 08:11:57 [lajava]
- lajava has joined #testing
- 08:12:04 [Hexcles]
- jgraham: they are concrete things to do but not goals
- 08:12:21 [Hexcles]
- ... need to find the underlying theme
- 08:12:55 [zcorpan]
- zcorpan has joined #testing
- 08:12:58 [MikeSmith]
- actions?
- 08:14:16 [Hexcles]
- [please refer to https://docs.google.com/document/d/1gie7LFb6cAUfabY86MYuWM7m7ux_FaKhDkLdpz0zWkg/edit ]
- 08:14:19 [boaz]
- boaz has joined #testing
- 08:15:33 [Hexcles]
- [people are discussing wording of the goals]
- 08:15:40 [Hexcles]
- RRSAgent, make minutes
- 08:15:40 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html Hexcles
- 08:30:01 [zcorpan]
- zcorpan has joined #testing
- 08:30:31 [zcorpan]
- zcorpan has joined #testing
- 08:31:42 [zcorpan]
- q+
- 08:34:19 [zcorpan]
- ack zcorpan
- 08:35:07 [zcorpan]
- zcorpan has joined #testing
- 08:38:15 [lgrahl]
- lgrahl has joined #testing
- 08:41:54 [boaz]
- present+
- 08:42:45 [ato]
- RRSAgent: make minutes
- 08:42:45 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html ato
- 08:43:12 [ato]
- Present- Boaz_Sender
- 08:43:33 [tidoust]
- tidoust has joined #testing
- 08:45:20 [marcosc]
- marcosc has joined #testing
- 08:45:28 [rego]
- rego has joined #testing
- 08:46:40 [marcosc]
- marcosc has joined #testing
- 08:47:45 [foolip]
- q?
- 08:51:18 [BitBot]
- (14wpt) [PR] Honry 04closed 13#11600: [sensors] Bring setUpdateSensorReadingFunction from sensor mojo inter… - https://git.io/JeOzb
- 08:51:44 [denis]
- RRSAgent: make minutes
- 08:51:44 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html denis
- 08:53:48 [zcorpan]
- zcorpan has joined #testing
- 08:56:08 [marcosc]
- marcosc has joined #testing
- 08:58:03 [marcosc_]
- marcosc_ has joined #testing
- 09:05:28 [BitBot]
- (14wpt) [issue] Hexcles opened 13#19114: Add a QUIC server to test WebTransport - https://git.io/JeOzj
- 09:14:31 [zcorpan]
- https://bocoup.github.io/wpt-disabled-tests-report/
- 09:17:32 [Hexcles]
- s/TOPIC: Ergonomics//
- 09:17:38 [Hexcles]
- s/Topic: ergonomics of test writing//
- 09:18:00 [Hexcles]
- i/scribenick AutomatedTester/Topic: QUIC
- 09:18:16 [Hexcles]
- RRSAgent, make minutes
- 09:18:16 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html Hexcles
- 09:19:19 [Hexcles]
- i/ScribeNick: AutomatedTester/Topic: QUIC
- 09:19:21 [Hexcles]
- RRSAgent, make minutes
- 09:19:21 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html Hexcles
- 09:19:28 [denis]
- denis has joined #testing
- 09:23:58 [Hexcles]
- s|scibenick: Hexcles|https://github.com/web-platform-tests/wpt/issues/19114
- 09:24:07 [Hexcles]
- RRSAgent, make minutes
- 09:24:07 [RRSAgent]
- I have made the request to generate https://www.w3.org/2019/09/17-testing-minutes.html Hexcles
- 09:32:24 [Hexcles]
- Hexcles has joined #testing
- 09:49:23 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19115 merge into 07master: [LayoutNG] Correct overflow inside anonymous blocks inside %-insets. - https://git.io/JeOgM
- 09:51:00 [BitBot]
- (14wpt) [PR] Honry requested 13#19116 merge into 07master: Fixed lint error in #18886 - https://git.io/JeOgS
- 09:53:53 [plinss_]
- plinss_ has joined #testing
- 09:57:39 [BitBot]
- (14wpt) [PR] CYBAI requested 13#19117 merge into 07master: Fix ordering of script elements in `the-script-element/module/module-vs-script-*.html` tests - https://git.io/JeOgp
- 10:58:38 [marcosc]
- marcosc has joined #testing
- 11:08:04 [wanming]
- wanming has joined #testing
- 11:18:13 [BitBot]
- (14wpt) [PR] fred-wang requested 13#19118 merge into 07master: Add tests to check that legacy maction/semantics elements are treated… - https://git.io/JeOa3
- 11:38:03 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19119 merge into 07master: Block Xr.DeviceChange event without FeaturePolicy and add tests - https://git.io/JeOaS
- 11:38:28 [BitBot]
- (14wpt) [PR] chromium-wpt-export-bot requested 13#19120 merge into 07master: Split skip-not-fully-active.html into two tests - https://git.io/JeOa9
- 11:44:57 [cybai]
- cybai has joined #testing
- 11:46:33 [BitBot]
- (14wpt) [PR] fred-wang 03merged 13#19118 into 07master: Add tests to check that legacy maction/semantics elements are treated… - https://git.io/JeOa3
- 11:55:06 [marcosc]
- marcosc has joined #testing
- 11:55:36 [eeeps]
- eeeps has joined #testing
- 11:56:26 [marcosc_]
- marcosc_ has joined #testing
- 11:57:32 [marcosc]
- marcosc has joined #testing
- 11:58:44 [marcosc_]
- marcosc_ has joined #testing
- 12:02:17 [jorydotcom]
- jorydotcom has joined #testing
- 12:02:43 [birtles]
- birtles has joined #testing
- 12:06:13 [rniwa]
- rniwa has joined #testing
- 12:06:13 [marcosc]
- marcosc has joined #testing
- 12:21:50 [marcosc]
- marcosc has joined #testing
- 12:30:53 [xiaoqian]
- xiaoqian has joined #testing
- 12:34:36 [denis]
- denis has joined #testing
- 12:44:24 [BitBot]
- (14wpt) [PR] jdm 03merged 13#19117 into 07master: Fix ordering of script elements in `the-script-element/module/module-vs-script-*.html` tests - https://git.io/JeOgp
- 12:59:49 [karl]
- karl has joined #testing
- 13:04:03 [Zakim]
- Zakim has left #testing
- 13:06:35 [BitBot]
- BitBot has joined #testing
- 13:12:33 [BitBot]
- BitBot has joined #testing
- 13:22:32 [ato]
- RRSAgent: stop