Meeting minutes
Presentation
Francois: Let's get started. Some people here not from the WebDX CG, which is great. The first part will be about the web-features project, developed by the WebDX CG.
Francois: Today, we’re going to talk about gathering signals to inform the W3C standardization process, using compatibility data maintained in the web-features project.
… Before we get there, it seems useful to explain what the web-features project is to get everyone on the same page.
… And that all starts with web developers.
Francois: If you’re heavily involved in standardization, you may sometimes forget that there is a whole world of web developers out there, who heavily depend on web technologies and who may not care at all about how the sausage is made. Which strikes me as completely fine!
… What matters more for web developers is to understand what functionality of the web platform they can use without problems in their applications.
… This is where the concept of features arises.
Francois: The notion of “feature” is meant to identify what web developers get and what they want from the web platform.
… Features are coarser-grained concept than individual functionality that specs may define. Features may map to an entire spec, to functionality defined within a single spec, or to functionality defined across multiple specs.
… What matters is that features are meant to capture the perspective of web developers.
… As such, any interoperability data that we can associate with a feature is going to be directly usable by web developers.
Francois: Actually, features appear early in the process, before incubation or standardization work even starts.
… Then they go through the standardization process.
… And then, ideally, they reach browsers, and get used by web developers.
… Ideally, we’d be able to track features from research to adoption.
Francois: For that, it would be good to converge on a shared catalog of web features.
… And that’s exactly the purpose of the web-features project, developed by the WebDX Community Group.
… Part of this project is to associate an implementation status to each feature so that developers can answer the question “Can I reasonably use this feature without having to worry about what browser and browser version my users actually use?”
… That implementation status is what we call “Baseline”.
Francois: Baseline has two milestones.
… A feature becomes Baseline as soon as it’s supported across latest versions of main browsers. That’s the “Baseline low” milestone.
… Then, after some time, a vast majority of users will have updated their browser to a codebase that includes the feature. When that happens, the feature becomes “Baseline high”. It signals that the feature is well established.
… I’m not going into details here, what matters is that the web-features project contains features and Baseline statuses for these features, and the whole thing is meant to capture a web developers perspective on the web platform.
… Also, under the hoods, the project uses support information from the MDN browser-compat-data project (or BCD). Effectively, features in the web-features project come with a mapping to BCD keys.
Francois: The WebDX Community Group has been working hand in hand with MDN and Can I Use on the concept of Baseline.
… The Baseline banners that you may have noticed on these websites recently use data from the web-features project under the hoods.
Francois: The WebDX Community Group is now working with other projects.
… That includes integration in Web Platform Tests and integration in State of CSS, State of JS and State of HTML surveys.
… But the question for today is: what about integration in the standardization process?
Francois: That is an excellent question, let’s look into it!
Francois: The W3C Process document already uses the concept of features.
… Features are notably used to assess “adequate implementation experience” and to set requirements at certain maturity levels.
… One example is the identification of “features at risk” before publication of a spec as a Candidate Recommendation.
Francois: Now, if we look at a simplified view of the W3C Recommendation track, we’ll get something like the following.
… Specs typically enter the Recommendation track after some period of incubation.
… Then they progress to Candidate Recommendation.
… And if all goes well, most of them will eventually be published as Recommendations.
… Transitions between each of these stages come with a set of requirements.
… It’s a simplified view. Some specs may stay at the Candidate Recommendation phase for instance.
Francois: Anyway, let’s see if we can combine the standardization perspective with the developers perspective.
… The standardization process does not require implementations before a spec reaches Proposed Recommendation.
… If the standardization process works as intended, features defined in a spec should only become Baseline late in the standardization process, and certainly *after* the spec has reached the Candidate Recommendation stage.
… The schema represented here is even wrong. “Baseline low” and “Baseline high” should be further to the right, I just didn’t have enough space on the slide!
Francois: That gives us an opportunity to detect divergences between the developers perspective and the standardization perspective, namely:
… Specs that define Baseline features but that are still being incubated probably signal a failure to standardize.
… Specs that define Baseline features but that are still Working Drafts may signal undue delays in the standardization process.
… On the contrary, Recommendations that got published some time ago but that still define non-Baseline features may be worth looking into.
… Finally, looking at the interval between publication as Recommendation and features reaching Baseline status could be useful to evaluate whether the process matches “reality”, meaning the developers perspective.
Francois: I’ve started to look into computing these signals.
… This is very much early stage and work in progress!
Francois: Code is available in a GitHub repository.
… It combines the Baseline information from web-features and the standardization status of specs retrieved from the browser-specs project, using BCD to create a mapping between features and specs.
… The code compiles a short listing with the first three indicators that I mentioned.
… The data used by the code does not contain publication dates, so no analysis of the interval between publication as Recommendation and features turning Baseline for now!
Francois: What does this exploration reveal?
… It identifies a few specs that are late “incubations”. Some of them are known issues: requestVideoFrameCallback(), Media Playback Quality. Note that the term “incubation” here also includes specs that are already in scope of a Working Group but that have not been published as First Public Working Drafts yet.
… It also flags a number of Working Drafts that could perhaps be worth publishing as Candidate Recommendations because at least some of the features they define have turned Baseline!
… It also reports a couple of Recommendations that are still not supported across browsers.
… We can come back to this list during discussion…
Francois: But before we do that, I wanted to note that this is all clunky, for different reasons.
… There aren’t so many features in the web-features project for now. I’m actually cheating in the code to extend coverage, pretending that each spec defines a single “feature” and doing the mapping to BCD keys myself. That’s far from ideal.
… Talking about BCD keys, they often map to unversioned spec URLs. It’s not clear how to map that to a given spec level, for example in the case of CSS specs.
… The analysis of Recommendations is incomplete too, because browser-specs focuses on listing the latest drafts of specs in a series.
Francois: There are also well-known reasons for delays in the standardization process. Some of them are listed here.
Francois: I listed here a couple of questions that I thought might come naturally.
… First one is: “Couldn’t you just have used Web Platform Tests to compute an interoperability status for specs?” In practice, that’s hard, because no one gets a 100% pass score on Web Platform Tests, and there’s no way to tell which failures could be ignored at first sight.
… Second one is: “Couldn’t you just have used BCD to compute an interoperability status for specs?” That’s what the code actually does to complete coverage, but that’s not ideal. First, focusing on specs instead of on features does not create a good developers perspective. Also, many specs define multiple features that seem worth analyzing separately. A good example is the Web Speech API. Speech synthesis would get reported in the list of “Late incubations” if it existed in the web-features project. It does not appear because the spec also contains speech recognition and the code cannot separate the two.
Francois: That’s it, thanks, up to you!
… I have a couple of suggestions for you here, but feel free to kickoff the discussion with something else!
Discussion
Vadmin: What about TC39/WhatWG stages mapping to baseline?
Francois: For TC39, the notion of stages can probably be taken into account easily.
… For WHATWG, it may be harder. It's a live document, so not sure how things would work.
Dom: The WHATWG stages, there's only one (3) that has a link to an implementation.
… TC39 stage 4 is where we'd expect baseline.
… One interesting lesson from web-features project, what does it mean for a feature to be baseline if it's not gone through the full process, or if it's not accessible in practice.
… Looking at web-features throughout their lifetime reveals a number of important questions.
Anssi: Do you consider the quality of implementation when thinking of baseline?
… It can be performance, ux, etc. WebGPU API for example which the software implementation might perform poorly.
… Is this a consideration? Do developers care?
Francois: Baseline status is meant to capture developer perspective. So if it's important to them, the status should capture it.
… The definition will continue to evolve over time too. Every year.
… There could be a need to look into other things than what we look at today.
… We rely on BCD, and BCD has a flag saying a thing is partially implemented. So if there's an important bug, we'll flag it. Not sure about UX though.
Dom: The point about accessibility is related here.
Francois: Good point. But also, the browser stops somewhere and then assistive technology takes over.
Anssi: can you extend BCD with additional fields? Can we extend the format to provide these additional information.
Florian: For accessibility in particular. BCD only targets browsers right now. But BCD could contribute to accessibility-support.io.
… Or use the partial-implementation flag to contain some of this data.
… openwebdocs/
Fantasai: You had a diagram that baseline should happen after REC. But there are a lot of cases where a thing will be widely available way before that. And we can't get to REC because of bugs. This happens frequently.
Francois: Totally agree. One question I have is: wondering if these signals will be useful. They will show the divergence between the developer perspective and the standardization process.
… At a minimum it signals something.
Fantasai: it's useful to have this signal. It points to where we need more effort.
Francois: My experience also is that Working Groups, more and more, have a tight feedback loop where they implement the spec, and the code.
… That delays the later stages. There's always something that remains to be done.
… Find it harder to push groups to move to Candidate Recommendation.
Fantasai: If the blocker is implementer experience, they should go to Candidate Recommendation. That's what it's for.
Francois: agreed. Not the feedback I'm seeing from WGs though.
Dom: On the divergence between standardization status and baseline status. In many cases it might be OK that a feature hasn't reached Candidate Recommendation.
… In other cases it might be problematic, a11y, localization, etc. issues.
fantasai: I think we should push back against the feeling that a spec needs implementation experience to go to Candidate Recommendation. We want groups to get wide review, and to have patent licensing in place, before wide deployment ideally, and going to CR ensure sthat.
Dom: The gap between specs and impl creates a risk for the platform that doesn't feel great.
Francois: Great point.
… Question around whether we need to look at other signals. In this exploration, I'm not sure it revealed things we didn't know already.
… Signals can be useful to give a perspective to other groups. But are there other signals we may try to compute?
… In the WebDX CG, we're also starting to looking into gathering developer signals in the broad sense.
… The overal goal of the project is to catalog the web features with identifiers. And if these identifiers are used across various sources, we can collect signals on features.
… Other thoughts that come to mind?
Dom: Building on what you said earlier. If you're in the deep on some of these incubations, you know fully well the details.
… Computing this info on a regular basis, exposing to the right folks (not sure who yet), makes sure we keep ourselves accountable.
… Having the infrastructure to emit the signals sounds like a good thing.
Anssi: Wondering if you've been thinking about gathering signals for web devs who are app-centric? What do these folks need?
… Feel like there's a lot of interest. But not enough visibility into the pain points.
… is this in scope for signals?
<dom> web-platform-dx/
Francois: Sounds like a useful thing to track. We need to target the developers who work on these types of apps (PWA) and get signals from them. Any ideas where to get them from?
<dom> web-platform-dx/
Francois: We may be able to get this via research. Which is a bit hard to conduct.
Anssi: State of * surveys. Might be possible to drill into this feedback.
Francois: We want to link this to web-features, definitely. Maybe there are other signals that can be more easily collected. Usage stats?
Vadim: feature grouping is the key advantages from web-features. But challenge to create them and align them with BCD. How is this going?
Francois: we have a lot of terminology. It's a challenge for sure. Lot's of discussions needed to converge on the definition of groups.
<dom> https://
Francois: Florian and Daniel, here, are doing a fantastic job on this. Can't thank them enough. Learning as we go.
Daniel: I'll add one thing. For contemporary features, it's straightforward, we can see what people are talking about.
… the challenge is about generalizing this for the entire web platform. How do we combine this with historical facts.
… some sub parts of arrays used to be new, and separate, at some point in the past. Now they're just part of arrays.
… What's a feature? A thing you can use? How do you relate that the evolution of the web and standards?
Anssi: responding to Kadir on the chat "can you talk to where you would use state of PWA"?
… we want to know what the gaps are for apps on the web.
… Every company has their own story, but we want to know what do developers really want for apps on the web.
Florian: Two weeks ago we had Andrzej from the gaming JS community on the WebDX call. This was very useful to have his perspective on surveys that he runs.
… Talking to specific communities helps a lot. I would be very interested in hearing from them if they consider some gaming feature baseline for example.
Anssi: agree. Would make baseline easier to relate to. Making it more concrete. What is the baseline for app developers, for game developers, etc. Probably a subset of the current baseline.
Francois: need to extend the coverage of the web-features project first.
Francois: In terms of signals. I'll try to make a dashboard visible somewhere.
… Trying to provide the signals to the relevant people when they need it.