Meeting minutes
background
Philip: this is the 2nd year MDN is running a big developer survey, it ran for the first time last year with 28K responses
… the findings had a lot to day with browser compat, testing, and inaccurate doc for frameworks/libraries
… we designed a complementary survey which also ran on MDN in 2020, the browser compat survey, completed with dev interviews
… had a lot to do about layout and styling, mostly with flexbox and grid
Philip: We're running the survey in 2020, has been running the past couple of weeks
… so far we have received 5,225 complete responses (fewer than last year)
… In terms of target audience, the gating question is doing some amount of coding for the Web
… so not for people who have abandonned the Web, or do design only without coding
… a new question in 2020 was about the environment of usage of Web technologies - the results show mostly still Web apps and pages (85%)
… in terms of who's taking the survey - a very large proportion of very experienced Web developers (29.5% with more than 10 years)
… don't know if the answers to gender have evolved since last year
… this year, we have asked about belonging to a minority group - shows ~17% identify as such (which will be able to use to build specific views of the results)
… in terms of origins of results, Germany and Russia are tied for #2
… overall, pretty broad representation
… The survey was translated in ~7 languages
… partly new this year - taking temperature on what different part of developers experiences are doing
… interpretation is a bit tricky
… the level of overall satisfaction hasn't changed much (~77%)
… browser compat remains a source of dissatisfaction
… but also shows neither satisfaction nor dissatisfaction with automated browser testing, perf, access, privacy & security improvement tools
… with some notable level of dissatisfaction (in particular for privacy/security)
… perhaps due to the many changes happening wrt privacy in browsers, where behavior vary across browsers
… The meat of the survey is the ranking exercise: you got 6 options at a time, pick the most/least frustrating of the options, multiple times
… this builds a ranking of the pain points
… the top 5 are similar to last year
… notably, the library documentation bit is no longer #2 - not sure how significant it is
… (actual change vs change of population of respondents)
… at the bottom of the list, documentation for the platform itself (HTML, CSS, JS) seems to be in a good shape
… maybe that's because people coming from MDN and interpret this as saying they're liking MDN? also a testament to the quality of the documentation in these areas
… (it ranks lower in pain compared to last year)
… Another addition this year: in what way do they participate in building the Web platform
… unsurprising, "none of the above" is the biggest number; very few contributions to WPT not surprising either
… Another repeat from last year is the question about browsers you support when developing
… We did change the question about browsers creating issues independent of whether they support
… IE creates issues for 60%, despite having IE only supported by ~20% - stark and interesting difference
… The following questions are only asked to the subset of participants who've said they use a given technology
… a large majority use CSS - the biggest pain point there is creating the layout specified
… different from e.g. JS where the issue will support for a given feature
… for HTML, it's customizability (e.g. form styling)
… Another addition this year: Web testing - it was unclear last year when we got answers if the frustration there was for manual testing or automated testing
… which this question helps shed some light
… 24% say time spent on manual testing, but also some level of pain in setting up and running automated tests
… also new for this year, a more specific question on accessibility
… A big part of the problem seems to be not knowing how to do it (44%), and lack of management support (21%)
… it sounds like the fact that accessibility ranked low in frustration last year was not because it's easy, but because it hasn't been applicable for many
… still on accessibility, more than half people don't test
… in terms of tooling, most of the people that took the survey have some level of agency on the tools they're using
Robert: do you believe the results are representative for all developers?
Philip: don't think they are - the completion rate went down from 30% last year to 22%
… the survey is longer with more general upfront questions which may not make sense from developers
… the ranking exercise where about 50% developers drop off - it's pretty time consuming
… it's also fairly similar at that stage from the one last year
… given the level of drop-off, we can't compare directly to last year results; we will do calibration to help with comparability
… we don't know with confidence how representative the results are
… but consistency with results from other surveys, last year results will help build that confidence
Philip: the survey will run until Nov 2
Dom: Survey lessons is key in understanding the needs from developers on the platforms - part of our bigger understanding of what our stakeholders need (which need also to integrate end users)
Boaz: the survey was devleoped by other contributors, beyond Chrome, W3C - we Bocoup were involved in the design via the MDN porduct advisory board
Jemma: thanks for the presentations - the big pain points wrt accessibility are very useful
… 44% being the biggest barrier - the ARIA APT group is looking at this space
Boaz: the group is providing guidance on how to use ARIA
… Aria practices Guidelines is being looked to be integrated in the WAI Education & Outreach efforts (instead of a WG Note which is not very legible to many developers)
… we've received lots of contributions on the guidelines
… it would be great ot measure if that has an impact on accessibility adoption
Jemma: another thought - I'm using MDN doc a lot, as co-chair of the WG, how can we build synergy with APG?
Boaz: I've raised this with Chris Mills, MDN content lead at Mozilla
… he's very enthusiastic about incorporate aria practices in MDN
… which also goes along the goal of making it more modular
<marie> dom: MDN PAB: https://developer.mozilla.org/en-US/docs/MDN/MDN_Product_Advisory_Board
Mike: in terms of what we could do better at W3C to act on these data, there are a lot of work we could be doing
… but that'll require effort
… we've been talking about setting up a dev advocate in every WG
… i've been working a lot on MDN & BCD personally - it would be nice to have more contributors
<MikeSmith> https://github.com/mdn/sprints/issues/3722
Mike: as an example - a few weeks ago, something came up a few weeks ago about sendBeacon
… this arose from a blog post
… saying that it is broken
… that level of frustration should be a source of learnings
… which I brought as an issue in MDN sprints to see if the MDN documentation can address the identified pain points
https://volument.com/blog/sendbeacon-is-broken
James: looking at some of the results, Philip touched on some of the things that are hard to understand, e.g. the browser support vs source of issues
<Jemma> My feedback to next year's survey would be focusing on more developer specific info and needs. For example, the most used screen reader survey ansewr can be found in web aim survey but the response is different because this MDN survey is rather targeting developers group. I think the biggest pain point question for web accessiblity was great info from the survey.
James: also surprising level of support for FF/Android compared to Safari/iOS (which doesn't align with actual compat issues with FF)
… compared to last year, I'm struggling to see obvious big conclusions we can draw from things
… the follow up study last year produced really actionable insights, probably worth doing it again this year
… but maybe for next year, instead of a giant survey we should be doing deep-dive info gathering to resolve ambiguities
… might be more actionable
Philip: we should go to the next year discussion
… I agree we will need follow up surveys
… last year we did this in ad-hoc fashion
… this year we have actual researchers to help us do that
… we need to figure out what actionable information we want to gather
… I have a few candidates
Sheila: lots of good ideas for next year surveys, here are some of the thoughts we have already identified
… we want to make sure the results are more representative of the web dev community at large
… if it's not fully representative (our hypothesis), it limits its usability to inform our action
… some of the ideas we've began to float is to gather input earlier on the research and recruitment process
… with people from more backgrounds to ensure the survey reflects the perspectives more people
… we are also thinking of identifying more demographics segmentation, e.g. barriers that impact disproporationate some segments of the community
… then there is active recruitment of respondents, e.g. from underrepresented community
… goal is to have a transparent & inclusive process
… we'll continue to collect thoughts on this
Jemma: +1 on the importance of research design
… on accessibility, the 2nd barrier was from decision makers
… having actionable information for decision makers would be great
… in this survey, the iOS screen reader is the most popular, which differs from the WebAIM survey
… which is probably explained by the different community here (of developers)
… so: more research design focused on developer responses, and questions that can be used for decision making process
<marie> dom: survey runs until 2 Nov. Please broadcast