13:58:05 <RRSAgent> RRSAgent has joined #mdn-dna
13:58:05 <RRSAgent> logging to https://www.w3.org/2020/10/27-mdn-dna-irc
13:58:08 <Zakim> RRSAgent, make logs Public
13:58:09 <Zakim> Meeting: MDN Developer Need Assessments: results and next steps
13:58:18 <smcgruer_[EST]> smcgruer_[EST] has joined #mdn-dna
13:58:54 <dom> Present+ Christian, Rego, Robert_Nymal, Zhoudan, Marie-Claire_Forgue, Dominique_hazael-Massieux, Atsushi_Shimono
14:00:34 <foolip> present+
14:00:45 <atsushi> atsushi has joined #mdn-dna
14:00:52 <dom> Present+ John_Rivielle, Miriam_Suzanne, Boaz_Sender, Mason_Freed, Mehmet_Oguz_Derin, Sheila_Moussa, Stephen_McGruer, Rick_Byers, Diekus,
14:03:02 <ali> ali has joined #mdn-dna
14:03:18 <boazsender> boazsender has joined #mdn-dna
14:03:27 <boazsender> present+
14:03:28 <jgraham> jgraham has joined #mdn-dna
14:03:38 <JonDavis> JonDavis has joined #mdn-dna
14:03:45 <JonDavis> JonDavis has left #mdn-dna
14:03:48 <miriam> miriam has joined #mdn-dna
14:04:43 <Jemma> Jemma has joined #mdn-dna
14:05:04 <Jemma> present+
14:05:40 <JonDavis_> JonDavis_ has joined #mdn-dna
14:05:59 <dom> Present+ Nicolas_Pena_Moreno, James_Graham
14:06:05 <dom> Topic: background
14:06:28 <dom> Philip: this is the 2nd year MDN is running a big developer survey, it ran for the first time last year with 28K responses
14:06:29 <rachelandrew> rachelandrew has joined #mdn-dna
14:06:48 <dom> ... the findings had a lot to day with browser compat, testing, and inaccurate doc for frameworks/libraries
14:07:17 <dom> ... we designed a complementary survey which also ran on MDN in 2020, the browser compat survey, completed with dev interviews
14:07:35 <dom> ... had a lot to do about layout and styling, mostly with flexbox and grid
14:07:47 <dom> -> https://mdn-web-dna.s3-us-west-2.amazonaws.com/MDN-Browser-Compatibility-Report-2020.pdf  Browser Compat report
14:08:12 <dom> Philip: We're running the survey in 2020, has been running the past couple of weeks
14:08:27 <dom> ... so far we have received 5,225 complete responses (fewer than last year)
14:09:14 <dom> ... In terms of target audience, the gating question is doing some amount of coding for the Web
14:09:27 <dom> ... so not for people who have abandonned the Web, or do design only without coding
14:09:52 <dom> ... a new question in 2020 was about the environment of usage of Web technologies - the results show mostly still Web apps and pages (85%)
14:10:37 <dom> ... in terms of who's taking the survey - a very large proportion of very experienced Web developers (29.5% with more than 10 years)
14:11:05 <dom> ... don't know if the answers to gender have evolved since last year
14:11:12 <Jon_Davis> present+
14:11:39 <dom> ... this year, we have asked about belonging to a minority group - shows ~17% identify as such (which will be able to use to build specific views of the results)
14:12:00 <dom> ... in terms of origins of results, Germany and Russia are tied for #2
14:12:08 <dom> ... overall, pretty broad representation
14:12:21 <dom> ... The survey was translated in ~7 languages
14:12:35 <MikeSmith> MikeSmith has joined #mdn-dna
14:12:44 <dom> ... partly new this year - taking temperature on what different part of developers experiences are doing
14:12:57 <dom> ... interpretation is a bit tricky
14:13:15 <dom> ... the level of overall satisfaction hasn't changed much (~77%)
14:13:29 <dom> ... browser compat remains a source of dissatisfaction
14:14:02 <dom> ... but also shows neither satisfaction nor dissatisfaction with automated browser testing, perf, access, privacy & security improvement tools
14:14:25 <dom> ... with some notable level of dissatisfaction (in particular for privacy/security)
14:14:46 <dom> ... perhaps due to the many changes happening wrt privacy in browsers, where behavior vary across browsers
14:15:16 <dom> ... The meat of the survey is the ranking exercise: you got 6 options at a time, pick the most/least frustrating of the options, multiple times
14:15:25 <dom> ... this builds a ranking of the pain points
14:15:31 <dom> ... the top 5 are similar to last year
14:15:47 <dom> ... notably, the library documentation bit is no longer #2 - not sure how significant it is
14:16:03 <dom> ... (actual change vs change of population of respondents)
14:16:34 <dom> ... at the bottom of the list, documentation for the platform itself (HTML, CSS, JS) seems to be in a good shape
14:17:00 <dom> ... maybe that's because people coming from MDN and interpret this as saying they're liking MDN? also a testament to the quality of the documentation in these areas
14:17:12 <dom> ... (it ranks lower in pain compared to last year)
14:17:25 <dom> ... Another addition this year: in what way do they participate in building the Web platform
14:17:48 <dom> ... unsurprising, "none of the above" is the biggest number; very few contributions to WPT not surprising either
14:18:20 <dom> ... Another repeat from last year is the question about browsers you support when developing
14:18:53 <dom> ... We did change the question about browsers creating issues independent of whether they support
14:19:16 <dom> ... IE creates issues for 60%, despite having IE only supported by ~20% - stark and interesting difference
14:19:42 <dom> ... The following questions are only asked to the subset of participants who've said they use a given technology
14:20:04 <dom> ... a large majority use CSS - the biggest pain point there is creating the layout specified
14:20:26 <dom> ... different from e.g. JS where the issue will support for a given feature
14:20:43 <dom> ... for HTML, it's customizability (e.g. form styling)
14:21:26 <dom> ... Another addition this year: Web testing - it was unclear last year when we got answers if the frustration there was for manual testing or automated testing
14:21:32 <dom> ... which this question helps shed some light
14:22:02 <dom> ... 24% say time spent on manual testing, but also some level of pain in setting up and running automated tests
14:22:13 <dom> ... also new for this year, a more specific question on accessibility
14:22:35 <dom> ... A big part of the problem seems to be not knowing how to do it (44%), and lack of management support (21%)
14:23:12 <dom> ... it sounds like the fact that accessibility ranked low in frustration last year was not because it's easy, but because it hasn't been applicable for many
14:23:54 <dom> ... still on accessibility, more than half people don't test
14:24:50 <dom> ... in terms of tooling, most of the people that took the survey have some level of agency on the tools they're using
14:27:10 <dom> Robert: do you believe the results are representative for all developers?
14:27:33 <dom> Philip: don't think they are - the completion rate went down from 30% last year to 22%
14:27:51 <dom> ... the survey is longer with more general upfront questions which may not make sense from developers
14:28:05 <dom> ... the ranking exercise where about 50% developers drop off - it's pretty time consuming
14:28:16 <dom> ... it's also fairly similar at that stage from the one last year
14:28:56 <dom> ... given the level of drop-off, we can't compare directly to last year results; we will do calibration to help with comparability
14:29:12 <dom> ... we don't know with confidence how representative the results are
14:29:31 <dom> ... but consistency with results from other surveys, last year results will help build that confidence
14:30:18 <dom> Philip: the survey will run until Nov 2
14:31:20 <dom> Present+ Rachel_Andrew, Mike5, Jon_Davis, Jen_Simmons, James_C, JaEun_Jemma_Ku, Ida, Florian_Scholz
14:33:05 <jgraham> present+
14:35:26 <Jemma> q+
14:35:30 <dom> Dom: Survey lessons is key in understanding the needs from developers on the platforms - part of our bigger understanding of what our stakeholders need (which need also to integrate end users)
14:36:06 <dom> Boaz: the survey was devleoped by other contributors, beyond Chrome, W3C - we Bocoup were involved in the design via the MDN porduct advisory board
14:36:20 <marie> ack jemma
14:36:24 <dom> Jemma: thanks for the presentations - the big pain points wrt accessibility are very useful
14:36:53 <dom> ... 44% being the biggest barrier - the ARIA APT group is looking at this space
14:37:10 <dom> Boaz: the group is providing guidance on how to use ARIA
14:38:01 <dom> ... Aria practices Guidelines is being looked to be integrated in the WAI Education &  Outreach efforts (instead of a WG Note which is not very legible to many developers)
14:38:12 <dom> ... we've received lots of contributions on the guidelines
14:38:20 <dom> ... it would be great ot measure if that has an impact on accessibility adoption
14:39:33 <dom> Jemma: another thought - I'm using MDN doc a lot, as co-chair of the WG, how can we build synergy with APG?
14:40:03 <dom> Boaz: I've raised this with Chris Mills, MDN content lead at Mozilla
14:40:20 <dom> ... he's very enthusiastic about incorporate aria practices in MDN
14:40:28 <dom> ... which also goes along the goal of making it more modular
14:42:13 <marie> dom: MDN PAB: https://developer.mozilla.org/en-US/docs/MDN/MDN_Product_Advisory_Board
14:45:19 <dom> Mike: in terms of what we could do better at W3C to act on these data, there are a lot of work we could be doing
14:45:25 <dom> ... but that'll require effort
14:45:52 <dom> ... we've been talking about setting up a dev advocate in every WG
14:46:39 <dom> ... i've been working a lot on MDN & BCD personally - it would be nice to have more contributors
14:46:44 <Jemma> q+
14:46:58 <MikeSmith> https://github.com/mdn/sprints/issues/3722
14:47:12 <dom> ... as an example - a few weeks ago, something came up a few weeks ago about sendBeacon
14:47:29 <dom> ... this arose from a blog post
14:47:46 <dom> ... saying that it is broken
14:47:59 <dom> ... that level of frustration should be a source of learnings
14:48:19 <dom> ... which I brought as an issue in MDN sprints to see if the MDN documentation can address the identified pain points
14:48:44 <dom> https://volument.com/blog/sendbeacon-is-broken
14:50:32 <dom> James: looking at some of the results, Philip touched on some of the things that are hard to understand, e.g. the browser support vs source of issues
14:50:59 <Jemma> My feedback to next year's survey would be focusing on more developer specific info and needs. For example, the most used screen reader survey ansewr can be found in web aim survey but the response is different because this MDN survey is rather targeting developers group. I think the biggest pain point question for web accessiblity was great info from the survey.
14:51:06 <dom> ... also surprising level of support for FF/Android compared to Safari/iOS (which doesn't align with actual compat issues with FF)
14:51:48 <dom> ... compared to last year, I'm struggling to see obvious big conclusions we can draw from things
14:52:20 <dom> ... the follow up study last year produced really actionable insights, probably worth doing it again this year
14:52:46 <dom> ... but maybe for next year, instead of a giant survey we should be doing deep-dive info gathering to resolve ambiguities
14:52:50 <dom> ... might be more actionable
14:53:04 <Jemma> q?
14:53:30 <dom> Philip: we should go to the next year discussion
14:53:38 <dom> ... I agree we will need follow up surveys
14:53:47 <dom> ... last year we did this in ad-hoc fashion
14:53:56 <dom> ... this year we have actual researchers to help us do that
14:54:15 <dom> ... we need to figure out what actionable information we want to gather
14:54:20 <dom> ... I have a few candidates
14:55:24 <dom> Sheila: lots of good ideas for next year surveys, here are some of the thoughts we have already identified
14:55:47 <dom> ... we want to make sure the results are more representative of the web dev community at large
14:56:06 <dom> ... if it's not fully representative (our hypothesis), it limits its usability to inform our action
14:56:18 <Jemma> q+
14:56:23 <dom> ... some of the ideas we've began to float is to gather input earlier on the research and recruitment process
14:56:46 <dom> ... with people from more backgrounds to ensure the survey reflects the perspectives more people
14:57:16 <dom> ... we are also thinking of identifying more demographics segmentation, e.g. barriers that impact disproporationate some segments of the community
14:57:30 <dom> ... then there is active recruitment of respondents, e.g. from underrepresented community
14:57:39 <dom> ... goal is to have a transparent & inclusive process
14:57:49 <dom> ... we'll continue to collect thoughts on this
14:58:09 <dom> Jemma: +1 on the importance of research design
14:58:12 <dom> ack Jemma
14:58:41 <dom> ... on accessibility, the 2nd barrier was from decision makers
14:58:48 <dom> ... having actionable information for decision makers would be great
14:59:16 <dom> ... in this survey, the iOS screen reader is the most popular, which differs from the WebAIM survey
14:59:27 <smcgruer_[EST]> smcgruer_[EST] has left #mdn-dna
14:59:33 <dom> ... which is probably explained by the different community here (of developers)
14:59:54 <dom> ... so: more research design focused on developer responses, and questions that can be used for decision making process
15:01:00 <marie> dom: survey runs until 2 Nov. Please broadcast
15:01:13 <Jon_Davis> Jon_Davis has left #mdn-dna
15:01:15 <dom> RRSAgent, draft minutes v2
15:01:15 <RRSAgent> I have made the request to generate https://www.w3.org/2020/10/27-mdn-dna-minutes.html dom
15:17:46 <foolip> jgraham: Oddly high numbers for Firefox Android showed up in the browser compat survey too. Filtering to people who had made sense in a free form question made that and other results seem more sensible. Something similar is probably happening in DNA 2020.
16:39:15 <jgraham> So we think there's probably a lot of "junk" responses for some reason?
17:13:59 <Zakim> Zakim has left #mdn-dna