06:31:59 RRSAgent has joined #webdx 06:32:03 logging to https://www.w3.org/2023/09/13-webdx-irc 06:32:03 RRSAgent, do not leave 06:32:03 RRSAgent, make logs public 06:32:04 Meeting: Developer experience research and spec development 06:32:04 Chair: François Daoust 06:32:04 Agenda: https://www.w3.org/2023/Talks/TPAC/breakouts/developer-research/ 06:32:06 Slideset: https://www.w3.org/2023/Talks/TPAC/breakouts/developer-research/ 06:32:08 clear agenda 06:32:12 agenda+ Pick a scribe 06:32:14 agenda+ Reminders: code of conduct, health policies, recorded session policy 06:32:18 agenda+ Goal of this session 06:32:20 agenda+ Discussion 06:32:22 agenda+ Next steps / where discussion continues 06:32:38 Ian has left #webdx 07:06:39 Ian has joined #webdx 07:06:43 Ian has left #webdx 07:09:35 Ian has joined #webdx 07:09:37 Ian has left #webdx 09:22:05 zcorpan has joined #webdx 09:27:19 patrickbrosset has joined #webdx 09:33:07 patrickbrosset has left #webdx 12:32:52 RRSAgent has joined #webdx 12:32:52 logging to https://www.w3.org/2023/09/13-webdx-irc 12:33:03 saschanaz has joined #webdx 12:35:00 JaseW has joined #webdx 12:36:00 cpn has joined #webdx 12:36:10 zcorpan has joined #webdx 12:36:15 ddbeck has joined #webdx 12:36:19 present+ Chris_Needham 12:36:21 present+ 12:37:56 whimboo_ has joined #webdx 12:37:56 ..Agenda.. .Scribe, picked, code of conduct, we're not recording 12:37:56 scribenick: JaseW 12:37:56 ...Goal is yto brielfy present what we've been playing with in the Web DX community group and exchange on IDs, how we could use the research in a more configurated manor 12:38:28 ...This is a continuation of a discussion started in TPAC last year in 2022 12:38:33 jgraham_ has joined #webdx 12:38:37 present+ 12:38:46 ..There are different research needs that go with it, we're going to focus on the middle part. When you're a group you want to focus on the API design and want feedback from developers 12:39:01 ..how do you get that feedback in a meaningful way 12:39:10 RRSAgent: make minutes 12:39:11 I have made the request to generate https://www.w3.org/2023/09/13-webdx-minutes.html zcorpan 12:39:25 ..one of the workstreams is the research workstream, its stated goal is to accelerate and improve the decision making. 12:39:53 RRSAgent: make logs public 12:39:54 ...This group is composed of browser vendors, but also MDN, open Web Docs, plus some others 12:40:20 ...We could make some MDN surveys that we've made. I won't present the results but these are available online 12:41:05 ...one was for browser, input for the interopt 2023, a more recent one was web components (prioritisation), that survey was conducted in collaboration with WebCG, the last one was web security (pain points devs may face) 12:41:14 kadirtopal has joined #webdx 12:41:39 bsmth has joined #webdx 12:41:52 ...the WebDX community group has been running these surveys to get feedback from developers. 12:42:12 ....the CG have contact points in different places to collaborate and try to exchange/promote surveys 12:42:20 marie has joined #webdx 12:42:26 present+ 12:42:38 ...that's the context of what the CG have been doing. But the question today is "how could this be useful to a standardization group" 12:42:56 s/..Agenda/tidoust: Agenda/ 12:43:34 ...Sometimes they have more formal surveys, they run under the name of someone in the group. We wonder whether there's benefit of providing a package to the grioup 12:43:35 RRSAgent: make minutes 12:43:36 I have made the request to generate https://www.w3.org/2023/09/13-webdx-minutes.html zcorpan 12:44:18 ... saying "ok you have this question on API design, this is how we would set it up so you can run a survey and get the feedback you need" 12:44:26 ...So that's it for now. Could someone here report back on if its been useful? 12:44:27 q 12:44:35 q+ 12:44:48 ack kadirtopal 12:45:29 ...kadirtopal: In addition to the questions you had, does anyone have a situation recently of handling research or user interviews that would have made the decision easier 12:45:54 ...where its added more colour, or more detail than a survey on twitter has anyone had that? 12:46:04 s/...kadirtopal/kadirtopal/ 12:46:45 ...??: I'm hoping to use that research to figure out what sort of documentation we need to write to help with security concepts. So that survey immediately was useful to me 12:47:38 ...Francois: The surveys we run are more quantitive in a sense, they can be more qualitive in the sense of interviews where you get more feedback (especially where you're asking a specific export). 12:47:50 ...Francois: Have you found something useful there? 12:48:24 Shares security survey on screen 12:48:33 s/...??/Florian Scholz 12:49:11 ...Is there any approach/guidelines/handholding we could do so people don't struggle with this any more. 12:49:45 ...Francois: Are you looking for a quick guide that says "this is what you should take into account? these are the channels you should use"? Is this a first thing you would be looking for?? 12:50:36 ???: Even just support with creating good surveys that are methodologically sound would be helpful. 12:50:38 ...Jon Kuperman - Not W3C related, event at a meta level learning how to craft good survey questions, we're worried if we put out a tweet how do we phrase it so its fair. I feel like there's a whole art to it so we understand and do it effectively 12:51:03 ...Francois: It could be a group that prepare a survey and say "well, you're introducing bias here etc", ok that's good feedback 12:51:09 q+ 12:51:15 ack kadirtopal 12:51:45 ... Im curious. Have you in TC39 had to deal with surveys where "yes its technically survey but we don't want to use it etc" 12:52:35 ...We've had Github issues where they've been linked to from a social network. But the bias can be where it came from and who linked it originally etc 12:53:01 scribe+ cpn 12:53:11 JaseW: Some questions can be loaded where you already dictate the answer in the question itself 12:53:39 ... People who follow a particular person may be more skewed towards voting in a certain way. 12:54:57 ...Francois [When working on Web Components] We were a bit worried we might get responses that didn't make any sense and there's no way to filter it out. 12:55:09 ...You could sort somethings out to some extent 12:55:29 ...Any other Qs? otherwise we could stop there and go on to the next session 12:55:55 patrickbrosset has joined #webdx 12:56:02 ...What were you hoping to get from this session? 12:57:05 ...Kadirtopal: I was interested in figuring out, where do most people have issues? Bringing in research that is largly "unsound" can be more damaging. Its a bit difficult to get hold of UX research, so i was curious, where's the biggest painpoints. 12:58:02 ...Francois Are there IDs for channels, where surveys can be run? Thank you for that channel that reaches developers in broad succession. Any other channels or venues? I'm think W3C that may publish a survey and may not attract people in the end and miss our intended audience 12:59:13 q+ 12:59:18 ...Things like logs, publishing platforms but then you might be falling into the same trap (as twitter) because it is the same audience, it's hard. MDN is great but I can't really think of anything else 13:00:33 ...I think one way for the problems you have, could be helpful if you create your own panel of responses. Have poeple opt in to be contacted and have them describe themselves, then when we do contact them we provide an incentive 13:00:49 ...Its a special population who are paid quite well 13:00:58 ...Francois: what's the incentive?? 13:01:03 ...Usually it's money 13:01:14 ...Francois: that's the one incentive we cannot provide 13:01:19 christos-bacharakis has joined #webdx 13:02:09 ...Doesn't that introduce a lot of biases? 13:02:30 ...yes, at least you have a chance to understand more about who you're getting, how good is that? Maybe better than sending something out on twitter 13:02:34 ...Chris? 13:02:39 ack cpn 13:03:09 jkup_ has joined #webdx 13:03:45 q+ 13:03:52 ...1. I'm CoChair of the media WG. We're used to APIs that have a very particular audience. There are places devs congregate, if we can identify particular communities where the technology is likely to be used then target those communities with a survey that could be a good way. It's not something we've done in the WG initially but that could be a good way, for our particular group that's something that could work 13:04:38 q+ 13:04:58 ...Chris Needham (cont) I get the point about social media, but I wonder about stack overflow, people who are building websites they might just go there to ask questions, it makes me wonder if that's a place to run these kind of surveys to get a broader range of people 13:05:47 ...I don't personally have a good relationship with SO management right now, they went on strike months ago, the rest of the moderators were involved with SO but I was not. If we have something we need from them then I'm happy to talk to them 13:06:09 ....?? (cont) If we have something concrete to take to them then I can do that 13:06:12 ack kadirtopal 13:06:44 ...?? Can you tell me more about why? (to Chris) 13:07:35 ... A lot of people coming into the WG are through relationships, occasionally because the technologies are quite niche Stack Overflow are not the best place to ask a Q so people will come to our repo instead. We get a certain amount through github issues in the spec 13:08:23 q+ 13:08:39 ... Chris N (cont): I guess we haven't run surveys because...It hasn't been something we've needed to use it for, well the case most recently we were making a prioritisation decision where we're trying to set a scope for what the Next API feature should be, I think that would be a valuable thing to do a survey. 13:08:41 ack jgraham_ 13:09:34 q+ 13:09:52 ....James: You're much better of focusing on a small set of people than going through ?? and taking a subset. 13:09:53 ack tidoust 13:11:04 ...I'm wondering if we're not trying to create something too complex perhaps? And what's needed is something some groups already do? (Audio WG invite some guest to their meetings). The GPU also start face to face and invite developers into the meetings to try out some things too 13:11:31 ... Francois: I think its still valuable to explore more formal research things 13:11:33 ack MichaelWilson 13:12:25 ... Michael: I do think we're lucky we have developers participating in the working group, one thing im concerned about is our sample size is small, so one of the reasons I came here is to see if we can increase the sample size. 13:12:35 ...Francois: Good input 13:12:36 q+ 13:12:50 ack kadirtopal 13:13:30 ...Robert: I agree with what James said. The design of web rpc might be the reason WHY people aren't using it. So if you don't know the target group today, you don't always know what the target group will be in the future 13:15:38 ...James: Technologies are specialised, if you ask a developer what do they think of our low level graphics APIs, their opinion isn't really relevant in that respect. They're not the target audience. With more context you would get better feedback, but it does feel like there are some bits of the web stack that are not intended for general developers. The low level stuff is really for people who are implementing those libraries 13:15:57 q+ 13:16:13 ack fscholz 13:17:56 ...Francois: We've been discussing re-running surveys too. the security survey. Its easy to re-run on the periodic basis. It's a much lager undertaking for the larger surveys it requires a lot of investment 13:17:56 q+ 13:19:32 ack kadirtopal 13:19:39 ...There is high level sentiment, there is a need for edge level data. Interopability is a big issue so the response was "lets address that" and we were like "what do we do"? We have the satisfaction survey now, which google is contributing. But then you need to look at what actual solutions need to be prioritised in this space 13:20:05 ...State Of HTML is the first time we're doin it, we just want to know if the sentiment is going up or down 13:21:33 ...Francois: I was thinking of points in times they have an API design question, i guess once you've solved that API design, do you run another survey to see if that design actually works. but it's going to be a different survey probably with different options 13:23:55 ...Francois Trying to summarise what I've been hearing: im hearing different levels that can be useful (developers is one level but it's not enough). Which means surveys that can reach a broader range of developers can be a good tool to have in your toolstack. Which means you need this evolution over time. Survey experts can help giving a "101" on main biases for each and every survey. In terms of where to run the survey, MDN seems to remain the mai[CUT] 13:24:04 ..le but we've heard about Stack Overflow. 13:24:48 ...??: In regards to MDN surveys. With Poll style surveys to help people make decisions, this isn't a developer satisfaction its more "the working group don't know what to work on next, you can vote" 13:25:52 ...Francois: We'll try to organize our thoughts and come up with some kind of proposal or WG chats. 13:25:59 [participants can also require a chair election] 13:26:23 ...Chris N: Even if a WG saw a need to run a survey, is that something you're suggesting the CG can help with? Would they help us form the questions? 13:26:30 q+ 13:26:46 ...Francois: the Web Components is a good example, so yes getting in touch with the Web CG is a good starting point 13:26:55 q- 13:27:25 ...Francois: Don't expect we're going to write the Qs down, we don't have those particular expertise. But with Web Components we provided feedback 13:27:54 MichaelWilson has left #webdx 13:28:08 ddbeck has left #webdx 13:28:12 RRSagent, draft minutes 13:28:13 I have made the request to generate https://www.w3.org/2023/09/13-webdx-minutes.html tidoust 13:57:47 tidoust has joined #webdx 13:59:22 duga has joined #webdx 15:06:57 tidoust has joined #webdx 16:19:58 duga has joined #webdx 16:24:42 RRSAgent, bye 16:24:42 I see no action items