– DRAFT –
Simplifying the Updatable REC Process

Marcos: who isn't familiar with the updateable rec process yet?

nigel: I'm in the early days of using it

Marcos: the assumptions that underpin the process and the experience in practice, what turned out the complexity; if going down this path, choose wisely

<fantasai> github: w3c/process#866

Marcos: more clarity on consequences and expectations of that process are likely needed
… being able to show what changes are, their status is good
… there may be alternative approaches
… with tooling, social changes, or even more radical changes
… we've had specs that tried using it but had to stop
… we managed to do it with Geolocation, but this was still a struggle
… having to write manual diffs is not very practical

fantasai: the Process require the text of the spec go through wide review, implemetnation review, AC review, etc
… the problem we want to solve is that doing this all took so long that it didn't get reflected to the Recommendation documnet (only in the editors draft)
… the reqiurement is to annotate the change as identified correction or addition
… and once they've been vetted and proved to match the Rec qualifications, they can be folded in the main Rec
… the process doesn't mandate anything about how these annotations are maintained, only that they need to be reviewable
… and the normative content of the Rec is the one that has gone through the proper review
… Marcos has identified that the generation of the diff markup is complex to manage
… There hasn't been a lot of training, not a lot of experience sharing on what may need to change to make this more workable

florian_irc: as we think through this, let's distinguish what can be done without process change
… things that need process changes, either because they're not needed, or because they're too constraining to make it workable
… we should approach process changes carefully

marcos: the Process itself is a bit hard to follow
… another aspect is that how the requirements are met: retaining Rec quality is the ultimate goal
… but his may be achiveable through other approaches (and thus Process changes)
… as one of the first adopters of the process, w3c/payment-request#996 is an example of managing the diff wasn't suitable
… we ended up choosing to go back to CR

fantasai: note that we're proposing to drop PR which may help with that process

marcos: Dom, you had something about webRTC?

Dom: WebRTC was published in 2021

dom: It was published in 2021, and we identified 48 changes to spec
… in that process we've ben experimenting with combination of tooling, process management
… we were able to generate the diff automatically from the base recommendation using some magic script from ECMA
… I think this made it possible, the WG agreed yesterday to bring 20-ish of those candidate amendments as proposed amendments
… it has also helped in some ways to structure discussion on the changes, because forced to discuss are these editorial or substantive, do we have tests etc.
… it also proved useful
… but feedback from WG was, I don't know that we would do that if you (dom) were not managing the process
… I'm concerned about that

<Zakim> dom, you wanted to relate the WebRTC experience

dom: my tooling is a bit experimental, can make some simplifications
… some of it is training and getting familiarity
… I' mthe only one in the WG that has full understanidng of post-REC process
… but still a lot of familiarization needed
… but we should ask ourself if the process is fit for purpose, given few cases of success
… but I'm a lot more optimistic today than 6 months ago, due to progress in tooling
… but we need more community support for this process
… sharing experience, building better tooling -- my tooling is a bit scrappy
… building into regular spec tooling
… integrate into GH management
… I think if we want to really claim that we have a solution to that problem space, need more attention and resources
… hopefully can experiment faster pace if we can strucutre commmunity effort

fantasai: can you tell us what you did?

dom: When you make normative change to WebRTC
… identify it as such
… for each PR that is made to the REC, there's an expectation that your document this JSON file
… what kind of amendment is it?
… give a human-level description of the change
… document the testcases that demonstrate the change was implemented
… link to the PR in WPT
… for anything not determined as editorial -- if you don't document this PR with the amendment -- it will be flagged as not ready for merging
… this allows generating the list of candidate/proposed amendments with links to relevant PRs and tests
… which will help for LC review process
… you can see that for each change it shows the annotations and the diff
… it takes the markup form the base REC, which we saved in the repo
… compares it to new markup, and generates diff markup by JS
… just need to say "this section with id X is new version of id X, or addition to the REC"

marcos: you still need the insertions and deletions

dom: They're generated from the PR

florian_irc: JSON is manually written, and it generates the diff from the pull request

dom: I think can further simplify also with conventions
… I haven't tried to get us there
… but in terms of editor's work, instead of having to deal with diff markup, you just have to deal with picking the right IDs for the changes in your document
… won't say that this is trivial -- initial version wasn't easy to use

<plh> dom++

dom: but at this point it's vastly easier than doing manual markup

Neil: Neil Soiffer, MathWG
… I'm mainly here to learn, so not completely up to date on the proposals
… but I know a number of the changes in our 25 years were simply editorail
… was this process simpler or harder, or changes don't matter if clarifications

dom: If class 1 or 2 changes, super simpler, just republish

<Zakim> florian_irc, you wanted to react to dom

Neil: we had insertions and deletions to track our changes, build up and make things harder to edit

fantasai: Yes, that's what we have now

<Zakim> nigel, you wanted to say there's a scale of usage, and marketing/information issue

nigel: My observation is there's a sense, it's been 3 years and everything should be easy
… but changing how REC publication work and bringing the community along has taken awhile and still not done

<florian_irc> [Process fir REC revision is documented here: https://www.w3.org/policies/process/#revising-rec ]

nigel: I think that's OK
… things like more mature tooling is part of the ansewr
… but wanted to reflect on... thread in Process CG
… where talking about perpetual CR as being the answer to living standards
… tried to suggest that updateable REC is better; what isn't working in updateable REC?
… and I've totally failed to get an answre
… but one point that came out was more about marketing and documentation and raising priority of the option

<sideshowbarker> I disagree that Nigel has totally failed to get an answer

nigel: in the thread I refer to an updateable REC, and response was "that's not a thing in the process"
… if you set the status correctly in ReSpec, it knows how to do it
… but not clear in the Process because no term for it
… Maybe have a definition

florian_irc: we've done part of what you asked
… as part of dropping PR phase, we did some editorial work to make it more visible
… but remember that *every* REC allows for changes
… it's about opting into changes that allow new features -- that is not allowed by default

<Zakim> florian_irc, you wanted to react to nigel

plh: Thank Dom for making progress on tooling

qa+ florian_irc

plh: today WGs are avoiding that process
… webAudio isn't going down that road, e.g.
… as fantasai alluded, there's what Process says and how we implement it, and these are two different things
… sometimes we just need to update the Guidebook
… I don't think we have any documentation in Guide about how to update RECs
… idk if the solution proposed by Dom woudl work on Marcos' PR
… but we need something simple, because it's hard to find editors at all
… problem shouldn't be editing mechanics, but content of the spec

<Zakim> florian_irc, you wanted to react to plh

florian_irc: in addition to being familiar with Process, one reason I didn't have much trouble was that those RECs didn't need many changes
… for MQ only 2 changes, for css-contain it was 3, and it doesn't require a script
… maybe a bit annoying but not that bad
… so that's part of why I didn't suffer
… another one is that those changes were non-intrusive
… I did hvae one that reworked a list, it would have been complicated to markup in detail
… but instead I said "this seciton replaced by this other one"
… so didn't need to diff it, just indicate the replacement
… If you have many such things piling up, it becomes challenging
… I'm curious what kinds of situations we get into things that are legitimately REC, but have a very large amount of intrusive changes
… if you hvae a continuing stream of overlapping changes, it's hard
… but why is that a REC?
… obviously might happen to add new features, but other than that how do we get there?

Marcos: when HTlm changed from browsing context to navigable
… or IDL from void to undefined
… when a dependency changes, then it's a problem
… ReSpec can't handle inline changes in the IDL because it's doing validation

florian_irc: you'd have to do at section level

Marcos: would have to invalidate one of them

Marcos: but working around tooling in that situation, kinda sad
… if you delete a section, IDL is still there and IDs generated, so tooling is not designed to handle
… with duplication

Marcos: from your tooling if you have a dfn and it got moved or rewritten?

dom: Wrt how do you get such changes, webRTC got to first REC after decade of effort
… but you get a lot of corrections from implementations
… a lot of tooling was about deduplicating IDs
… again main improvements could be made, this was just making it workable for my document

dom: one approach we take is virtually think of, we're making the editorial change of calling this series of steps with name

Thread referenced by Nigel and sideshowbarker

dom: and then diff separately
… but feels a bit hackish
… how much we can improve depends on fundamental requirement that spec needs to surface the changes such that they can be reviewed
… that may be where looking for alternatives might be interesting
… text on /TR right now needs to be clear what's normative or not

fantasai: One of my goals was that people referencing the spec -- everyone who isn't in the process of eidting the spec -- could look at /TR and not need to dig through editor's drafts or a pile of unmerged PRs

gkellogg: We have multiple specs, and need to foster updates for them that aren't in charter atm
… but WG should get that once it finishes REC process
… we're marking the specs as being updatable, but most ppl working on specs are more cargo-cultish

<plh> (we don't have tutorials for this because of lack of tooling for updatable-rec)

gkellogg: if something is broken in PR helpful
… fuzzy on some of the mechanism works here

<nigel> +1 to training/documentation being helpful, obviously

gkellogg: so config file that references PRs?
… presumably could figure out by looking at PR
… PR is not merged? is the result of an update done by an unmerged PR which magically updates? or expected to be merged into ED?

dom: what I was presenting was just my tooling, it's not official
… it's a random JSON file in the WebRTC repo. Maybe someday becomes more than that.
… And yes, expects fully merged PR
… principle is that the ED should be easy to read
… it's only when you generate for REC publication that the diffs are surfaced with the tool
… but there's no tutorial because right now because the only user of this system is me
… need help from spec authoring tools to make it more usable, because right now it's just for me in my WG
… not production-ready

florian_irc: there's a little bit of how to mark up the changes manually, and that's not a tutorial and not the ideal state of the world

gkellogg: nightmare stories about trying to do this

Marcos: [shows the markup for ins/del]

gkellogg: that's the manual well, so in the hypothetical future something generates that

Marcos: Florian, to your questiona bout where do you get such massive changes
… it turned out as we update this algorithm to correct it, it made more sense for us to move a bunch of stuff
… rewrite all these things that were inlined into the algo to have definitions
… so part of it was editorial, and some of it was normative, so we ended up with 240-line diff

Marcos: when there are changes to the process, would be good if as those changes happen, we could beta-test them
… test-drive the thing
… e.g. Tab was also surprised when all this stuff came up
… and frantic rush to incorporate all this stuff
… so would be nice to have a better way of coordinating all that

<florian_irc> +1 to better coordination

marcos: wrt marketing side, talkd about having a specific designation
… we have REC which is gold star; Candidate Recommendation is a bit sad
… maybe it can have a better name
… but if reality is going to be specs in perpetual CR that's sad
… because e.g. WHATWG have their final state stay final

marcos: can't we deputize people to take care of horizontal review on an ongoing basis, so that they get all the vetting before even landing the PR?
… then we can have a perpetual REC

<florian_irc> fantasai: but then your REC is not useful because people need to work off a pile of PRs

marcos: it works for me
… I just have 5-6 people to verify my PRs

<Zakim> sideshowbarker, you wanted to comment

sideshowbarker: One thing, over last few months, started talking to a number of chairs, contributors, individuals
… was working with Web Assembly about transition to CR
… and I didn't know the answers to their quesitons
… I looked in Process doc, but was confused, never looked at Process before
… Many people expected me to understnad Process but I didn't
… These are ppl who spoke with me candidly, they're at odds with their own AC rep about what they should do
… not on AC Forum or anything either
… So I wanted to explain the Process to them, so had to figure it out myself.
… as part of this, I wanted to ask them what do you want? what's the value of a WG compared to CG?
… some people said, we just want a W3C URL for our spec. They should care about, but what they were saying was they dind't care about RF commitments.
… that surprised me, because I imagine their org cares even if they don't
… but some what their company wants is RF commitments
… and then they want to minimize process, because they want to do the leas tamoutn of work
… quickly and easily

sideshowbarker: other things they complained about is wide review
… they think it's time-consuming make-work, inefficient
… wrt updateable RECs was, it's a non-problem for them
… because all they want is RF commitments, and they can get those in CR, then it's hard for me to even articulate to them what they get out of going to REC
… I said we had wide review, it's a vlaue; but some people don't value that
… so we need to help socialize what's the value of that
… we have a11y and i18n experts, they'll help you get the work done better -- but not articulating that well
… part of response was also they don't see the value of AC Review
… if required to go to REC, or required to get their spec published
… they don't value that, it's an obstacle rather than adding value
… what they want to do is auto-publish and maintain their spec
… i.e. CR Drafts
… and then some interval get a snapshot for RF commitments
… so that's the background I'm getting

sideshowbarker: The naming. I go tinto that issue, and I almost regret, because not a positive experience
… Some weighed into that conversation, but got pushback
… They are confused by the fact that if we have this thing, where they can auto-publish and maintain their spec and get RF commitments
… doens't give the right labelling to their target audience
… Andreas suggested a new name, but we have this issue of naming with Candidate Rec. They don't like either Candidate or Recommendation.

florian_irc: Lot in your feedback, Mike, so won't address all but high-level reaction
… the things they claim they don't care about, they actually care about
… the reason why W3C stamp is valuable is because of those things
… so we should make as painless as we can, but the fact that they don't see the value doesn't mean they don't care about

florian_irc: wrt let's review the PRs and then land them, it's ok if it's one or two, itt's fine
… but it's hard to see the state of the spec if there's a lot
… in particular, when some of these PRs are longlived. If 1-7 weeks it's fine
… but if some PR is open for 3 years becaue waiting for implementations, that's not workable
… alternative thing where we have a half-baked REC with the changes, we started with that
… we had CSS2 editors draft, and we had many changes, but some were ready and others were not
… we did that in a document where not distinguished
… and we had no point that we had all the things ready at once to publish REC until all done
… coudl have held back changes, but then they wouldn't be in the spec for people to see

florian_irc: when doing usual cycle of WD -> CR -> PR -> REC
… you have criteria you meet in a certainorder
… but currently the amendments are the same requirements but in a different order
… that's confusing. We should address
… I think removing Proposed REC will help with that

<astearns> wonders if merging PRs but being able to divine the state of each change from the PR itself (instead of markup in the spec) could be workable

florian_irc: I haven't figured it out yet, but maybe it cna help

fantasai:
… everyone, including those not in the WG should be able to look at the TR doc to get the current state of the WG consensus and what they should be implementing

florian_irc: If an implementer looks at the spec, and implements it, an then is told you implemented the wrong thing, that's not OK

https://lists.w3.org/Archives/Public/spec-prod/

Meeting closed.