That last time this sort of thing came up, Hixie justified the inconsistency as being intended for two different contexts.

That is, authors should create cues with whitespace around '-->' but the parser should still accept cues which have none.

What Ralph said. The rules for authors and the rules for parsers are intentionally not 1:1. If you're writing a parser, don't read the section with requirements for authors.

The thing is, you're defining the grammar of the file in the parsing algorithm -- Anything that the parser accepts is valid. It's redundant and inconsistent to have a separate definition for a "valid" file which does not match the parsing algorithm.

The inconsistency between the two has already led to erroneous implementations, and that pattern is likely to continue. The rules are much more clearly defined in the "authoring section" than they are in the "parsing section", and are much more absolute (language like "One or more space characters" vs "skip whitespace").

I'm sure people have good intensions for having two inconsistent and conflicting definitions of the format, but I don't actually see how this is helpful for getting consistent implementations. It just does not seem to be helping at all.

Additionally, if the parser is supposed to accept cue-times regardless of whether or not there is whitespace surrounding the "-->" separator, then why exactly are authors being told to surround the separator with whitespace? This is creating two different sets of rules, and the more lenient one is the one that actually matters.

It doesn't really matter what an author outputs as long as the parser accepts it, therefore the authoring rules (which are different from the parsing algorithm) are really just suggestions for readability, and have no real importance.

I think we've been having this argument (in a variety of forums) for nearly a year now, and there has still been no proper explanation as to what is useful about having two definitions when only one of them has any importance, and that one is actually defined quite vaguely.

I agree that it is currently very confusing to have two sets of rules especially because people don't read the spec as a whole but read the parts they need when they need it, and an author might read only the parsing part. 

I understand the purpose of the way the spec is written: telling authors to write content only compliant to a small part of what a parser can parse, so that future extensions of the language can be defined more easily and old parser can parse them.

I think it would be very valuable to have a clear explanation of this matter in Section 4 and cross-references from 4.1 to 4.2 and from 4.2 to 4.1.

The authoring rules are also checked by validators, e.g. http://quuz.org/webvtt/ and spaces are not valid. Browsers should not be used to check if files are valid - that doesn't work for HTML either.

Having said that, I am all for making the differences small, in particular where it leads to smaller files. What do people think about having the space be optional in this particular case?

Also, I support the idea to  add a clear statement at the beginning that authors need to follow the Syntax spec and not the Parser spec. Maybe we can even run Anne's validator at the W3C and link to that.

 http://quuz.org/webvtt/ says it is a "validator" but there is no such thing defined in the spec. Is a validator something that checks the parsing rules or the syntax rules? "valid" or "validate" is used only 4 times in the spec. Maybe clearly defining what it means for WebVTT would help too.

It doesn't make sense to have two sets of rules for the same language, especially when those rules contradict each other. It's kind of nonsense and is more likely to lead to quirks-mode stuff than any positive impact on the ability to extend the language at a later date. I don't see most human authors checking their documents against lint/validators to make sure they match both definitions of the language. It would be nice if people did that, sure, but let's be realistic -- People generally don't do that. They see if it works, and if it does, they're happy. If the language really wants to force spaces around the separator, then it should reject things that don't surround the separator with spaces. It should not have "optional laws" that may or may not be followed. Optional laws just encourage criminal behaviour :)

Cyril: http://dev.w3.org/html5/webvtt/#conformance explains what "conformance" means by pointing to the HTML spec (in particular to http://www.whatwg.org/specs/web-apps/current-work/#conformance-requirements-for-authors). There is definitely a need to update that section.

Caitlin: I don't think you want to follow a moral absolutism principle in real life either ;-) http://en.wikipedia.org/wiki/Moral_absolutism

I think when it comes to mathematical structures like computer languages, absolutism is a bit more acceptable, even if the absolute "right" allows for a lot of freedom -- It still needs to be absolutely correct, otherwise you don't have a single language, you actually have N languages where N > 1.

(In reply to comment #3)
> The inconsistency between the two has already led to erroneous
> implementations, and that pattern is likely to continue.

Well, that's bad, but it indicates that implementors are reading the syntax section rather than the parsing section, which is a mistake.

> The rules are much
> more clearly defined in the "authoring section" than they are in the
> "parsing section", and are much more absolute (language like "One or more
> space characters" vs "skip whitespace").

"skip whitespace" is unambiguously defined, like the rest of the parsing section, AFAICT.

> I'm sure people have good intensions for having two inconsistent and
> conflicting definitions of the format, but I don't actually see how this is
> helpful for getting consistent implementations. It just does not seem to be
> helping at all.

The rules in the parsing section are intended to get consistent implementations.

Would it be helpful if the parsing section said that certain cases are parse errors (without changing the behavior for UAs that don't care about parse errors)? The HTML parser does that, and it's probably helpful for writing a validator.

(In reply to comment #9)
> Cyril: http://dev.w3.org/html5/webvtt/#conformance explains what
> "conformance" means by pointing to the HTML spec (in particular to
> http://www.whatwg.org/specs/web-apps/current-work/#conformance-requirements-
> for-authors). 
Sorry but it does not 'explain' anything. All I see is a red text which points to the whole HTML5 spec, not to the pointer you're giving here.

> There is definitely a need to update that section.
Yes, in particular defining classes of conformance products. This should then enable rewriting the spec (and in particular section 4.1 and 4.2) using clear conformance statements with the right subjects: "WebVTT generator shall ..." or "WebVTT UA shall ...".

Implementors read the entire draft, and see that the syntax specification is in many cases defined much more clearly than the parsing algorithm, and live in the real world where authors create write a document in a certain grammar, and user agents can interpret that that very same grammar. It's easy to see why portions of the syntax spec (which contradict the parsing algorithm) find their way into implementations. The thing is, how do you solve this problem? Clearly nobody is interested in unifying the two languages, so the best we can hope for is moving some text around to try and highlight the fact that they mean two different things, without actually correcting the contradictions. I mean personally I don't think that's a very good solution, but I'm not the one editing the draft.

I think we should get off of the argument about these two sections of the draft though, and move onto the bug issues themselves, though.

1) "Skip Whitespace is unambiguously defined" -- In an entirely different document, which is not even linked to in the WebVTT draft.

2) Allowing "00:00.500position:10%" is weird, it's not clear why this would be an allowed phrase. I think that it would be good to be explicitly clear that this is in fact the intended behaviour of the parsing algorithm, or else modify it so that a more reasonable behaviour is used instead. And of course, having the syntax spec and parsing algorithm not contradict each other here would be a big plus.

3) Not allowing "00:00.500-->00:01.500" is a bit weird too. I admit that it seems weird to not allow #2, but to allow #3. However, I sort of see "-->" as an operator token, rather than an identifier or constant like the cue-setting keywords or timestamps. That's just the way my head works, so milage may vary. I don't think there's a problem with requiring whitespace between the two, but I do think that the contradiction between the syntax and parsing algorithm should be removed. If it is meant to be lenient, it should be lenient in both sections, in the same way. And if it's meant to be strict, then it should be strict in the same way in each section. Is there any good reason why it shouldn't just be lenient? As I've said before, it's less code to implement a lenient version which just skips whitespace, rather than testing for whitespace and discarding the cue if it's not found. If lenient makes more sense, then what is the harm in specifying this leniency in the "syntax" section?

I mean, I know the response is just going to be "well you should just ignore the syntax spec and follow the rules over here", the question is, why are the rules different? If the answer is "because it lets us change the language later on" well, I think there are probably other ways to ensure that you can extend the language in a backwards compatible manner, without introducing this kind of confusion. That's all we want, is to avoid confusion so that it's easier for each implementation to be more or less on the same page.

(In reply to comment #13)
> If
> lenient makes more sense, then what is the harm in specifying this leniency
> in the "syntax" section?

In this particular case, I would guess the authoring requirement is purely to get more readable cues. I admit that it's not a strong argument and I'm not against changing the syntax to allow omitting whitespace around the arrow.

(Case study: the HTML parser parses "<!doctypehtml>" and "<!doctype html>" exactly the same, but the former is not valid (for authors) purely because it looks ugly.)

(In reply to comment #13)
> 1) "Skip Whitespace is unambiguously defined" -- In an entirely different
> document, which is not even linked to in the WebVTT draft.

It's referenced. If you implement a spec without following the references, you're gonna have a bad time. :-)

That said, a direct link would be quite helpful. Silvia, I don't know if respec supports it, but Anolis supports cross-spec xref which would give a direct link to HTML's definition using markup like: <span data-anolis-spec=html>skip whitespace</span>

>(Case study: the HTML parser parses "<!doctypehtml>" and "<!doctype html>" exactly the same, but the former is not valid (for authors) purely because it looks ugly.)

Is there any reason for it to be supported? Is there any guarantee that every user agent does actually support it? (Not limited to mainstream browsers).

It's slightly safer for HTML because it's a lot bigger and a concern to more people, so there are more eyes carefully looking over the entire draft to ensure proper behaviour. But just because it's slightly safer does not make it impervious, it's actually not a very intelligent way to write a grammar, because again, you wind up with two separate languages rather than just one, and people will always be confused about which one they need to speak and listen to.

If you take a French Canadian and send her to Paris, she would encounter people who speak French, but it would not be the same language that she speaks natively. In some cases, her interactions would go over well because there is a certain union of understanding in some cases. But in many cases, communication between her and Parisian locals might be completely misunderstood or misinterpreted, or just ignored.

In computer languages, there's slightly more of a power to establish a "One True Language" (if you can get each application and user to speak it the same way). But this isn't going to happen if you define two separate languages and stick them together claiming they're equal, they're not.

(In reply to comment #3)
> The thing is, you're defining the grammar of the file in the parsing
> algorithm -- Anything that the parser accepts is valid. It's redundant and
> inconsistent to have a separate definition for a "valid" file which does not
> match the parsing algorithm.

It's not.  The file format tells authors how they should write files.  That doesn't mean that files will only be accepted if they precisely match that format.

This concept is used across the platform.  For example, there are a massive number of things that can make an HTML document nonconforming, but which will still be accepted by parsers in a precisely defined way.

If you want to discuss this, I recommend asking about the concept on the whatwg mailing list instead of this bug.  (I recommend the whatwg list, not the WebVTT list, since this is a concept that spans many specs and isn't specific to WebVTT.)

(In reply to comment #13)
> Implementors read the entire draft, and see that the syntax specification is
> in many cases defined much more clearly than the parsing algorithm,

If you can find any case where the syntax is more precise than the parser, please file a bug.

> The inconsistency between the two has already led to erroneous implementations, 

If this ever happens, then the implementor is *catastrophically* misunderstanding how to read web specs.  The only way to write an interoperable implementation is to use the parser and ignore the syntax.  If you write an implementation of a web format based on its syntax instead of its parser, you are *guaranteed* to create something which is broken and incorrect.

When a communication is misunderstood, it is always the responsibility of the speaker, not the listener. That said, it is the fault of the draft when it fails to communicate effectively with implementors, not the other way around. This isn't lawerese, the goal here is for the message to be understood consistently. And the goal is not met, the way this is all written. If you communicate two messages in one, of course they will be confused, as has happened repeatedly already. It might be a widespread convention for the web platform, but if it is, then that is an error. It is not effective communication.

Im not going to convince people of this mistake, and it wont change. But it is absolutely not effective, so we should not be using this same bad style in the webvtt draft.

Just to be clear, the proof that it is poorly communicated, is the fact that it is not understood.

If it is not understood, write it better. Communicate effectively.

> Im not going to convince people of this mistake, and it wont change. But it is absolutely not effective, so we should not be using this same bad style in the webvtt draft.

You're not going to convince people because your arguments are underinformed and not compelling.  That doesn't mean there aren't improvements that can be made (we've discussed putting a "implementors! you're in the wrong place!" note on the syntax section in the past), but most of this bug seems to be "spec writing 101", explaining basic principles of web specs.  That's a fine discussion to have, but this isn't a great place to have it.

Exactly aligning the syntax and parser so that people can implement the format based on the syntax misses the important distinction between the two.  Experience has shown that critical aspects of format specification are handled badly by syntax definitions, such as error handling and forwards-compatibility.  No changes to the syntax section will make it appropriate for implementors to use instead of the parser.

It is informed, the information is here: this style of communicating rules for the WebVTT language has historically caused many problems. WebKit/Blink developers are not stupid people, their brains are quite capable of understanding a message.

However the message communicated by the draft (and likely other drafts as well) is communicated poorly. It is not at all surprising that you don't see the rules of one section being followed perfectly, because there is a second set of rules being communicated in the same document, and this second set of rules contradicts the other one.

You can't seriously expect people to disregard a huge section of the draft just because you say that Authors are supposed to follow different rules (WHY!? This does not make sense!). People don't work that way, and the evidence is in the various implementations which stumble over this mangling.

Clearly, this style of defining the language does not work, and can not work. And there is no practical, good reason for defining it in this way.

The reason I'm not going to convince people to do things in a more uniform way, is because for whatever reason you lot like to ignore the fact that this style just doesn't work in practice. It's like you all prefer to just ignore the way humans actually behave, and prefer to say that misunderstanding is the fault of the reader and not the fault of the communicator. There's no changing your mind about this if you're going to ignore evidence, so there's no point trying.

However, for this particular small part of the web, it is worth advocating for a more clearly defined language, so that it can actually be expected to be implemented consistently, and get some uniform behaviour between different user agents.

This is simply not going to happen if the spec is a jumbled mess of different sets of contradictory rules, no matter how many <h1>THIS IS NOT RELEVANT TO PARSER IMPLEMENTORS</h1> you add. Like, it just isn't going to work that way.

Have a set of rules for parsing, develop a parsing algorithm which follows those rules, treat documents which do not conform to these rules as erroneous. Tell authors to write documents which conform to the rules. Vagueness and contradictory sets of rules are not going to fly in practice, people do not operate that way.

(Sorry, but I don't have time or desire to read a wall of text, based on the SNR of this thread so far.  Others might continue as they like, of course.)

Of course, nobody ever likes to read that they're wrong about something and be shown evidence of it.

There is no use repeating arguments. We all agree that there need to be changes made to the spec. Let's be constructive about it.

Here's what I propose:

I'm going to follow your proposal to remove the requirement of whitespace around "-->".

I'm also going to change the spec to be more obvious about what the sections are and who needs to read and follow what.

I'm going to put some sort of flagging into the parser for syntax errors, even if the parser can deal with them.

I'm also going to make the links direct links to the specific sections in the HTML spec that are relevant - or pull out the relevant pieces so we can have a stand-alone readable WebVTT spec.

(In reply to comment #25)
> I'm also going to make the links direct links to the specific sections in
> the HTML spec that are relevant

Excellent.

> - or pull out the relevant pieces so we can
> have a stand-alone readable WebVTT spec.

Please don't do that. It's bad to have things duplicated in specs, for various reasons.

For a discussion of why there are things that are parsed consistently but are considered invalid see:

   http://whatwg.org/html#syntax-errors

Note in particular that _everything_ is parsed consistently in WebVTT (and HTML, and most other Web formats). Even line noise is parsed in a predictable way. But we define a subset of that as "valid" to help authors avoid making unintentional mistakes (things where what it looks like it'll do to a human doesn't match what it'll actually do), to help them leave us space to extend the languages in the future, etc (other reasons are given at the link above). HTH.

I would argue that this is NOT a mechanism for future proofing, BECAUSE:

1. If only a subset of validly parsed language is considered valid, you are saying that documents which do not match this subset don't count when breaking changes are introduced to the parsing of the language. What this effectively does is takes documents which previously are entirely valid and parsed by the algorithm, and breaks them. In and of itself, this is not effective future proofing.

2. If extensions to the language break the existing rules for "valid" documents, then this accomplishes absolutely nothing. If extensions to the language do not break previously valid documents, then it's a moot point because the new language is effectively backwards compatible.

Having separate mandates for "pretty" documents vs "parsable" documents, and claiming that only "pretty" documents are actually valid, simply does not make sense, and absolutely leads to demonstrable confusion.

I've been saying this repeatedly, and I know it falls on deaf ears, and I know people want to have fanciful imaginings of how humans actually operate (assuming that HTML5 authors actually read the draft, for example, is pretty far fetched -- they'll read summaries and annotated documentation which falls out of sync with the draft). Because of this, you will end up with "invalid" documents, and they will not be considered "invalid" because they will "work". They might not be pretty, but if the parser allows them, then they are as valid as anything else that it allows. This does NOT help future-proofing the language. Ultimately, the rules that the parser follows are the only rules that really matter to authors, and pretending that there is some other standard of validity that they will follow is just fanciful.

It might not be the model followed by CSS3 or other sections of HTML5 where there are a great deal more eyes on the draft, but I don't see it working here, and evidence of it not working has already been presented in two bugs.

Don't get me wrong, I'm not trying to be all sour and start a heated argument, if you find some way to demonstrate that this strategy is logically sound and actually works, maybe my perspective will change. This is more about discourse and exchange of ideas than angrily saying "you're wrong!", I hope that's understood by readers of my comments.

OK, so here I am, months later, triaging bugs.

I'm very inclined to WONTFIX this one, because the difference between syntax and parser is very much intentional and I don't see a reason to align them on this specific point.

Silvia, you mentioned some things you want to clarify, can you open new bugs for those and then close this one?

Bug 22591 is already opened for the conformance section.