This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.
In Tokenization is subject to the following constraint replace: 1. Attribute values are not tokenized. with 1. Attribute values, PIs and comments are not tokenized unless explicitly provided in the query.
The whole constraint will be dropped as decided in meeting 131, since it is wrong. Of course, attribute values, PIs and comments need to be tokenized, when queries on those node types are going to be supported, which is the normal case. Moreover, when those nodes are tokenized, e.g. during indexing documents, we typically do not know what queries we have to answer. Therefore, we should simply drop this statement about nodes not being tokenized.
Because you participated in the TF when this bug was resolved, we presume that your concerns are addressed appropriately. We are therefore marking this bug as CLOSED.