This section describes the
full-text selections which contain the full-text
operators in a full-text contains
expression
(FTContainsExpr), as
well as the match options which modify the matching semantics of the
full-text selections. In the following, the syntax for each type of
full-text selection is given together with an informal statement of
its meaning.
As shown in the grammar, a full-text selection consists of search
conditions possibly involving logical operators (FTOr), followed by an
arbitrary number of positional filters (FTPosFilter).
The syntax and semantics of the individual full-text selection
operators follow.
This XML document
is the source document for examples in this section.
The first five tokens in this example using the sample tokenization would be "Improving", "the", "usability", "of", and "a".
Unless stated otherwise, the results
assume a case-insensitive match.
3.2 Search Tokens and Phrases
FTWords finds matches that contain the specified
tokens and phrases.
FTWords consists of two parts: a mandatory
FTWordsValue part and an optional
FTAnyallOption part. FTWordsValue specifies the tokens and phrases
that must be contained in the matches. FTAnyallOption specifies how
containment is checked.
In general, the tokens and phrases in
FTWordsValue are specified using a nested XQuery expression.
To simplify notation, the enclosing braces may be omitted if FTWordsValue consists of a single string literal.
The following rules specify how an FTWordsValue
matches tokens and phrases. First, the
FTWordsValue is converted to a sequence of
strings as though it were an argument to a function with the expected
type of xs:string*.
If the sequence is empty, the FTWords yields no matches.
Otherwise, each of those strings is tokenized into a
sequence of tokens as
described in Section 4.1 Tokenization.
Then, FTAnyallOption is checked.
If FTAnyallOption is "any", the sequence of tokens for each string is
considered as a phrase.
If the sequence of tokens is empty,
then the phrase contributes nothing to the set of matches for the FTWords.
Otherwise, a match is found in the tokenized form of
the text being searched, whenever that form contains a subsequence of tokens
that corresponds to the sequence of query tokens in an implementation-defined
way and that subsequence of tokens covers consecutive token positions in
the tokenized text. If the value of the FTWordsValue contains more
than one string,
the different strings are considered to be alternatives, i.e., the search context
must contain at least one of the generated phrases.
Each resulting match will contain exactly one such phrase.
If FTAnyallOption is "all", the sequence of tokens for each string is
considered as a phrase.
If any such sequence of tokens is empty, the FTWords yields no matches.
The resulting matches must contain all of the
generated phrases.
If FTAnyallOption is "phrase", the tokens from all the strings are
concatenated in a single sequence, which is considered as a phrase.
If the sequence of tokens is empty, the FTWords yields no matches.
The
resulting matches must contain the generated phrase.
If FTAnyallOption is "any word", the tokens from all the strings are
combined into a single set.
If the set is empty, the FTWords yields no matches.
The search context must contain at least
one of the tokens in the set.
Each resulting match will contain exactly one such token.
If FTAnyallOption is "all words", the tokens from all the strings are
combined into a single set.
If the set is empty, the FTWords yields no matches.
The resulting matches must contain all
of the tokens in the set.
If the FTWordsValue evaluates to
a single string, the use of "any", "all", and "phrase" in
FTAnyallOption produces the same
results.
If FTAnyallOption is omitted, "any" is
the default.
The following expression returns the sample book element,
because its title
element contains the token "Expert":
//book[./title contains text "Expert"]
The following expression returns the sample book element,
because its title
element contains the phrase "Expert Reviews":
//book[./title contains text "Expert Reviews"]
The following expression returns the sample book element,
because its title
element contains the two tokens "Expert" and "Reviews":
//book[./title contains text {"Expert", "Reviews"} all]The following expression returns false for our sample document, because
the p element doesn't
contain the phrase "Web Site Usability" although it contains all of the tokens
in the phrase:
//book//p contains text "Web Site Usability"
The following expression returns book numbers of book elements by
"Marigold" with a title about "Web Site Usability", sorting them in descending
score order:
for $book in /books/book[.//author contains text "Marigold"]
let score $score := $book/title/@shortTitle contains text "Web Site Usability"
where $score > 0.8
order by $score descending
return $book/@number
3.4 Match Options
Full-text match options modify the matching behaviour of
the primary full-text selection to which
they are applied.
[Definition:
Match options modify the set of tokens
in the query, or how they are matched against tokens in the
text.]
[Definition:
Each of the alternatives of production
FTMatchOption
other than FTExtensionOption
corresponds to one match option group. ]
The match options from any given group are mutually exclusive, i.e.,
only one of these settings can be in effect, whereas match options of
different groups can be combined freely.
It is a static error
[err:FTST0019]
if, within a single FTMatchOptions,
there is more than one match option of any given
match option group.
For example, if the FTCaseOption "lowercase"
is specified, then "uppercase" cannot also be specified as part of the same
FTMatchOptions.
Although match options only take effect in the application of
FTWords, the syntax also allows to specify
match options that modify the non-primitive full-text selection
"(" FTSelection ")". Such a higher-level match option
provides a default for the respective match option group for any
embedded FTPrimary, just as
match option declarations
in the Prolog
provide default match options for the whole query.
Match options are propagated through the query via the static context.
For each of the seven match option groups,
the static context has a component
that contains one option from that group.
The seven settings are initialized by the implementation
in accordance with the table in
Appendix C Static Context Components,
and are modified
by any FTOptionDecls
in the Prolog.
The resulting settings are then propagated unchanged
to every FTContainsExpr in the module
(including those in VarDecls and FunctionDecls,
and including any that happen to be nested within
another FTContainsExpr).
At any given FTContainsExpr,
the settings from the static context
are copied to the FTContainsExpr's inner settings,
which are then propagated down the syntax tree.
At each FTPrimaryWithOptions,
the locally specified match options (if any)
overwrite the corresponding inner setting(s).
At each FTWords,
the inner settings are used
as the effective match options
for tokenizing the query strings
and matching them against the tokens in the text.
(These inner settings could be seen
as a parallel set of components in the static context,
but Section 4 Semantics models them
as structures that get passed as parameters
to various semantic functions.)
Thus, when a match option appears in an FTSelection,
it applies to the associated FTPrimary,
but not to any FTContainsExprs
that happen to be embedded within that FTPrimary.
Instead, for a nested FTContainsExpr,
the default match options are those declared in the Prolog
or, if not declared in the Prolog,
then supplied by the implementation's initial values.
An FTMatchOption
applies to the FTPrimary that immediately precedes it.
That FTPrimary is either
an FTWords (possibly qualified by an FTTimes),
an FTExtensionSelection, or
a parenthesized FTSelection.
[Definition:
The order in which effective match options for an
FTWords are applied
is called the match option application order.]
This order is significant
because match options are not always commutative.
For example,
synonym(stem(word))
is not always the same as
stem(synonym(word)).
The match option application order is subject to some constraints:
The Language Option must be applied first
The Stemming Option must be applied before the Case Option and the
Diacritics Option
Aside from these constraints, the full order of the application of match
options is implementation-defined.
More information on
their semantics is given in 4.2.5 Match Options Semantics.
If no match options declarations are present in the prolog and the
implementation does not define any overwriting of the static context
components for the match options, the query:
/books/book/title contains text "usability"
is, assuming "de" is the implementation-defined default language,
equivalent to the query:
/books/book/title contains text "usability"
using language "de"
using no wildcards
using no thesaurus
using no stemming
using case insensitive
using diacritics insensitive
using no stop words We describe each match option group in more detail in the following
sections.
3.4.1 Language Option
[Definition: A
language option
modifies token matching by specifying the language of search tokens and
phrases.]
The StringLiteral following the keyword language
designates one language. It must be castable to xs:language; otherwise, an
error is raised: [err:XPTY0004]XP30.
The "language" option influences tokenization, stemming, and stop
words in an implementation-defined way. The "language" option MAY influence the behavior of other match options in an implementation-defined way.
The set of standardized language identifiers is defined in [BCP 47].
The set of valid language identifiers among the standardized set is implementation-defined.
An implementation MAY choose to use private extensions introduced by a
singleton 'x' for additional language identifiers, or other singletons
for registered extensions as described in sec. 2.2.6 of [BCP 47].
It is implementation-defined what additional language identifiers, if any, are valid.
If an invalid language identifier is specified, then the behavior is implementation-defined.
If the implementation chooses to raise an error in that case,
it must raise [err:FTST0009].
An implementation MUST treat
language identifiers that [BCP 47] defines as equivalent as
identifying the same language. For example "mn" and "MN" are equivalent,
as language tags are case insensitive, and
"de" and "deu" are equivalent, as they are different codes for the same language.
However, it is implementation-defined whether an implementation
treats a particular language identifier with script, region, or variant portions
as equivalent to the language identifier without them. For example, an
implementation may treat "en-UK" as equivalent "en" and "en-US" but
"sr-Latn" as different from "sr" and "sr-Cyrl".
The default language is specified in the static context.
When an XQuery and XPath Full Text processor evaluates text in a document
that is governed by an xml:lang attribute and
the portion of the full-text query doing that evaluation contains an FTLanguageOption that
specifies a different language from the language specified by the governing xml:lang attribute,
the language-related behavior of that full-text query is implementation-defined.
This is an example where
the language option is used to select the appropriate stop word list:
//book[@number="1"]/content//p contains text "salon de thé"
using stop words default using language "fr"
3.4.2 Wildcard Option
[Definition: A
wildcard option
modifies token and phrase matching by specifying whether or not
wildcards are recognized in query strings.]
When the "wildcards" option is used,
wildcard syntax may be included within query strings.
A wildcard consists of
an indicator (a period or full stop, "."),
optionally followed by a qualifier.
Each wildcard in a query token will match zero or more characters
within a token in the text being searched, as described below.
The number of characters that can be matched depends on the qualifier.
The forms of wildcard syntax specified by this document are:
A single period, without any qualifiers:
Matches a single arbitrary character.
A period immediately followed by a single question mark, "?":
Matches either no characters or one character.
A period immediately followed by a single asterisk, "*":
Matches zero or more characters.
A period immediately followed by a single plus sign, "+":
Matches one or more characters.
A period immediately followed by a sequence of characters
that matches the regular expression {[0-9]+,[0-9]+}:
Matches a number of characters, where the number is
no less than
the number represented by the series of digits before the comma,
and
no greater than
the number represented by the series of digits following the comma.
If a period in the query string
is immediately followed by a left curly brace,
but the subsequent characters
do not conform to the given regular expression,
then an error is raised:
[err:FTDY0020].
A question mark, asterisk, plus sign, or left curly brace
that is not immediately preceded by a period
is not treated as a qualifier.
For example, using the sample tokenization and "wildcards",
the query string "wil+"
does not match the search text "will" or "willlllll",
but only matches the search text "wil".
(The sample tokenization treats the plus sign as punctuation.)
When "wildcards" is used,
any character in a query string
can be "escaped" by immediately preceding it with a backslash, "\".
That is,
a backslash immediately followed by any character
represents that character literally,
preventing any special interpretation
that the "wildcards" option might otherwise attach to it.
In particular:
Escaping a period prevents its interpretation as a wildcard.
Escaping a question mark, asterisk, plus sign, or left curly brace
ensures that it is not interpreted as a qualifier.
An escaped backslash ("\\") represents a literal backslash.
If a query string is terminated by an unescaped backslash,
an error is raised:
[err:FTDY0020].
Note:
A query string of the form "abc\"xyz"
does not represent
the three characters "abc"
followed by a literal double-quote
followed by the three characters "xyz".
Instead, this is a malformed StringLiteral,
and the processor will report a syntax error
[err:XPST0003]XP30.
When the "no wildcards" option is used,
no wildcards are recognized in query strings.
Periods, question marks, asterisks, plus signs, left curly braces,
and backslashes
are always recognized as ordinary text characters.
The default is "no wildcards".
The following expression returns true, because the p element
contains "well":
//book[@number="1"]/p contains text "w.ll" using wildcards
The following expression returns true, because the title element
contains "site":
//book[@number="1"]/title contains text ".?site" using wildcards
The following expression returns true, because the title element
contains "improving":
//book[@number="1"]/title contains text "improv.*" using wildcards
The following expression raises error [err:FTDY0020],
because the query string uses incorrect syntax:
//book[@number="1"]/p contains text "wi.{5,7]" using wildcardsThe following expression returns true,
because the title contains "site":
//book[@number="1"]/title contains text "\s\i\t\e" using wildcards
The following expression returns true,
because the title contains "Usability":
//book[@number="1"]/title contains text "Usab.+\\" using wildcards
(Note that "\\" represents a literal backslash,
which the sample tokenization treats as punctuation.)
The following expression raises error [err:FTDY0020],
because the query string ends with an unescaped backslash:
//book[@number="1"]/p contains text "will\" using wildcards
The following expression returns false, because the p element
does not contain the phrase "w ll":
//book[@number="1"]/p contains text "w.ll" using no wildcards
(Note that, without wildcards, the sample tokenization
will treat the period in "w.ll" as punctuation,
thus producing "w" and "ll" as separate tokens.)
3.4.3 Thesaurus Option
[Definition: A
thesaurus option
modifies token and phrase matching by specifying whether a thesaurus is used or
not.]
If thesauri are used, the thesaurus option specifies information to locate
the thesauri either by default or through a URI
reference. It also states the relationship to be applied and how many
levels within the thesaurus to be traversed.
If the thesaurus option specifies a thesaurus with a relative URI,
that relative URI is resolved to an absolute URI using the base URI in
the static context and that absolute URI is used to identify the thesaurus.
If the URI specifies a thesaurus that is not found in the statically
known thesauri, an error is raised [err:FTST0018].
Thesauri add related tokens and phrases to the query or change query tokens.
Thus, the
user may narrow, broaden, or otherwise modify the query using
synonyms, hypernyms (more generic terms), etc. The search is performed
as though the user has specified all related query tokens and phrases
in a disjunction (FTOr).
Note:
A thesaurus may be standards-based or locally-defined. It may be a
traditional thesaurus, or a taxonomy, soundex, ontology, or topic
map. How the thesaurus is represented is implementation-dependent.
An FTThesaurusID may optionally contain a StringLiteral to specify
the relationship sought between
tokens and phrases written in the query
and terms in the thesaurus.
Relationships include, but are not limited to, the relationships
and their abbreviations presented in [ISO 2788] and
their equivalents in other languages. The set of relationships supported by an
implementation is implementation-defined, but
implementations SHOULD support the relationships
defined in [ISO 2788]. The following list of terms have the
meanings
defined in [ISO 2788]. If a query specifies thesaurus
relationships not supported by the thesaurus, or does not specify a
relationship,
the behavior is implementation-defined.
equivalence relationships (synonyms): PREFERRED TERM (USE),
NONPREFERRED USED FOR TERM (UF);
hierarchical relationships: BROADER TERM (BT),
NARROWER TERM (NT), BROADER TERM GENERIC (BTG), NARROWER TERM GENERIC (NTG),
BROADER TERM PARTITIVE (BTP), NARROWER TERM PARTITIVE (NTP),
TOP Terms (TT); and
associative relationships: RELATED TERM (RT).
An FTThesaurusID may also optionally include an FTLiteralRange to specify
the number of levels to be queried in hierarchical relationships.
An FTLiteralRange is a constrained form of FTRange,
and specifies a (possibly empty) range of integer values according to the same rules.
Note:
For historical reasons,
an implementation MAY allow an FTLiteralRange
to have subexpressions more general than IntegerLiterals,
and MAY even allow
its subexpressions to be dynamically evaluated.
The effect of specifying a particular range of levels in an FTThesaurusID
is implementation-defined.
This includes cases involving empty ranges, negative levels, or
levels not supported by the thesaurus.
If no levels are specified,
the default is to query all levels in hierarchical relationships
or to query an implementation-defined number of levels
in hierarchical relationships.
The "thesaurus" option specifies that string matches include
tokens that can be found in one of the specified thesauri.
When "default" is used in place of a FTThesaurusID, the thesauri
specified in the static context are used, which are either given by the
prolog declaration for the thesaurus option, or, if no such
declaration exists a system-defined default thesaurus with a
system-defined relationship. The
default thesaurus may be used in combination with other explicitly
specified thesauri.
The "no thesaurus" option specifies that no thesaurus will be
used.
The default is "no thesaurus".
The following expression returns true, because it finds a content
element containing "task" which the thesaurus identified as a synonym for
"duty":
.//book/content contains text "duty" using
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "UF"
The following expression returns a book element, because it finds a
content element containing "users", which is a
narrower term of "people":
doc("http://bstore1.example.com/full-text.xml")
/books/book[./content contains text "people" using
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "NT" at most 2 levels]Assuming the thesaurus available at URL
"http://bstore1.example.com/UsabilitySoundex.xml"
contains soundex capabilities, the following query
returns a book element containing "Marigold" which
sounds like "Merrygould":
doc("http://bstore1.example.com/full-text.xml")
/books/book[. contains text "Merrygould" using thesaurus at
"http://bstore1.example.com/UsabilitySoundex.xml" relationship
"sounds like"]
3.4.4 Stemming Option
[Definition: A stemming option
modifies token and
phrase matching by specifying whether stemming is applied or not.
]
The "stemming" option specifies that matches may contain tokens
that have the same stem as the tokens and phrases written in the
query. It is implementation-defined what a stem of a token is.
The "no stemming" option specifies that the tokens and
phrases are not stemmed.
It is implementation-defined whether the stemming is based on an
algorithm, dictionary, or mixed approach.
The default is "no stemming".
The following expression returns true, because the title of the specified
book contains "improving" which has the same stem as
"improve":
/books/book[@number="1"]/title contains text "improve" using stemming
3.4.5 Case Option
| [241] | FTCaseOption | ::= | ("case" "insensitive")
| ("case" "sensitive")
| "lowercase"
| "uppercase" |
[Definition: A case option
modifies the matching of tokens and phrases by specifying how uppercase and
lowercase characters are considered.]
There are four possible character case options:
Using the option "case insensitive", tokens and phrases are matched,
regardless of the case of characters of the query tokens and phrases.
Using the option "case sensitive", tokens and phrases are matched,
if and only if the case of their characters is the same as written in the
query.
Using the option "lowercase", tokens and phrases are matched, if
and only if they match the query without regard to character case, but contain
only lowercase characters.
Using the option "uppercase", tokens and phrases are matched, if
and only if they match the query without regard to character case, but contain
only uppercase characters.
The default is "case insensitive".
The effect of the case options is also influenced by the query's
default collation
(see Section
2.1.1 Static ContextXQ and
Section
4.4 Default Collation DeclarationXQ).
The following table summarizes how these interact.
Case Matrix| Case option \ Default collation | UCC (Unicode Codepoint Collation) | CCS (some generic case-sensitive collation) | CCI (some generic case-insensitive collation) |
|---|
| case insensitive | compare as if both lower | case-insensitive variant of CCS if it exists, else error | CCI |
|---|
| case sensitive | UCC | CCS | case-sensitive variant of CCI if it exists, else error |
|---|
| lowercase | compare using UCC after applying fn:lower-case() to the query
string
| compare using CCS after applying fn:lower-case() to the query
string | CCI |
|---|
| uppercase | compare using UCC after applying fn:upper-case() to the query
string | compare using CCS after applying fn:upper-case() to the query
string | CCI |
|---|
Note:
In this table, "else error" means "Otherwise, an error
is raised: [err:FOCH0002]FO30".
The phrase "if it exists" is used, because
the case-sensitive collation CCS does not always have a
case-insensitive variant (and, even if one exists, it may not be
possible to determine it algorithmically), and because the
case-insensitive collation CCI does not always have a case-sensitive
variant (and, even if one exists, it may not be possible to determine
it algorithmically).
The following expression returns false, because the title element
doesn't contain "usability" in lower-case characters:
//book[@number="1"]/title contains text "Usability" using lowercase
The following expression returns true, because the character case is not
considered:
//book[@number="1"]/title contains text "usability" using case insensitive
3.4.6 Diacritics Option
[Definition: A
diacritics option
modifies token and phrase matching by specifying how diacritics are considered.
]
There are two possible diacritics options:
The option "diacritics" "insensitive" matches tokens and
phrases with and without diacritics. Whether diacritics are written in
the query or not is not considered.
The option "diacritics" "sensitive" matches tokens and phrases only
if they contain the diacritics as they are written in the query.
The default is "diacritics insensitive".
The effect of the diacritics options is also influenced by the query's
default collation
(see Section
2.1.1 Static ContextXQ and
Section
4.4 Default Collation DeclarationXQ).
The following table summarizes how these interact.
Diacritics Matrix| Diacritics option \ Default collation | UCC (Unicode Codepoint Collation) | CDS (some generic diacritics-sensitive collation) | CDI (some generic diacritics-insensitive collation) |
|---|
| diacritics insensitive | UCC comparison, but without considering diacritics | diacritics-insensitive variant of CDS
if it exists, else error | CDI |
|---|
| diacritics sensitive | UCC | CDS | diacritics-sensitive variant of CDI if it exists, else error |
|---|
Note:
In this table, "else error" means "Otherwise, an error
is raised: [err:FOCH0002]FO30".
The phrase "if it exists" is used, because
the diacritics-sensitive collation CDS does not always have a
diacritics-insensitive variant (and, even if one exists, it may not be
possible to determine it algorithmically), and because the
diacritics-insensitive collation CDI does not always have a
diacritics-sensitive variant (and, even if one exists, it may not be
possible to determine it algorithmically).
The following expression returns true, because the token "Véra" in the
editor element is matched, as the acute accent is not
considered in the comparison:
//book[@number="1"]//editor contains text "Vera" using diacritics insensitive
This returns false, because the editor element does not
contain the token "Vera" in this exact form, i.e. without any diacritics:
//book[@number="1"]/editors contains text "Vera" using diacritics sensitive
3.4.7 Stop Word Option
[Definition: A
stop word option
controls matching of tokens by specifying whether stop words are used or not.
Stop words are tokens in the query that match any token in the text being
searched.
]
More precisely,
a stop word option defines a collection of stop words according to the rules below.
Then, in every FTWords to which the stop word option applies,
each query token is checked:
if it appears
(using an implementation-defined comparison)
in the specified collection of stop words,
it is considered a stop word.
Normally a stop word matches
exactly one token, but there may be implementation-defined conditions, under
which a stop word may match a different number of tokens.
Tokens matched by stop words
retain their position numbers
and are counted by
FTDistance
and
FTWindow filters.
FTStopWords specifies the list
of stop words either explicitly as a comma-separated list of string
literals, or by the keyword at followed by a literal URI.
If the URI specifies a list of stop words that is not found in the statically
known stop word lists, an error is raised [err:FTST0008].
Whether the stop word
list is resolved from the statically known stop word lists or given explicitly,
no tokenization is performed on the stop words: they are used as they occur
in the list.
If the stop words option specifies a stop word list with a relative URI,
that relative URI is resolved to an absolute URI using the base URI in
the static context and that absolute URI is used to identify the stop word list.
Multiple stop word lists may be combined using "union" or "except".
The keywords "union" and "except" are applied from left to right. If "union" is specified, every string occurring in the lists
specified by the left-hand side or the right-hand side is a stop
word. If "except" is specified, only strings occurring in the list
specified by the left-hand side but not in the list specified
by the right-hand side are stop words.
The "stop words default" option specifies that an
implementation-defined collection of stop words is used.
The "no stop words" option specifies that no stop words are
used. This is equivalent to specifying an empty list of stop
words.
The default is "no stop words".
Note:
Some implementations may apply stop word lists during indexing and be
unable to comply with query-time requests to not apply those stop words. An
implementation may still support stop-word options (and therefore not raise
[err:FTST0006])
by applying any additional stop words specified in the query.
Pre-application of irrevocable stop word lists falls under
implementation-defined tokenization behavior in this case, and a query that
specifies "no stop words" may still have some words ignored.
In addition, an implementation that applies irrevocable stop
word lists at indexing time may therefore, as part of the implementation-defined
tokenization, fail to count those stop words in the token counts. Since the
query strings will be tokenized in accordance with the same rules, those
stop words would likewise not count in the position counts for the query string.
Thus, irrevocable stop words of this sort are invisible to the normal rules of
full-text matching defined in this specification, and are handled purely as
a tokenization issue. The examples in this specification assume that stop words
are not removed at tokenization in this way.
The following expression returns true, because the document contains the phrase
"propagating few errors":
/books/book[@number="1"]//p contains text "propagating of errors"
using stop words ("a", "the", "of") Note the asymmetry in the stop word semantics: the property of
being a stop word is only relevant to query terms, not to document
terms. Hence, it is irrelevant for the above-mentioned match whether
"few" is a stop word or not, and on the other hand we do not want the
query above to match "propagating" followed by 2 stop words, or even a
sequence of 3 stop words in the document.
Similarly, the following expression also returns true, because the document
contains the text "completion, while propagating few errors":
/books/book[@number="1"]//p contains text "in the propagating of"
using stop words ("a", "in", "the", "of")
This expression, however, returns false, because the p element in the document
ends with "errors." so there are not enough tokens to match the stop words in
the query:
/books/book[@number="1"]//p contains text "propagating few errors of the"
using stop words ("a", "in", "the", "of")
The following expression returns false. In this case specifying "few" as
a stop word has no effect, since "few" does not appear in the query.
Although the words "propagating" and "errors" appear in the text being
searched, the phrase
"propagating errors" cannot be matched, since that phrase does not occur.
/books/book[@number="1"]//p contains text "propagating errors"
using stop words ("few")The following expression returns false, because "of" is not in the p
element between "propagating" and "errors":
/books/book[@number="1"]//p contains text "propagating of errors"
using no stop words
The following expression uses the stop words list specified at the
URL. Assuming that the specified stop word list contains the word
"then", this query
is reduced to a query on the phrase "planning X conducting", allowing any
token as a substitute for X. It returns a book element,
because its content element contains "planning then
conducting". It would also return the book if the
phrases "planning and conducting" and "planning before conducting"
had been in its content:
doc("http://bstore1.example.com/full-text.xml")
/books/book[.//content contains text "planning then
conducting" using stop words at
"http://bstore1.example.com/StopWordList.xml"]
The following expression returns books containing "planning then
conducting", but not does not return books containing "planning
and conducting", since it is exempting "then" from being a stop word:
doc("http://bstore1.example.com/full-text.xml")
/books/book[.//content contains text "planning then conducting"
using stop words at "http://bstore1.example.com/StopWordList.xml"
except ("the", "then")]
3.4.8 Extension Option
[Definition: An
extension option is a match option that acts in an
implementation-defined way.
]
An extension option consists of an identifying QName and a StringLiteral.
Typically, a particular option will be recognized by some implementations and
not by others. The syntax is designed so that option declarations can be
successfully parsed by all implementations.
The QName of an extension option must resolve to a namespace URI
and local name, using
the statically known namespaces.
Note:
There is no default namespace for options.
Each implementation recognizes an
implementation-defined
set of namespace
URIs used to denote extension options.
If the namespace part of the QName is not a namespace recognized by the
implementation as one used to denote extension option, then the extension option
is ignored.
Otherwise, the effect of the extension option, including its error behavior,
is implementation-defined.
For example, if the local part of the QName is
not recognized, or if the StringLiteral does not conform to the rules
defined by the implementation for the particular extension option, the implementation may choose
whether to report an error, ignore the extension option, or take some
other action.
Implementations may impose rules on where particular extension options may
appear relative to other match options, and the
interpretation of an option declaration may depend on its position.
An extension option must not be used to change the syntax accepted by the
processor, or to suppress the detection of static errors. However, it may be
used without restriction to modify the set of tokens in the query or how they
are matched against tokens in the text being searched.
An extension option has the same scope as other match options.
The following examples illustrate several possible uses for extension
options:
This extension option is set as part of the static context of all
full-text expressions in the module and might be used to ensure that
queries are insensitive to Arabic short-vowels.
declare namespace exq = "http://example.org/XQueryImplementation";
declare ft-option using option exq:diacritics "short-vowel insensitive";
This extension option applies only to the matching in the full-text
selection in which it is found and might be used to specify how compound words
should be matched.
declare namespace exq = "http://example.org/XQueryImplementation";
//para[. contains text
("Kinder" ftand "Platz" distance exactly 1 words)
using stemming
using option exq:compounds "distance=1" ]
3.6 Positional Filters
[Definition:
Positional filters are postfix operators that serve to
filter matches based on various constraints on their positional
information.]
Recall that the grammar rule for FTSelection
allows an arbitrary number of positional filters to follow an
FTOr. In a group of multiple adjacent positional filters,
FTOrder filters are applied first, and then the other
positional filters are applied from left to right, skipping the
FTOrder filters.
That is, the first filter is applied to the
result of the FTOr, the second is applied to the
result of that first application, and so on.
An FTOr consists of
one or more FTAnds (separated by ftor),
each of which could be
an FTPosFilter applied to an embedded FTOr, enclosed in parentheses.
3.6.1 Ordered Selection
[Definition: An
ordered selection consists of a full-text selection followed by
the postfix operator "ordered".]
An ordered selection constrains the order of tokens and
phrases to be the same as the order in which they are written in the
operand selection.
The default is unordered. Unordered is in effect when ordered is
not specified in the query. Unordered cannot be written explicitly in
the query.
An ordered selection selects matches which satisfy the operand full-text
selection and which also satisfy the following constraint: the order
that the matching tokens or phrases have in the text being searched
is the same order that the corresponding query tokens or phrases have in the
operand selection. In both cases, the ordering is determined from the minimum
start positions of the constituent tokens.
The following expression returns true, because titles of book elements
contain "web site" and "usability" in the order in which they are written in
the query, i.e., "web site" must precede "usability":
//book/title contains text ("web site" ftand "usability") orderedThe following expression returns false, because although "Montana" and "Millicent"
both appear in the book element, they do not appear in the order they
are written in the query:
//book[@number="1"] contains text ("Montana" ftand "Millicent") ordered
3.6.2 Window Selection
[Definition: A
window selection consists of a full-text selection followed
by one of the (complex) postfix operators derived from FTWindow.]
A window selection selects matches which satisfy the operand full-text
selection and for which the matched tokens and phrases, more precisely the
individual StringIncludes of that match, are found
within a number of FTUnits (words, sentences, and paragraphs).
The number of FTUnits is
specified by an AdditiveExpr that is converted as though it were an argument to a
function with the expected type of xs:integer.
A window selection may cross element
boundaries. The size of the window is not affected by the presence or
absence of element boundaries. Stop words are included in the
computation of the window size whether they are ignored by the query or not.
A window selection examines the matches generated by the preceding
portion of the FTSelection, and selects those for
which the matched
tokens and phrases (more precisely, the individual StringIncludes of
that match) are all found within a window whose size is a specified
number of FTUnits (words, sentences, or paragraphs); for each such
window, the window selection then generates a match containing the
merge of those StringIncludes, plus any StringExcludes that fall
within the window.
The following expression returns true, because "web", "site", and "usability" are
within a window of 5 tokens in the title element:
/books/book/title contains text "web" ftand "site"
ftand "usability" window 5 words
The following expression returns true, because "web" and "site" in the order they are
written in the query and either "usability" or "testing" are within a
window of at most 10 tokens:
/books/book contains text ("web" ftand "site" ordered)
ftand ("usability" ftor "testing") window 10 wordsThe following expression returns false, because the
instances of "web site" and "usability" in the title element are
not within a window of 3. The phrase "Web Site Usability" in the attribute
does not apply because the attribute is not part of the string value of the node.
A similar query with a window of 5 would return true.
/books/book//title contains text "web site" ftand
"usability" window 3 words
The following expression returns the sample book element,
because its number attribute is 1 and it contains a
window of 2 words which contains an occurrence of "efficient"
but not an occurrence of "and". There is just one such matching window
in the sample text and it contains "enable efficient".
/books/book[@number="1" and . contains text "efficient"
ftand ftnot "and" window 2 words]
The following expression returns the empty sequence, because in the selected
book element, there is no occurrence of "efficient"
within a window of 3 tokens which would not also contain an occurrence
of "and":
/books/book[@number="1" and . contains text "efficient"
ftand ftnot "and" window 3 words]
In order to allow meaningful results for nested positional filters,
e.g., a window selection embedded inside a distance selection, the
resulting matches for window selections are formed from the input matches
that satisfy the window constraint as follows. All StringIncludes of
such a match are coerced into a single StringInclude that spans all
token positions from the smallest to the largest position of any input
StringIncludes. This is explained in more detail in Section 3.6.3 Distance Selection.
3.6.3 Distance Selection
[Definition: A
distance selection consists of a full-text selection followed
by one of the (complex) postfix operators derived from FTDistance.]
A distance selection selects matches which satisfy the operand full-text
selection and for which the matched tokens and phrases satisfy the
specified distance conditions.
Distances in the search context are measured in units of
tokens, sentences, or paragraphs.
Roughly speaking, the distance between two matches
is the number of intervening units,
so a distance of zero tokens (sentences, paragraphs) means
no intervening tokens (sentences, paragraphs).
More precisely,
given two matches,
we first determine their order
by sorting on starting position and if necessary on ending position.
Let M1 be the "earlier" and M2 be the "later".
(If there are overlapping tokens involved,
the designations "earlier" and "later" may not be intuitively obvious.)
Then the distance between the two is
M2's starting position minus M1's ending position, minus 1.
When computing distances
in the search context,
a distance selection
may cross element boundaries;
they affect the distance computed
only to the extent that they affect the tokenization of the search context.
Stop words are counted in those computations whether they are ignored
or not.
When a distance selection applies a distance condition to more than two matches,
the distance condition is required to hold on each successive pair of matches.
An FTDistance expresses a distance condition in terms of
an FTUnit and an FTRange.
An FTUnit can be
words, sentences, or paragraphs,
where words refers to a distance measured in tokens.
An FTRange specifies a range of integer values
by providing a minimum and/or maximum value for some integer quantity.
(Here, where the FTRange appears in an FTDistance, that quantity is a distance.
When it appears in an FTTimes, the quantity is a number of occurrences.)
Each one of the AdditiveExpr
specified in an FTRange is converted as though it were an
argument to a function with the expected parameter type of
xs:integer.
Let the value of the first (or only) operand be M. If "from" is
specified, let the value of the second operand be N.
If "exactly" is specified, then the range is the closed interval [M,
M]. If "at least" is specified, then the range is the half-closed interval
[M, unbounded). If "at most" is specified, then the range is the half-closed
interval (unbounded, M]. If "from-to" is specified, then the range is the closed
interval [M, N]. Note: If M is greater than N, the range is empty.
Here are some examples of FTRanges:
'exactly 0' specifies the range [0, 0].
'at least 1' specifies the range [1,unbounded).
'at most 1' specifies the range (unbounded, 1].
'from 5 to 10' specifies the range [5, 10].
The following expression returns false, because "completion" and "errors" are
less than 11 tokens apart:
/books/book contains text ("completion" ftand "errors"
distance at least 11 words)The following expression returns true:
/books/book contains text "web" ftand "site" ftand
"usability" distance at most 2 words
The search context contains two occurrences of the phrase
"the usability of a web site"
(once in the <title> and once in the <content>).
In this phrase,
the tokens "usability" and "web" have a distance of 2 words,
and the tokens "web" and "site" have a distance of 0 words,
both of which satisfy the constraint distance at most 2 words.
(The tokens "usability" and "site" have a distance of 3 words,
but this does not cause the distance filter to fail,
because these are not successive matches.)
Thus, the full-text selection yields two matches,
and the whole expression yields true.
(The phrase "Improving Web Site Usability" would also satisfy the given full-text selection,
but in the sample document it occurs in an attribute value,
and so does not contribute to the string value or the tokenization of the book element.)
The following expression returns the empty sequence, because
between any token "usability" and the token in any occurrence of the phrase
"web site" that is the nearest to the token "usability" there is always more
than one intervening token:
/books/book[.//p contains text "web site"
ftand "usability" distance at most 1 words]
The following expression returns the book title, because for
the occurrences of the tokens "web" and "users" in the note
element only one intervening token appears:
/books/book[. contains text "web"
ftand "users" distance at most 1 words]/title
In order to allow meaningful results for nested positional filters,
e.g., a distance selection embedded inside another distance selection, the
resulting matches for distance selections are formed from the input matches
that satisfy the distance constraint as follows. All StringIncludes of
such a match are coerced into a single StringInclude that spans all
token positions from the smallest to the largest position of any input
StringIncludes. Thus, a distance selection that embeds a window or a
distance selection takes the result of the embedded selection as a
single unit.
The following gives an example of nested distance selections:
/books/book contains text ((("richard" ftand "nixon") distance at most 2 words)
ftand
(("george" ftand "bush") distance at most 2 words)
distance at least 20 words)
This expression allows to find book elements that contain, for instance,
"Richard M. Nixon" and "George W. Bush" at least 20 words apart. The
matches for the inner distance selections are treated as single units
(represented by StringIncludes) by the outer distance
selection. Suppose such phrases are present in
the search context, then the outer distance selection
enforces a constraint on the number of intervening tokens ("at least
20") between the
last token of "Richard M. Nixon" and the first token of "George
W. Bush".
3.6.4 Scope Selection
[Definition: A
scope selection consists of a full-text selection followed
by one of the (complex) postfix operators derived from FTScope.]
A scope selection selects matches which satisfy the operand full-text
selection and for which the matched tokens and phrases are
contained in the same scope or in different scopes.
Possible scopes are sentences and paragraphs.
By default, there are no restrictions on the scope of the
matches.
The following expression returns false, because the tokens "usability" and "Marigold"
are not contained within the same sentence:
//book contains text "usability" ftand "Marigold" same sentence
The following expression returns true, because the tokens "usability" and "Marigold"
are contained within different sentences:
//book contains text "usability" ftand "Marigold" different sentence
The following expression returns a book element, because it contains
"usability" and "testing" in the same paragraph:
//book[. contains text "usability" ftand "testing" same paragraph]
The following expression returns a book element, because "site" and
"errors" appear in the same sentence:
//book[. contains text "site" ftand "errors" same sentence]
It is possible that both "same sentence" and "different sentence" conditions are
simultaneously safisfied for several tokens and/or phrases within the same
document fragment. This can be observed if there are occurrences of the tokens
and/or phrases both within the same sentence and within difference sentences. For
example, consider the following document fragment.
<introduction>
... The usability of a Web site is how well the site supports the user in
achieving specified goals. ... Expert reviews and usability testing are methods of
identifying problems in layout, terminology, and navigation. ...
</introduction>
This sample will satisfy both conditions ("usability" ftand "reviews")
different sentence and ("usability" ftand "reviews") same
sentence. The tokens "usability" and "reviews" occur both in different sentences
(the first and second shown sentences) and in the same sentence (the second shown
sentences.)
The above observation also holds for the "same paragraph" and "different paragraph"
conditions.
3.6.5 Anchoring Selection
| [238] | FTContent | ::= | ("at" "start") | ("at" "end") | ("entire" "content") |
[Definition: An
anchoring selection consists of a full-text selection followed
by one of the postfix operators "at start", "at end", or "entire content".]
An anchoring selection selects matches which satisfy the operand full-text
selection and for which the matched tokens and phrases are
the first, last, or all tokens in the tokenized form of the items being searched.
Using the "at start" operator, tokens or phrases are matched, if they
cover the first token position in the tokenized string value
of the item being searched.
Using the "at end" operator, tokens or phrases are matched, if they
cover the last token position in the tokenized string value of the
item being searched.
Using the "entire content" operator, tokens or phrases are matched, if they
cover all token positions of the tokenized string value of the
item being searched.
The following expression returns each title element starting with the
phrase "improving the usability of a web site":
/books//title[. contains text "improving the usability
of a web site" at start]
The following expression returns the p element of the sample,
because it ends with the phrase
"propagating few errors":
/books//p[. contains text "propagat.*" using wildcards ftand "few
errors" distance at most 2 words at end]
Since the distance operator doesn't imply an ordering, the last example
would also yield a match if the p element ended with, say,
"few errors are propagated".
The following expression returns each note element whose entire content
is "this book has been approved by the web site users association":
/books//note[. contains text "this book has been
approved by the web site users association" entire content]
The following example returns true because
both the content and the note elements match:
/books//* contains text "Association" at end
3.8 Extension Selections
[Definition: An extension selection is a full-text selection whose semantics are
implementation-defined.]
Typically, a particular extension will be recognized
by some implementations and not by others. The syntax is designed so that
extension selections can be successfully parsed by all implementations, and
so that fallback behavior can be defined for implementations that do not
recognize a particular extension.
An extension selection consists of one or more pragmas followed by a full-text selection enclosed in curly braces. See
Section
3.14 Extension ExpressionsXQ for information on
pragmas in general.
A pragma is denoted by the delimiters (# and #), and
consists of an identifying QName followed by implementation-defined
content.
The content of a pragma may consist of any string of characters that does not contain the ending delimiter #). The QName of a
pragma must resolve to a namespace URI and local name, using the statically known namespaces.
Note:
Since there is no default namespace for
pragmas, a pragma QName must include a namespace prefix.
Each implementation recognizes an
implementation-defined set of namespace
URIs used to denote pragmas.
If the namespace part of a pragma QName is not recognized by the
implementation as a pragma namespace, then the pragma
is ignored. If all the pragmas in an FTExtensionSelection are ignored, then full-text extension selection is just
the full-text selection enclosed in curly braces; if this full-text selection is absent, then a static error is
raised [err:XQST0079]XQ30.
If an implementation recognizes the namespace of one or more pragmas in an FTExtensionSelection, then the value
of the FTExtensionSelection, including its error behavior, is implementation-defined. For example, an implementation that recognizes the namespace of
a pragma QName, but does not recognize the local part of the QName, might choose
either to raise an error or to ignore the pragma.
It is a static error [err:XQST0013]XQ30
if an implementation recognizes a pragma but
determines that its content is invalid.
If an implementation recognizes a pragma, it must report any static
errors in the following full-text selection even if it will not apply that
selection.
The following examples illustrate three ways in which extension selections might be
used.
A pragma can be used to furnish a hint for how to evaluate the
following full-text selection, without actually changing the result.
For example:
declare namespace exq = "http://example.org/XQueryImplementation";
/books/book/author[name contains text (# exq:use-index #) {'Berners-Lee'}]
An implementation that recognizes the exq:use-index pragma might use an
index to evaluate the full-text selection that follows. An implementation that
does not recognize this pragma would evaluate the full-text selection in its normal
way.
A pragma might be used to modify the semantics of the following
full-text selection in ways that would not (in the absence of the pragma) be
conformant with this specification. For example, a pragma might be used to
change distance counting so that adjacent words are at a distance of 1
(otherwise they would be at a distance of 0):
declare namespace exq = "http://example.org/XQueryImplementation";
/books/book[.//p contains text (# exq:distance #) { "web site"
ftand "usability" distance at most 1 words }]
Such changes to the language
semantics must be scoped to the expression contained within the curly
braces following the pragma.
A pragma might contain syntactic constructs that are
evaluated in place of the following full-text selection. In this case, the
following selection itself (if it is present) provides a fallback for use by
implementations that do not recognize the pragma. For example:
declare namespace exq = "http://example.org/XQueryImplementation";
//city[. contains text (# exq:classifier with class 'Animals' #)
{"animal" using thesaurus at "http://example.org/thesaurus.xml"
relationship "RT"}]
Here an implementation that recognizes the pragma will return the result of
evaluating the proprietary syntax with class 'animals',
while an implementation that does not recognize the pragma will instead
return the result of the thesaurus option.
If no fallback expression is required, or
if none is feasible, then the expression between the curly braces may be
omitted, in which case implementations that do not recognize the pragma will
raise a static error.
