The auto-generated part didn't correctly create the proper assertions here. I have corrected that (it now checks for output "CAUGHT" for catching the error) and pushed to the repo.

The test assumes that it is incorrect to directly call system-property:

<!-- raises XPST0017 -->
<xsl:evaluate xpath="'system-property(''xsl:version'')'" />

But also assumes that it is allowed to wrap the call to system-property in a function item, because in that case, its context is outside the xsl:evaluate (and we do not explicitly disallow this in the spec, I think):

<!-- succeeds -->
<xsl:evaluate xpath="'$f(''xsl:version'')'">
   <xsl:with-param name="f" select="system-property#1" />
</xsl:evaluate>

It then tests variants and permutations of the above calls (i.e., different ways of capturing the static and dynamic context).

I'm now seeing many tests (e.g. system-property-101a) fail like this:

XPath assertion not(empty(/output/static-context/result)) failed

I can't see why the test would output an /output/static-context/result element. This is what it actually produces:

<output evaluation-method="static"
        eval-static-variables="true"
        ns-normal="true">
   <global-variables>
      <static-context>
         <static-call result-for="system-property(('xsl:version'))">3.0</static-call>
         <ref-call result-for="system-property#1(('xsl:version'))">3.0</ref-call>
         <let-call result-for="let $f := system-property#1 return $f(('xsl:version'))">3.0</let-call>
         <anon-call result-for="function() { system-property(('xsl:version')) }()">3.0</anon-call>
         <partial-call result-for="system-property(?)(('xsl:version'))">3.0</partial-call>
         <lookup-call result-for="let $f := function-lookup(QName('http://www.w3.org/2005/xpath-functions', 'system-property'), 1) return (if($f instance of function(*)) then $f(('xsl:version')) else ())">3.0</lookup-call>
      </static-context>
      <dynamic-context>
         <static-call result-for="system-property(('xsl:version'))">3.0</static-call>
         <ref-call result-for="system-property#1(('xsl:version'))">3.0</ref-call>
         <let-call result-for="let $f := system-property#1 return $f(('xsl:version'))">3.0</let-call>
         <anon-call result-for="function() { system-property(('xsl:version')) }()">3.0</anon-call>
         <partial-call result-for="system-property(?)(('xsl:version'))">3.0</partial-call>
         <lookup-call result-for="let $f := function-lookup(QName('http://www.w3.org/2005/xpath-functions', 'system-property'), 1) return (if($f instance of function(*)) then $f(('xsl:version')) else ())">3.0</lookup-call>
      </dynamic-context>
   </global-variables>
</output>

Looks like when I rerun the tests, I wasn't rerunning all of them, failing to see I introduced an error in another area than I was fixing...

The problem was in an overzealously copied test assertion (when fixing the previous issue) that wasn't supposed to end up in the a/b/c versions of these tests.

Thanks, fixed now. I also simplified the assertions a bit.

Thanks. I'm still getting failures on the d/e/f teats. It looks like the try/catch is catching XPST0017 (can't quite see where that comes from...) but Saxon is throwing XTDE3160. I think XTDE3160 is correct here: dynamic errors in xsl:evaluate are returned unchanged, static errors are returned as XTDE3160. I don't remember the history of that, but that's what the spec says.

I believe that XTDE3160 is used if the syntax is incorrect, but syntactically, there's nothing wrong with these xsl:evaluate/@xpath strings.

XPST0017 (quote):
If the expanded QName and arity in a named function reference do not match the name and arity of a function signature in the static context, a static error is raised [err:XPST0017].

Since fn:system-property is not available in xsl:evaluate, it seems to me that XPST0017 is the expected error here. From what I read in XP30, this error is raised for static function calls, partial function application and named function references.

The XPath strings that each throw this error are (only argument differs):

system-property(('xsl:xsd-version'))
system-property#1(('xsl:xsd-version'))
let $f := system-property#1 return $f(('xsl:xsd-version'))
function() { system-property(('xsl:xsd-version')) }()
system-property(?)(('xsl:xsd-version'))

Oh wait, I see what you mean, in parentheses of XTDE3160 it says "(that is, if a static error occurs when analyzing the string according to the rules of the XPath specification)."

I thought that only happens on a *syntax* error, but as written, it seems to apply to any kind of static error. Rather unfortunate, it limits the types of errors you can catch this way with xsl:evaluate...

I like the "we raise the same error as when you weren't using xsl:evaluate" better.

> I don't remember the history of that, but that's what the spec says.

I tried to check back, but the first introduction of this error is in the first WD of XSLT 2.1 as XTDE2150, later renumbered as XTDE3160. There has not been any (un)minuted discussion about it, nor any raised bug.

It is unfortunate, mainly because we don't support nested errors, so this leaves no way for the user to catch on specific errors (other then introspecting the error text).

I will fix the test though.

Thanks, these tests are now working in Saxon (after fixing one more Saxon bug...).

I think the reason we decided that xsl:evaluate converts all static errors into a single code may have been that in general, with static errors, a parser often reports more than one, and it may be rather arbitrary what code comes back; also in a typical use case, the XPath expression is read from some external source and if it's wrong, you're unlikely to want to take different recovery action depending on exactly how it's wrong.