HTML5

2.6.2 Parsing URLs

To parse a URL url into its component parts, the user agent must use the following steps:

1. Strip leading and trailing whitespace from url.

2. Parse url in the manner defined by RFC 3986, with the following exceptions:

   • Add all characters with code points less than or equal to U+0020 or greater than or equal to U+007F to the <unreserved> production.
   • Add the characters U+0022, U+003C, U+003E, U+005B .. U+005E, U+0060, and U+007B .. U+007D to the <unreserved> production.
   • Add a single U+0025 PERCENT SIGN character as a second alternative way of matching the <pct-encoded> production, except when the <pct-encoded> is used in the <reg-name> production.
   • Add the U+0023 NUMBER SIGN character to the characters allowed in the <fragment> production.

3. If url doesn't match the <URI-reference> production, even after the above changes are made to the ABNF definitions, then parsing the URL fails with an error. [RFC3986]

   Otherwise, parsing url was successful; the components of the URL are substrings of url defined as follows:

   <scheme>

   The substring matched by the <scheme> production, if any.

   <host>

   The substring matched by the <host> production, if any.

   <port>

   The substring matched by the <port> production, if any.

   <hostport>

   If there is a <scheme> component and a <port> component and the port given by the <port> component is different than the default port defined for the protocol given by the <scheme> component, then <hostport> is the substring that starts with the substring matched by the <host> production and ends with the substring matched by the <port> production, and includes the colon in between the two. Otherwise, it is the same as the <host> component.

   <path>

   The substring matched by one of the following productions, if one of them was matched:

   • <path-abempty>
   • <path-absolute>
   • <path-noscheme>
   • <path-rootless>
   • <path-empty>

   <query>

   The substring matched by the <query> production, if any.

   <fragment>

   The substring matched by the <fragment> production, if any.

   <host-specific>

   The substring that follows the substring matched by the <authority> production, or the whole string if the <authority> production wasn't matched.

These parsing rules are a willful violation of RFC 3986 and RFC 3987 (which do not define error handling), motivated by a desire to handle legacy content. [RFC3986] [RFC3987]

To resolve a URL to an absolute URL relative to either another absolute URL or an element, the user agent must use the following steps. Resolving a URL can result in an error, in which case the URL is not resolvable.

1. Let url be the URL being resolved.

2. Let encoding be determined as follows:

   If the URL had a character encoding defined when the URL was created or defined

   The URL character encoding is as defined.

   If the URL came from a script (e.g. as an argument to a method)

   The URL character encoding is the script's URL character encoding.

   If the URL came from a DOM node (e.g. from an element)

   The node has a Document, and the URL character encoding is the document's character encoding.

3. If encoding is a UTF-16 encoding, then change the value of encoding to UTF-8.

4. If the algorithm was invoked with an absolute URL to use as the base URL, let base be that absolute URL.

   Otherwise, let base be the base URI of the element, as defined by the XML Base specification, with the base URI of the document entity being defined as the document base URL of the Document that owns the element. [XMLBASE]

   For the purposes of the XML Base specification, user agents must act as if all Document objects represented XML documents.

   It is possible for xml:base attributes to be present even in HTML fragments, as such attributes can be added dynamically using script. (Such scripts would not be conforming, however, as xml:base attributes are not allowed in HTML documents.)

5. Parse url into its component parts.

6. If parsing url resulted in a <host> component, then replace the matching substring of url with the string that results from expanding any sequences of percent-encoded octets in that component that are valid UTF-8 sequences into Unicode characters as defined by UTF-8.

   If any percent-encoded octets in that component are not valid UTF-8 sequences (e.g. sequences of percent-encoded octets that expand to surrogate code points), then return an error and abort these steps.

   Apply the IDNA ToASCII algorithm to the matching substring, with both the AllowUnassigned and UseSTD3ASCIIRules flags set. Replace the matching substring with the result of the ToASCII algorithm.

   If ToASCII fails to convert one of the components of the string, e.g. because it is too long or because it contains invalid characters, then return an error and abort these steps. [RFC3490]

7. If parsing url resulted in a <path> component, then replace the matching substring of url with the string that results from applying the following steps to each character other than "%" (U+0025) that doesn't match the original <path> production defined in RFC 3986:

   1. Encode the character into a sequence of octets as defined by UTF-8.
   2. Replace the character with the percent-encoded form of those octets. [RFC3986]

   For instance if url was "//example.com/a^b☺c%FFd%z/?e", then the <path> component's substring would be "/a^b☺c%FFd%z/" and the two characters that would have to be escaped would be "^" and "☺". The result after this step was applied would therefore be that url now had the value "//example.com/a%5Eb%E2%98%BAc%FFd%z/?e".

8. If parsing url resulted in a <query> component, then replace the matching substring of url with the string that results from applying the following steps to each character other than "%" (U+0025) that doesn't match the original <query> production defined in RFC 3986:

   1. If the character in question cannot be expressed in the encoding encoding, then replace it with a single 0x3F octet (an ASCII question mark) and skip the remaining substeps for this character.
   2. Encode the character into a sequence of octets as defined by the encoding encoding.
   3. Replace the character with the percent-encoded form of those octets. [RFC3986]

9. Apply the algorithm described in RFC 3986 section 5.2 Relative Resolution, using url as the potentially relative URI reference (R), and base as the base URI (Base). [RFC3986]

10. Apply any relevant conformance criteria of RFC 3986 and RFC 3987, returning an error and aborting these steps if appropriate. [RFC3986] [RFC3987]

    For instance, if an absolute URI that would be returned by the above algorithm violates the restrictions specific to its scheme, e.g. a data: URI using the "//" server-based naming authority syntax, then user agents are to treat this as an error instead.

11. Let result be the target URI (T) returned by the Relative Resolution algorithm.

12. If result uses a scheme with a server-based naming authority, replace all "\" (U+005C) characters in result with "/" (U+002F) characters.

13. Return result.

Some of the steps in these rules, for example the processing of "\" (U+005C) characters, are a willful violation of RFC 3986 and RFC 3987, motivated by a desire to handle legacy content. [RFC3986] [RFC3987]

2.6.5 URL manipulation and creation

To fragment-escape a string input, a user agent must run the following steps:

1. Let input be the string to be escaped.

2. Let position point at the first character of input.

3. Let output be an empty string.

4. Loop: If position is past the end of input, then jump to the step labeled end.

5. If the character in input pointed to by position is in the range U+0000 to U+0020 or is one of the following characters:

   • """ (U+0022) character
   • "#" (U+0023) character
   • "%" (U+0025) character
   • U+003C LESS-THAN SIGN character (<)
   • ">" (U+003E) character
   • "[" (U+005B) character
   • "\" (U+005C) character
   • "]" (U+005D) character
   • "^" (U+005E) character
   • "{" (U+007B) character
   • "|" (U+007C) character
   • "}" (U+007D) character

   ...then append the percent-encoded form of the character to output. [RFC3986]

   Otherwise, append the character itself to output.

   This escapes any ASCII characters that are not valid in the URI <fragment> production without being escaped.

6. Advance position to the next character in input.

7. Return to the step labeled loop.

8. End: Return output.

2.6.6 Dynamic changes to base URLs

When an xml:base attribute is set, changed, or removed, the attribute's element, and all descendant elements, are affected by a base URL change.

When a document's document base URL changes, all elements in that document are affected by a base URL change.

The following are base URL change steps, which run when an element is affected by a base URL change (as defined by the DOM Core specification):

If the element creates a hyperlink

If the absolute URL identified by the hyperlink is being shown to the user, or if any data derived from that URL is affecting the display, then the href attribute should be re-resolved relative to the element and the UI updated appropriately.

For example, the CSS :link/:visited pseudo-classes might have been affected.

If the element is a q, blockquote, ins, or del element with a cite attribute

If the absolute URL identified by the cite attribute is being shown to the user, or if any data derived from that URL is affecting the display, then the URL should be re-resolved relative to the element and the UI updated appropriately.

Otherwise

The element is not directly affected.

For instance, changing the base URL doesn't affect the image displayed by img elements, although subsequent accesses of the src IDL attribute from script will return a new absolute URL that might no longer correspond to the image being shown.

           attribute DOMString protocol;
           attribute DOMString host;
           attribute DOMString hostname;
           attribute DOMString port;
           attribute DOMString pathname;
           attribute DOMString search;
           attribute DOMString hash;

---

The attributes defined to be URL decomposition IDL attributes must act as described for the attributes with the same corresponding names in this section.

In addition, an interface with a complement of URL decomposition IDL attributes defines an input, which is a URL that the attributes act on, and a common setter action, which is a set of steps invoked when any of the attributes' setters are invoked.

The seven URL decomposition IDL attributes have similar requirements.

On getting, if the input is an absolute URL that fulfills the condition given in the "getter condition" column corresponding to the attribute in the table below, the user agent must return the part of the input URL given in the "component" column, with any prefixes specified in the "prefix" column appropriately added to the start of the string and any suffixes specified in the "suffix" column appropriately added to the end of the string. Otherwise, the attribute must return the empty string.

On setting, the new value must first be mutated as described by the "setter preprocessor" column, then mutated by %-escaping any characters in the new value that are not valid in the relevant component as given by the "component" column. Then, if the input is an absolute URL and the resulting new value fulfills the condition given in the "setter condition" column, the user agent must make a new string output by replacing the component of the URL given by the "component" column in the input URL with the new value; otherwise, the user agent must let output be equal to the input. Finally, the user agent must invoke the common setter action with the value of output.

When replacing a component in the URL, if the component is part of an optional group in the URL syntax consisting of a character followed by the component, the component (including its prefix character) must be included even if the new value is the empty string.

The previous paragraph applies in particular to the ":" before a <port> component, the "?" before a <query> component, and the "#" before a <fragment> component.

For the purposes of the above definitions, URLs must be parsed using the URL parsing rules defined in this specification.

Attribute	Component	Getter Condition	Prefix	Suffix	Setter Preprocessor	Setter Condition
protocol	<scheme>	—	—	":" (U+003A)	Remove all trailing ":" (U+003A) characters	The new value is not the empty string
host	<hostport>	input is an authority-based URL	—	—	—	The new value is not the empty string and input is an authority-based URL
hostname	<host>	input is an authority-based URL	—	—	Remove all leading "/" (U+002F) characters	The new value is not the empty string and input is an authority-based URL
port	<port>	input is an authority-based URL, and contained a <port> component (possibly an empty one)	—	—	Remove all characters in the new value from the first that is not in the range ASCII digits, if any. Remove any leading "0" (U+0030) characters in the new value. If the resulting string is empty, set it to a single "0" (U+0030) character.	input is an authority-based URL, and the new value, when interpreted as a base-ten integer, is less than or equal to 65535
pathname	<path>	input is a hierarchical URL	—	—	If it has no leading "/" (U+002F) character, prepend a "/" (U+002F) character to the new value	input is hierarchical
search	<query>	input is a hierarchical URL, and contained a <query> component (possibly an empty one)	"?" (U+003F)	—	Remove one leading "?" (U+003F) character, if any	input is a hierarchical URL
hash	<fragment>	input contained a non-empty <fragment> component	"#" (U+0023)	—	Remove one leading "#" (U+0023) character, if any	—

2.7 Fetching resources

2.7.1 Terminology

User agents can implement a variety of transfer protocols, but this specification mostly defines behavior in terms of HTTP. [HTTP]

The HTTP GET method is equivalent to the default retrieval action of the protocol. For example, RETR in FTP. Such actions are idempotent and safe, in HTTP terms.

The HTTP response codes are equivalent to statuses in other protocols that have the same basic meanings. For example, a "file not found" error is equivalent to a 404 code, a server error is equivalent to a 5xx code, and so on.

The HTTP headers are equivalent to fields in other protocols that have the same basic meaning. For example, the HTTP authentication headers are equivalent to the authentication aspects of the FTP protocol.

---

A referrer source is either a Document or a URL.

2.7.2 Processing model

When a user agent is to fetch a resource or URL, optionally from an origin origin, optionally using a specific referrer source as an override referrer source, and optionally with a synchronous flag, a manual redirect flag, a force same-origin flag, and/or a block cookies flag, the following steps must be run. (When a URL is to be fetched, the URL identifies a resource to be obtained.)

1. If there is a specific override referrer source, and it is a URL, then let referrer be the override referrer source, and jump to the step labeled clean referrer.

2. Let document be the appropriate Document as given by the following list:

   If there is a specific override referrer source

   The override referrer source.

   When navigating

   The active document of the source browsing context.

   When fetching resources for an element

   The element's Document.

3. While document is an iframe srcdoc document, let document be document's browsing context's browsing context container's Document instead.

4. If the origin of Document is not a scheme/host/port tuple, then set referrer to the empty string and jump to the step labeled clean referrer.

5. Let referrer be the document's address of document.

6. Clean referrer: Remove any <fragment> component from referrer.

7. If referrer is not the empty string, is not a data: URL, is not a javascript: URL, and is not the URL "about:blank", then generate the address of the resource from which Request-URIs are obtained as required by HTTP for the Referer (sic) header from referrer. [HTTP]

   Otherwise, the Referer (sic) header must be omitted, regardless of its value.

8. If the algorithm was not invoked with the synchronous flag, perform the remaining steps asynchronously.

9. This is the main step.

   If the resource is identified by an absolute URL, and the resource is to be obtained using an idempotent action (such as an HTTP GET or equivalent), and it is already being downloaded for other reasons (e.g. another invocation of this algorithm), and this request would be identical to the previous one (e.g. same Accept and Origin headers), and the user agent is configured such that it is to reuse the data from the existing download instead of initiating a new one, then use the results of the existing download instead of starting a new one.

   Otherwise, if the resource is identified by an absolute URL with a scheme that does not define a mechanism to obtain the resource (e.g. it is a mailto: URL) or that the user agent does not support, then act as if the resource was an HTTP 204 No Content response with no other metadata.

   Otherwise, if the resource is identified by the URL about:blank, then the resource is immediately available and consists of the empty string, with no metadata.

   Otherwise, at a time convenient to the user and the user agent, download (or otherwise obtain) the resource, applying the semantics of the relevant specifications (e.g. performing an HTTP GET or POST operation, or reading the file from disk, dereferencing javascript: URLs, etc).

   For the purposes of the Referer (sic) header, use the address of the resource from which Request-URIs are obtained generated in the earlier step.

   For the purposes of the Origin header, if the fetching algorithm was explicitly initiated from an origin, then the origin that initiated the HTTP request is origin. Otherwise, this is a request from a "privacy-sensitive" context. [ORIGIN]

10. If the algorithm was not invoked with the block cookies flag, and there are cookies to be set, then the user agent must run the following substeps:

    1. Wait until ownership of the storage mutex can be taken by this instance of the fetching algorithm.

    2. Take ownership of the storage mutex.

    3. Update the cookies. [COOKIES]

    4. Release the storage mutex so that it is once again free.

11. If the fetched resource is an HTTP redirect or equivalent, then:

    If the force same-origin flag is set and the URL of the target of the redirect does not have the same origin as the URL for which the fetch algorithm was invoked

    Abort these steps and return failure from this algorithm, as if the remote host could not be contacted.

    If the manual redirect flag is set

    Continue, using the fetched resource (the redirect) as the result of the algorithm. If the calling algorithm subsequently requires the user agent to transparently follow the redirect, then the user agent must resume this algorithm from the main step, but using the target of the redirect as the resource to fetch, rather than the original resource.

    Otherwise

    First, apply any relevant requirements for redirects (such as showing any appropriate prompts). Then, redo main step, but using the target of the redirect as the resource to fetch, rather than the original resource. For HTTP requests, the new request must include the same headers as the original request, except for headers for which other requirements are specified (such as the Host header). [HTTP]

    The HTTP specification requires that 301, 302, and 307 redirects, when applied to methods other than the safe methods, not be followed without user confirmation. That would be an appropriate prompt for the purposes of the requirement in the paragraph above. [HTTP]

12. If the algorithm was not invoked with the synchronous flag: When the resource is available, or if there is an error of some description, queue a task that uses the resource as appropriate. If the resource can be processed incrementally, as, for instance, with a progressively interlaced JPEG or an HTML file, additional tasks may be queued to process the data as it is downloaded. The task source for these tasks is the networking task source.

    Otherwise, return the resource or error information to the calling algorithm.

If the user agent can determine the actual length of the resource being fetched for an instance of this algorithm, and if that length is finite, then that length is the file's size. Otherwise, the subject of the algorithm (that is, the resource being fetched) has no known size. (For example, the HTTP Content-Length header might provide this information.)

The user agent must also keep track of the number of bytes downloaded for each instance of this algorithm. This number must exclude any out-of-band metadata, such as HTTP headers.

The application cache processing model introduces some changes to the networking model to handle the returning of cached resources.

The navigation processing model handles redirects itself, overriding the redirection handling that would be done by the fetching algorithm.

Whether the type sniffing rules apply to the fetched resource depends on the algorithm that invokes the rules — they are not always applicable.

2.7.3 Encrypted HTTP and related security concerns

Anything in this specification that refers to HTTP also applies to HTTP-over-TLS, as represented by URLs representing the https scheme. [HTTPS]

User agents should report certificate errors to the user and must either refuse to download resources sent with erroneous certificates or must act as if such resources were in fact served with no encryption.

User agents should warn the user that there is a potential problem whenever the user visits a page that the user has previously visited, if the page uses less secure encryption on the second visit.

Not doing so can result in users not noticing man-in-the-middle attacks.

2.7.4 Determining the type of a resource

The Content-Type metadata of a resource must be obtained and interpreted in a manner consistent with the requirements of the Media Type Sniffing specification. [MIMESNIFF]

The sniffed type of a resource must be found in a manner consistent with the requirements given in the Media Type Sniffing specification for finding the sniffed-type of the relevant sequence of octets. [MIMESNIFF]

The rules for sniffing images specifically and the rules for distinguishing if a resource is text or binary are also defined in the Media Type Sniffing specification. Both sets of rules return a MIME type as their result. [MIMESNIFF]

It is imperative that the rules in the Media Type Sniffing specification be followed exactly. When a user agent uses different heuristics for content type detection than the server expects, security problems can occur. For more details, see the Media Type Sniffing specification. [MIMESNIFF]

2.7.5 Extracting character encodings from meta elements

The algorithm for extracting a character encoding from a meta element, given a string s, is as follows. It either returns a character encoding or nothing.

1. Let position be a pointer into s, initially pointing at the start of the string.

2. Loop: Find the first seven characters in s after position that are an ASCII case-insensitive match for the word "charset". If no such match is found, return nothing and abort these steps.

3. Skip any space characters that immediately follow the word "charset" (there might not be any).

4. If the next character is not a "=" (U+003D), then move position to point just before that next character, and jump back to the step labeled loop.

5. Skip any space characters that immediately follow the equals sign (there might not be any).

6. Process the next character as follows:

   If it is a """ (U+0022) character and there is a later """ (U+0022) character in s

   If it is a "'" (U+0027) character and there is a later "'" (U+0027) character in s

   Return the encoding corresponding to the string between this character and the next earliest occurrence of this character.

   If it is an unmatched """ (U+0022) character

   If it is an unmatched "'" (U+0027) character

   If there is no next character

   Return nothing.

   Otherwise

   Return the encoding corresponding to the string from this character to the first space character or ";" (U+003B) character, or the end of s, whichever comes first.

This algorithm is distinct from those in the HTTP specification (for example, HTTP doesn't allow the use of single quotes and requires supporting a backslash-escape mechanism that is not supported by this algorithm). While the algorithm is used in contexts that, historically, were related to HTTP, the syntax as supported by implementations diverged some time ago. [HTTP]

2.7.7 CORS-enabled fetch

When the user agent is required to perform a potentially CORS-enabled fetch of an absolute URL URL with a mode mode that is either "No CORS", "Anonymous", or "Use Credentials", optionally using a referrer source referrer source, with an origin origin, and with a default origin behaviour default which is either "taint" or "fail", it must run the first applicable set of steps from the following list. The default origin behaviour is only used if mode is "No CORS". This algorithm wraps the fetch algorithm above, and labels the obtained resource as either CORS-same-origin or CORS-cross-origin, or blocks the resource entirely.

If the URL has the same origin as origin

If the URL is a data: URL

If the URL is a javascript: URL

If the URL is about:blank

Run these substeps:

1. Fetch URL, using referrer source if one was specified, with the manual redirect flag set.

2. Loop: Wait for the fetch algorithm to know if the result is a redirect or not.

3. Follow the first appropriate steps from the following list:

   If the result of the fetch is a redirect, and the origin of the target URL of the redirect is not the same origin as origin

   Set URL to the the target URL of the redirect and return to the top of the potentially CORS-enabled fetch algorithm (this time, one of the other branches below might be taken, based on the value of mode).

   If the result of the fetch is a redirect

   The origin of the target URL of the redirect is the same origin as origin.

   Transparently follow the redirect and jump to the step labeled loop above.

   Otherwise

   The resource is available, it is not a redirect, and its origin is the same origin as origin.

   The tasks from the fetch algorithm are queued normally, and for the purposes of the calling algorithm, the obtained resource is CORS-same-origin.

If mode is "No CORS" and default is taint

The URL does not have the same origin as origin.

Fetch URL, using referrer source if one was specified.

The tasks from the fetch algorithm are queued normally, but for the purposes of the calling algorithm, the obtained resource is CORS-cross-origin. The user agent may report a cross-origin resource access failure to the user (e.g. in a debugging console).

If mode is "No CORS"

The URL does not have the same origin as origin, and default is fail.

Discard any data fetched as part of this algorithm, and prevent any tasks from such invocations of the fetch algorithm from being queued. For the purposes of the calling algorithm, the user agent must act as if there was a fatal network error and no resource was obtained. The user agent may report a cross-origin resource access failure to the user (e.g. in a debugging console).

If mode is "Anonymous" or "Use Credentials"

The URL does not have the same origin as origin.

Run these steps:

1. Perform a cross-origin request with the request URL set to URL, using referrer source if one was specified, with the source origin set to origin, and with the omit credentials flag set if mode is "Anonymous" and not set otherwise. [CORS]

2. Wait for the CORS cross-origin request status to have a value.

3. Jump to the appropriate step from the following list:

   If the CORS cross-origin request status is not success

   Discard all fetched data and prevent any tasks from the fetch algorithm from being queued. For the purposes of the calling algorithm, the user agent must act as if there was a fatal network error and no resource was obtained. If a CORS resource sharing check failed, the user agent may report a cross-origin resource access failure to the user (e.g. in a debugging console).

   If the CORS cross-origin request status is success

   The tasks from the fetch algorithm are queued normally, and for the purposes of the calling algorithm, the obtained resource is CORS-same-origin.