Reporting API

4. Report Delivery

Over time, various features will queue up a list of reports in the user agent’s reporting cache. The user agent will periodically grab the list of currently pending reports, and deliver them to the associated endpoints. This document does not define a schedule for the user agent to follow, and assumes that the user agent will have enough contextual information to deliver reports in a timely manner, balanced against impacting a user’s experience.

That said, a user agent SHOULD make a effort to deliver reports as soon as possible after queuing, as a report’s data might be significantly more useful in the period directly after its generation than it would be a day or a week later.

4.1. Media Type

The media type used when POSTing reports to a specified endpoint is application/reports+json.

4.2. Queue data as type for endpoint group on settings

Given a serializable object (data), a string (type), another string (endpoint group), an environment settings object (settings), and an optional URL (url), the following algorithm will create a report, and add it to reporting cache’s queue for future delivery.

1. Let report be a new report object with its values initialized as follows:

   body

   data

   user agent

   The current value of navigator.userAgent

   group

   endpoint group

   type

   type

   timestamp

   The current timestamp.

   attempts

   0

2. If url was not provided by the caller, let url be settings’s creation URL.

3. Set url’s username to the empty string, and its password to null.

4. Set report’s url to the result of executing the URL serializer on url with the exclude fragment flag set.

5. Append report to the reporting cache.

6. Let environment be settings’s realm execution context’s realm’s ECMAScript global environment.

7. Execute §5.2 Notify reporting observers on environment with report with environment and report.

Note: reporting observers can only observe reports from the same environment settings object.

Note: We strip the username, password, and fragment from the serialized URL in the report. See §10.1 Capability URLs.

Note: The user agent MAY reject reports for any reason. This API does not guarantee delivery of arbitrary amounts of data, for instance.

Note: Non user agent clients (with no JavaScript engine) should not interact with reporting observers, and thus should return in step 6.

4.3. Choose an endpoint from a group

Note: This algorithm is the same as the target selection algorithm used for DNS SRV records.

Given an endpoint group (group), this algorithm chooses an arbitrary eligible endpoint from the group, if there is one, taking into account the priority and weight of the endpoints.

1. If group is expired, return null.

   Note: In this case, the user agent MAY remove group from its client, or it may wait and collect garbage en masse at some point in the future as described in §7.2 Garbage Collection.

2. Let endpoints be a copy of group’s endpoints list.

3. Remove every endpoint from endpoints that is pending.

4. If endpoints is empty, return null.

5. Let priority be the minimum priority value of each endpoint in endpoints.

6. Remove every endpoint from endpoints whose priority value is not equal to priority.

7. If endpoints is empty, return null.

8. Let total weight be the sum of the weight value of each endpoint in endpoints.

9. Let weight be a random number ≥ 0 and < total weight.

10. For each endpoint in endpoints:

    1. If weight is less than endpoint’s weight, return endpoint.

    2. Subtract endpoint’s weight from weight.

11. It should not be possible to fall through to here, since the random number chosen earlier will be less than total weight.

4.4. Send reports

A user agent sends reports by executing the following steps:

1. Let reports be a copy of the list of queued report objects in reporting cache.

2. Let endpoint map be an empty map of endpoint objects to lists of report objects.

3. For each report in reports:

   1. Let origin be the origin of report’s url.

   2. Let client be the entry in the reporting cache for origin.

   3. If there exists an endpoint group (group) in client’s endpoint-groups list whose name is report’s group:

      1. Let endpoint be the result of executing §4.3 Choose an endpoint from a group on group.

      2. If endpoint is a not null:

         1. Append report to endpoint map’s list of reports for endpoint.

         2. Skip to the next report.

   4. For each parent origin that is a superdomain match for origin [RFC6797]:

      1. Let client be the entry in the reporting cache for parent origin.

      2. If there exists an endpoint group (group) in client’s endpoint-groups list whose name is report’s group and whose subdomains flag is "include":

         1. Let endpoint be the result of executing §4.3 Choose an endpoint from a group on group.

         2. If endpoint is an endpoint:

            1. Append report to endpoint map’s list of reports for endpoint.

            2. Skip to the next report.

   5. If we reach this step, the report did not match any endpoint and the user agent MAY remove report from the reporting cache directly. Depending on load, the user agent MAY instead wait for §7.2 Garbage Collection at some point in the future.

4. For each (endpoint, reports) pair in endpoint map:

   1. Let origin map be an empty map of origins to lists of report objects.

   2. For each report in reports:

      1. Let origin be the origin of report’s url.

      2. Append report to origin map’s list of reports for origin.

   3. For each (origin, per-origin reports) pair in origin map, execute the following steps asynchronously:

      1. Let result be the result of executing §4.5 Attempt to deliver reports to endpoint on endpoint, origin, and per-origin reports.

      2. If result is "Success":

         1. Set endpoint’s failures to 0, and its retry_after to null.

         2. Remove each report in reports from the reporting cache.

         Otherwise, if result is "Remove Endpoint":

         1. Remove endpoint from the reporting cache.

            Note: reports remain in the reporting cache for potential delivery to other endpoints.

         Otherwise (if result is "Failure"):

         1. Increment endpoint’s failures.

         2. Set endpoint’s retry_after to a point in the future which the user agent chooses.

            Note: We don’t specify a particular algorithm here, but user agents are encouraged to employ some sort of exponential backoff algorithm which increases the retry period with the number of failures, with the addition of some random jitter to ensure that temporary failures don’t lead to a crush of reports all being retried on the same schedule.

            Add in a reasonable reference describing a good algorithm. Wikipedia, if nothing else.

Note: User agents MAY decide to attempt delivery for only a subset of the collected reports or endpoints (because, for example, sending all the reports at once would consume an unreasonable amount of bandwidth, etc). As reports are only removed from the cache when they’re successfully delivered, skipped reports will simply be delivered later.

4.5. Attempt to deliver reports to endpoint

Given an endpoint (endpoint), an origin (origin), and a list of reports (reports), this algorithm will construct a request, and attempt to deliver it to endpoint. It returns "Success" if that delivery succeeds, "Remove Endpoint" if the endpoint explicitly removes itself as a reporting endpoint by sending a 410 response, and "Failure" otherwise.

1. Let collection be a new ECMAScript Array object [ECMA-262].

2. For each report in reports:

   1. Let data be a new ECMAScript Object with the following properties [ECMA-262]:

      age

      The number of milliseconds between report’s timestamp and the current time.

      type

      report’s type

      url

      report’s url

      user_agent

      report’s user agent

      body

      report’s body

      Note: Client clocks are unreliable and subject to skew. We therefore deliver an age attribute rather than an absolute timestamp. See also §11.2 Clock Skew

   2. Increment report’s attempts.

   3. Append data to collection.

3. Let request be a new request with the following properties [FETCH]:

   url

   endpoint’s url

   origin

   origin

   header list

   A new header list containing a header named "Content-Type" whose value is "application/reports+json"

   client

   null

   window

   "no-window"

   skip-service-worker flag

   Set.

   initiator

   ""

   destination

   "report"

   destination

   ""

   mode

   "cors"

   unsafe-request flag

   set

   credentials

   "include"

   body

   The string resulting from executing the JSON.stringify() algorithm on collection [ECMA-262]

4. Queue a task to fetch request.

5. Wait for a response (response).

6. If response’s status is an OK status (200-299), return "Success".

7. If response’s status is 410 Gone [RFC7231], return "Remove Endpoint".

8. Return "Failure".

5. Reporting Observers

A reporting observer observes some types of reports from JavaScript, and is represented in JavaScript by the ReportingObserver object.

Each ECMAScript global environment has a registered reporting observer list, which is an ordered set of reporting observers.

Any reporting observer that is in a registered reporting observer list is considered registered.

Each ECMAScript global environment has a report buffer, which is a list of reports that have been generated in that ECMAScript global environment. This list is initially empty, and the reports are stored in the same order in which they are generated.

Note: The purpose of the report buffer is to allow reporting observers to observe reports that were generated earlier than that observer could be created (via the buffered option). For example, some reports might be generated during an earlier stage of page loading than when an observer could first be created, or before a JavaScript library is loaded that wishes to observe these reports.

Note: Reporting observers are only relevant for user agents with JavaScript engines.

5.1. Interface ReportingObserver

interface ReportBody {
};

interface Report {
  readonly attribute DOMString type;
  readonly attribute DOMString url;
  readonly attribute ReportBody? body;
};

[Constructor(ReportingObserverCallback callback, optional ReportingObserverOptions options)]
interface ReportingObserver {
  void observe();
  void disconnect();
  ReportList takeRecords();
};

callback ReportingObserverCallback = void (sequence<Report> reports, ReportingObserver observer);

dictionary ReportingObserverOptions {
  sequence<DOMString> types;
  boolean buffered = false;
};

typedef sequence<Report> ReportList;

A Report is the application exposed representation of a report. type returns type, url returns url, and body returns body.

Each ReportingObserver object has these associated concepts:

• A callback function set on creation.

• A ReportingObserverOptions dictionary called options.

• A list of Report objects called the report queue, which is initially empty.

A ReportList represents a sequence of Reports, providing developers with all the convenience methods found on JavaScript arrays.

The ReportingObserver(callback, options) constructor, when invoked, must run these steps:

1. Create a new ReportingObserver object observer.

2. Set observer’s callback to callback.

3. Set observer’s options to options.

4. Return observer.

The observe() method, when invoked, must run these steps:

1. Let environment be the ECMAScript global environment associated with the context object.

2. Append the context object to the registered reporting observer list of its associated ECMAScript global environment.

3. If the context object’s buffered option is false, return.

4. Set context object’s buffered option to false.

5. For each report in the report buffer associated with environment, execute §5.3 Add report to observer with report and the context object.

The disconnect() method, when invoked, must run these steps:

1. If the context object is not registered, return.

2. Remove the context object from the registered reporting observer list of its associated ECMAScript global environment.

The takeRecords() method, when invoked, must run these steps:

1. Let reports be a copy of the context object’s report queue.

2. Empty the context object’s report queue.

3. Return reports.

5.2. Notify reporting observers on environment with report

This algorithm makes report’s contents available to any registered reporting observers on the provided ECMAScript global environment environment.

1. For each ReportingObserver observer registered with environment, execute §5.3 Add report to observer on report and observer.

2. Append report to the report buffer associated with environment.

3. If the report buffer now contains more than 100 reports, remove the item at the beginning of the report buffer.

5.3. Add report to observer

Given a report report and a ReportingObserver observer, this algorithm adds report to observer’s report queue, so long as report’s type is observable by observer.

1. If report’s type is not visible to ReportingObservers, return.

2. If observer’s options has a non-empty types member which does not contain report’s type, return.

3. Create a new Report r with type initialized to report’s type, url initialized to report’s url, and body initialized to report’s body.

how to polymorphically initialize body?

3. Append r to observer’s report queue.

4. If the size of observer’s report queue is 1, Queue a task to §5.4 Invoke reporting observers with notify list with a copy of the registered reporting observer list of the ECMAScript global environment associated with observer.

5.4. Invoke reporting observers with notify list

This algorithm invokes observer callback functions for reports of previously observed behavior.

1. For each ReportingObserver observer in notify list:

   1. If observer’s report queue is empty, then continue.

   2. Let reports be a copy of observer’s report queue

   3. Empty observer’s report queue

   4. Invoke observer’s callback with a list of arguments consisting of reports and observer, and observer as the callback this value. If this throws an exception, report the exception.

6. Report Types

6.1. Deprecation

Deprecation reports indicate that a browser API or feature has been used which is expected to stop working in a future update to the browser.

Deprecation reports have the report type "deprecation".

Deprecation reports are visible to ReportingObservers.

interface DeprecationReportBody : ReportBody {
  readonly attribute DOMString id;
  readonly attribute Date? anticipatedRemoval;
  readonly attribute DOMString message;
  readonly attribute DOMString? sourceFile;
  readonly attribute unsigned long? lineNumber;
  readonly attribute unsigned long? columnNumber;
};

A deprecation report’s body, represented in JavaScript by DeprecationReportBody, contains the following fields:

• id: an implementation-defined string identifying the feature or API that will be removed. This string can be used for grouping and counting related reports.

• anticipatedRemoval: A JavaScript Date object (rendered as an ISO 8601 string) indicating roughly when the browser version without the specified API will be generally available (excluding "beta" or other pre-release channels). This value should be used to sort or prioritize warnings. If unknown, this field should be null, and the deprecation should be considered low priority (removal may not actually occur).

• message: A human-readable string with details typically matching what would be displayed on the developer console. The message is not guaranteed to be unique for a given id (eg. it may contain additional context on how the API was used).

• sourceFile: If known, the file which first used the indicated API, or null otherwise.

• lineNumber: If known, the line number in sourceFile where the indicated API was first used, or null otherwise.

• columnNumber: If known, the column number in sourceFile where the indicated API was first used, or null otherwise.

{
  "type": "deprecation",
  "age": 27,
  "url": "https://example.com/",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0",
  "body": {
    "id": "websql",
    "anticipatedRemoval": "1/1/2020",
    "message": "WebSQL is deprecated and will be removed in Chrome 97 around January 2020",
    "sourceFile": "https://example.com/index.js",
    "lineNumber": 1234,
    "columnNumber": 42
  }
}

6.2. Intervention

Intervention reports indicate that a user agent has decided not to honor a request made by the application (e.g. for security, performance or user annoyance reasons). Note that the report properties are the same as those for deprecation reports, except for the absence of anticipatedRemoval.

Intervention reports have the report type "intervention".

Intervention reports are visible to ReportingObservers.

interface InterventionReportBody : ReportBody {
  readonly attribute DOMString id;
  readonly attribute DOMString message;
  readonly attribute DOMString? sourceFile;
  readonly attribute unsigned long? lineNumber;
  readonly attribute unsigned long? columnNumber;
};

An intervention report’s body, represented in JavaScript by InterventionReportBody, contains the following fields:

• id: an implementation-defined string identifying the specific intervention that occurred. This string can be used for grouping and counting related reports.

• message: A human-readable string with details typically matching what would be displayed on the developer console. The message is not guaranteed to be unique for a given id (e.g. it may contain additional context on what led to the intervention).

• sourceFile: If known, the file which first used the indicated API, or null otherwise.

• lineNumber: If known, the line number in sourceFile of the offending behavior (which prompted the intervention), or null otherwise.

• columnNumber: If known, the column number in sourceFile of the offending behavior (which prompted the intervention), or null otherwise.

{
  "type": "intervention",
  "age": 27,
  "url": "https://example.com/",
  "body": {
    "id": "audio-no-gesture",
    "message": "A request to play audio was blocked because it was not triggered by user activation (such as a click).",
    "sourceFile": "https://example.com/index.js",
    "lineNumber": 1234,
    "columnNumber": 42
  }
}

6.3. Crash

Crash reports indicate that the user was unable to continue using the page because the browser (or one of its processes necessary for the page) crashed. For security reasons, no details of the crash are communicated except for a unique identifier (which can be interpreted by the browser vendor), and optionally the reason for the crash (such as "oom").

Crash reports have the report type "crash".

Crash reports are visible to ReportingObservers.

interface CrashReportBody : ReportBody {
  readonly attribute DOMString crashId;
  readonly attribute DOMString? reason;
};

An crash report’s body, represented in JavaScript by CrashReportBody, contains the following fields:

• crashId: An implementation-defined unique identifier (100 character maximum string). This identifier will not be meaningful to developers directly, but it can potentially be supplied to the browser vendor for more details.

• reason: A more specific classification of the type of crash that occured. Currently, the only valid reason is "oom" (omitted otherwise), indicating an out-of-memory crash.

{
  "type": "crash",
  "age": 42,
  "url": "https://example.com/",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0",
  "body": {
    "crashId": "30437694edfeae5b",
    "reason": "oom"
  }
}

7. Implementation Considerations

7.1. Delivery

The user agent SHOULD attempt to deliver reports as soon as possible to provide feedback to developers as quickly as possible. However, when this desire is balanced against the impact on the user, the user wins. With that in mind, the user agent MAY delay delivery of reports based on its knowledge of the user’s activities and context.

For instance, the user agent SHOULD prioritize the transmission of reporting data lower than other network traffic. The user’s explicit activities on a website should preempt reporting traffic.

The user agent MAY choose to withhold report delivery entirely until the user is on a fast, cheap network in order to prevent unnecessary data cost.

The user agent MAY choose to prioritize reports from particular origins over others (perhaps those that the user visits most often?)

7.2. Garbage Collection

Periodically, the user agent SHOULD walk through the cached reports and endpoints, and discard those that are no longer relevant. These include:

• endpoint groups which are expired

• endpoint groups which have not been used in some arbitrary period of time (perhaps a ~week?)

• endpoints whose failures exceed some user-agent-defined threshold (~5 seems reasonable)

• reports whose attempts exceed some user-agent-defined threshold (~5 seems reasonable)

• reports which have not been delivered in some arbitrary period of time (perhaps ~2 days?)

For any reports that are discarded, these reports should also be removed from the report buffer of any reporting observer.

8. Sample Reports

POST / HTTP/1.1
Host: example.com
...
Content-Type: application/reports+json

[{
  "type": "csp",
  "age": 10,
  "url": "https://example.com/vulnerable-page/",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0",
  "body": {
    "blocked": "https://evil.com/evil.js",
    "directive": "script-src",
    "policy": "script-src 'self'; object-src 'none'",
    "status": 200,
    "referrer": "https://evil.com/"
  }
}, {
  "type": "hpkp",
  "age": 32,
  "url": "https://www.example.com/",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0",
  "body": {
    "date-time": "2014-04-06T13:00:50Z",
    "hostname": "www.example.com",
    "port": 443,
    "effective-expiration-date": "2014-05-01T12:40:50Z"
    "include-subdomains": false,
    "served-certificate-chain": [
      "-----BEGIN CERTIFICATE-----\n
      MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT\n
      ...
      HFa9llF7b1cq26KqltyMdMKVvvBulRP/F/A8rLIQjcxz++iPAsbw+zOzlTvjwsto\n
      WHPbqCRiOwY1nQ2pM714A5AuTHhdUDqB1O6gyHA43LL5Z/qHQF1hwFGPa4NrzQU6\n
      yuGnBXj8ytqU0CwIPX4WecigUCAkVDNx\n
      -----END CERTIFICATE-----",
      ...
    ]
  }
}, {
  "type": "nel",
  "age": 29,
  "url": "https://example.com/thing.js",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0",
  "body": {
    "referrer": "https://www.example.com/",
    "server-ip": "234.233.232.231",
    "protocol": "",
    "status-code": 0,
    "elapsed-time": 143,
    "age": 0,
    "type": "http.dns.name_not_resolved"
  }
}]

9. Automation

For the purposes of user-agent automation and application testing, this document defines a number of extension commands for the [WebDriver] specification.

9.1. Generate Test Report

The Generate Test Report extension command simulates the generation of a report for the purposes of testing. This report will be observed by any registered reporting observers.

The extension command is defined as follows:

dictionary GenerateTestReportParameters {
  required DOMString message;
  DOMString group;
};

The remote end steps are:

1. If parameters is not a JSON Object, return a WebDriver error with WebDriver error code invalid argument.

2. Let message be the result of trying to get parameters’s message property.

3. If message is null, return a WebDriver error with WebDriver error code invalid argument.

4. Let group be the result of trying to get parameters’s group property.

5. If group is null, set group to "default".

6. Let body be a new object that can be serialized into a JSON text, containing a single string field, body_message.

7. Set body_message to message.

8. Let settings be the environment settings object of the current browsing context’s active document.

9. Execute §4.2 Queue data as type for endpoint group on settings with body, "test", group, and settings.

10. Return success with data null.

10. Security Considerations

10.1. Capability URLs

Some URLs are valuable in and of themselves. To mitigate the possibility that such URLs will be leaked via this reporting mechanism, we strip out credential information and fragment data from the URL we store as a report’s originator. It is still possible, however, for a feature to unintentionally leak such data via a report’s body. Implementers SHOULD ensure that URLs contained in a report’s body are similarly stripped.

11. Privacy Considerations

11.1. Network Leakage

Because this reporting mechanism is out-of-band, and doesn’t rely on a page being open, it’s entirely possible for a report generated while a user is on one network to be sent while the user is on another network, even if they don’t explicitly open the page from which the report was sent.

Consider mitigations. For example, we could drop reports if we change from one network to another. <https://github.com/w3c/BackgroundSync/issues/107>

11.2. Clock Skew

Each report is delivered along with an age property, rather than the timestamp at which it was generated. We do this because each user’s local clock will be skewed from the clock on the server by an arbitrary amount. The difference between the time the report was generated and the time it was sent will be stable, regardless of clock skew, and we can avoid the fingerprinting risk of exposing the clock skew via this API.

11.3. Cross-origin correlation

If multiple origins all use the same reporting endpoint, that endpoint may learn that a particular user has interacted with a certain set of websites, as it will receive origin-tagged reports from each. This doesn’t seem worse than the status quo ability to track the same information from cooperative origins, and doesn’t grant any new tracking ability above and beyond what’s possible with <img> today.

11.4. Subdomains

This specification allows any resource on a host to declare a set of reporting endpoints for that host and each of its subdomains. This doesn’t have privacy implications in and of itself (beyond those noted in §11.5 Clearing the reporting cache), as the reporting endpoints themselves don’t take any real action, as features will need to opt-into using these reporting endpoints explicitly. Those features certainly will have privacy implications, and should carefully consider whether they should be enabled across origin boundaries.

11.5. Clearing the reporting cache

A user agent’s reporting cache contains data about a user’s activity on the web, and user agents ought to handle this data carefully. In particular, if a user agent gives users the ability to clear their site data, browsing history, browsing cache, or similar, the user agent MUST also clear the reporting cache. Note that this includes both the pending reports themselves, as well as the endpoints to which they would be sent. Both MUST be cleared.

11.6. Disabling Reporting

Reporting is, to some extent, a question of commons. In the aggregate, it seems useful for everyone for reports to be delivered. There is direct benefit to developers, as they can fix bugs, which means there’s indirect benefit to users, as the sites they enjoy will be more stable and enjoyable. As a concrete example, Content Security Policy grants something like herd immunity to cross-site scripting attacks by alerting developers about potential holes in their sites' defenses. Fixing those bugs helps every user, even those whose user agents don’t support Content Security Policy.

The calculus, of course, depends on the nature of data that’s being delivered, and the relative maliciousness of the reporting endpoints, but that’s the value proposition in broad strokes.

That said, it can’t be the case that this general benefit be allowed to take priority over the ability of a user to individually opt-out of such a system. Sending reports costs bandwidth, and potentially could reveal some small amount of additional information above and beyond what a website can obtain in-band ([NETWORK-ERROR-LOGGING], for instance). User agents MUST allow users to disable reporting with some reasonable amount of granularity in order to maintain the priority of constituencies espoused in [HTML-DESIGN-PRINCIPLES].

12. IANA Considerations

12.1. The Report-To Header

The permanent message header field registry should be updated with the following registration: [RFC3864]

Header field name

Report-To

Applicable protocol

http

Status

standard

Author/Change controller

W3C

Specification document

This specification (see §3.1 The Report-To HTTP Response Header Field)

12.2. The application/reports+json Media Type

Type name

application

Subtype name

reports+json

Required parameters

N/A

Optional parameters

N/A

Encoding considerations

Encoding considerations are identical to those specified for the "application/json" media type. See [RFC8259].

Security considerations

See §10 Security Considerations.

Interoperability considerations

This document specifies the format of conforming messages and the interpretation thereof.

Published specification

§4.1 Media Type

Applications that use this media type

Fragment identifier considerations

Additional information

N/A

Person and email address to contact for further information

This document’s editors.

Intended usage:

COMMON

Restrictions on usage:

N/A

Author

This document’s editors.

Change controller

W3C

Provisional registration?

Yes.