Trace Context Level 2

Distributed tracing is a methodology implemented by tracing tools to follow, analyze and debug a transaction across multiple software components. Typically, a distributed trace traverses more than one component which requires it to be uniquely identifiable across all participating systems. Trace context propagation passes along this unique identification. Today, trace context propagation is implemented individually by each tracing vendor. In multi-vendor environments, this causes interoperability problems, like:

• Traces that are collected by different tracing vendors cannot be correlated as there is no shared unique identifier.
• Traces that cross boundaries between different tracing vendors can not be propagated as there is no uniformly agreed set of identification that is forwarded.
• Vendor specific metadata might be dropped by intermediaries.
• Cloud platform vendors, intermediaries and service providers, cannot guarantee to support trace context propagation as there is no standard to follow.

In the past, these problems did not have a significant impact as most applications were monitored by a single tracing vendor and stayed within the boundaries of a single platform provider. Today, an increasing number of applications are highly distributed and leverage multiple middleware services and cloud platforms.

This transformation of modern applications calls for a distributed tracing context propagation standard.

The trace context specification defines a universally agreed-upon format for the exchange of trace context propagation data - referred to as trace context. Trace context solves the problems described above by

• providing a unique identifier for individual traces and requests, allowing trace data of multiple providers to be linked together.
• providing an agreed-upon mechanism to forward vendor-specific trace data and avoid broken traces when multiple tracing tools participate in a single transaction.
• providing an industry standard that intermediaries, platforms, and hardware providers can support.

A unified approach for propagating trace data improves visibility into the behavior of distributed applications, facilitating problem and performance analysis. The interoperability provided by trace context is a prerequisite to manage modern micro-service based applications.

The current version of the Trace Context specification is targeted for implementations by applications and services, including web applications running within a browser. Web Browsers or User Agents are not currently in scope as a target implementation.

Trace context is split into two individual propagation fields supporting interoperability and vendor-specific extensibility:

• traceparent describes the position of the incoming request in its trace graph in a portable, fixed-length format. Its design focuses on fast parsing. Every tracing tool MUST properly set traceparent even when it only relies on vendor-specific information in tracestate
• tracestate extends traceparent with vendor-specific data represented by a set of name/value pairs. Storing information in tracestate is optional.

Tracing tools can provide two levels of compliant behavior interacting with trace context:

• At a minimum they MUST propagate the traceparent and tracestate headers and guarantee traces are not broken. This behavior is also referred to as forwarding a trace.
• In addition they MAY also choose to participate in a trace by modifying the traceparent header and relevant parts of the tracestate header containing their proprietary information. This is also referred to as participating in a trace.

A tracing tool can choose to change this behavior for each individual request to a component it is monitoring.

The traceparent request header represents the incoming request in a tracing system in a common format, understood by all vendors. Here’s an example of a traceparent header.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

The tracestate request header includes the parent in a potentially vendor-specific format:

tracestate: congo=t61rcWkgMzE

For example, say a client and server in a system use different tracing vendors: Congo and Rojo. A client traced in the Congo system adds the following headers to an outbound HTTP request.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=t61rcWkgMzE

Note: In this case, the tracestate value t61rcWkgMzE is the result of Base64 encoding the parent ID (b7ad6b7169203331), though such manipulations are not required.

The receiving server, traced in the Rojo tracing system, carries over the tracestate it received and adds a new entry to the left.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01
tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE

You'll notice that the Rojo system reuses the value of its traceparent for its entry in tracestate. This means it is a generic tracing system (no proprietary information is being passed). Otherwise, tracestate entries are opaque and can be vendor-specific.

If the next receiving server uses Congo, it carries over the tracestate from Rojo and adds a new entry for the parent to the left of the previous entry.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
tracestate: congo=ucfJifl5GOE,rojo=00f067aa0ba902b7

Note: ucfJifl5GOE is the Base64 encoded parent ID b9c7c989f97918e1.

Notice when Congo wrote its traceparent entry, it is not encoded, which helps in consistency for those doing correlation. However, the value of its entry tracestate is encoded and different from traceparent. This is ok.

Finally, you'll see tracestate retains an entry for Rojo exactly as it was, except pushed to the right. The left-most position lets the next server know which tracing system corresponds with traceparent. In this case, since Congo wrote traceparent, its tracestate entry should be left-most.

The traceparent HTTP header field identifies the incoming request in a tracing system. It has four fields:

• version
• trace-id
• parent-id
• trace-flags

HEXDIGLC = DIGIT / "a" / "b" / "c" / "d" / "e" / "f" ; lowercase hex character
value           = version "-" version-format

version         = 2HEXDIGLC   ; this document assumes version 00. Version ff is forbidden

version-format   = trace-id "-" parent-id "-" trace-flags
trace-id         = 32HEXDIGLC  ; 16 bytes array identifier. All zeroes forbidden
parent-id        = 16HEXDIGLC  ; 8 bytes array identifier. All zeroes forbidden
trace-flags      = 2HEXDIGLC   ; 8 bit flags.

static final byte FLAG_SAMPLED = 1; // 00000001
static final byte FLAG_RANDOM = 2; // 00000010
...
boolean sampled = (traceFlags & FLAG_SAMPLED) == FLAG_SAMPLED;
boolean random = (traceFlags & FLAG_RANDOM) == FLAG_RANDOM;

Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
base16(version) = 00
base16(trace-id) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(parent-id) = 00f067aa0ba902b7
base16(trace-flags) = 01  // sampled

Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00
base16(version) = 00
base16(trace-id) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(parent-id) = 00f067aa0ba902b7
base16(trace-flags) = 00  // not sampled

The main purpose of the tracestate HTTP header is to provide additional vendor-specific trace identification information across different distributed tracing systems and is a companion header for the traceparent field. It also conveys information about the request’s position in multiple distributed tracing graphs.

If the vendor failed to parse traceparent, it MUST NOT attempt to parse tracestate. Note that the opposite is not true: failure to parse tracestate MUST NOT affect the parsing of traceparent.

The tracestate HTTP header MUST NOT be used for any properties that are not defined by a tracing system. [BAGGAGE] MAY be used for defining and propagating such application level properties.

list  = list-member 0*31( OWS "," OWS list-member )
list-member = (key "=" value) / OWS

key = ( lcalpha / DIGIT ) 0*255 ( keychar )
keychar    = lcalpha / DIGIT / "_" / "-"/ "*" / "/" / "@"
lcalpha    = %x61-7A ; a-z

value    = 0*255(chr) nblk-chr
nblk-chr = %x21-2B / %x2D-3C / %x3E-7E
chr      = %x20 / nblk-chr

tracestate: rojo=00f067aa0ba902b7

tracestate: rojo=00f067aa0ba902b7,congo=t61rcWkgMzE

A vendor receiving a request without a traceparent header SHOULD generate traceparent headers for outbound requests, effectively starting a new trace. A possible reason for not doing this could be a performance sensitive scenario when the vendor decides to not sample a request. Note that for most scenarios, vendors are expected to generate the header even when not sampling, to propagate the sampling decision downstream.

A vendor receiving a traceparent request header MUST send it to outgoing requests. It MAY mutate the value of this header before passing it to outgoing requests.

If the value of the traceparent field wasn't changed before propagation, tracestate MUST NOT be modified as well. Unmodified header propagation is typically implemented in pass-through services like proxies. This behavior may also be implemented in a service which currently does not collect distributed tracing information.

Following is the list of allowed mutations:

• Update parent-id: The value of the parent-id field can be set to the new value representing the ID of the current operation. This is the most typical mutation and should be considered a default.
• Update sampled: The value of the sampled field reflects the caller's recording behavior: either trace data was dropped or may have been recorded out-of-band. This can be indicated by toggling the flag in both directions. This mutation gives the downstream vendor information about the likelihood that its parent's information was recorded. The parent-id field MUST be set to a new value with the sampled flag update.
• Restart trace: All properties (trace-id, parent-id, trace-flags) are regenerated. This mutation is used in services that are defined as a front gate into secure networks and eliminates a potential denial-of-service attack surface. Vendors SHOULD clean up tracestate collection on traceparent restart. There are rare cases when the original tracestate entries must be preserved after a restart. This typically happens when the trace-id is reverted back at some point of the trace flow, for instance, when it leaves the secure network. However, it SHOULD be an explicit decision, and not the default behavior.
• Downgrade the version: This version of the specification (00) defines the behavior for a vendor that receives a traceparent header of a higher version. In this case, the first mutation is to downgrade the version of the header. Other mutations are allowed in combination with this one.

Vendors MUST NOT make any other mutations to the traceparent header.

Vendors receiving a tracestate request header MUST send it to outgoing requests. It MAY mutate the value of this header before passing to outgoing requests. When mutating tracestate, the order of unmodified key/value pairs MUST be preserved. Modified keys MUST be moved to the beginning (left) of the list.

Following are allowed mutations:

• Add a new key/value pair. The new key/value pair SHOULD be added to the beginning of the list. Adding a key/value pair MUST NOT result in the same key being present multiple times.
• Update an existing value. The value for any given key can be updated. Modified keys SHOULD be moved to the beginning (left) of the list.
• Delete a key/value pair. Any key/value pair MAY be deleted. Vendors SHOULD NOT delete keys that were not generated by them. The deletion of an unknown key/value pair will break correlation in other systems. This mutation enables three scenarios. The first is that proxies can block certain tracestate keys for privacy and security concerns. The second scenario is a truncation of long tracestates. Finally, vendors MAY also discard duplicate keys that were not generated by them.

The traceparent field MUST NOT contain any personally identifiable information. One way to achieve this is to randomly generate all trace IDs using a random number generator that does not expose any personally identifiable information. Any random number generator used for generating trace IDs MUST NOT rely on any information as input or seed state that can potentially be personally identifiable.

Another privacy risk of the traceparent field is the ability to correlate requests made as part of a single transaction. A downstream service may track and correlate two or more requests made in a single transaction and may make assumptions about the identity of the caller of a request based on information from another request.

Note that these privacy concerns of the traceparent field are theoretical rather than practical. Some services initiating or receiving a request MAY choose to restart a traceparent field to eliminate those risks completely. Vendors SHOULD find a way to minimize the number of distributed trace restarts to promote interoperability of tracing vendors. Instead of restarts, different techniques may be used. For example, services may define trust boundaries of upstream and downstream connections and the level of exposure that any requests may bring. For instance, a vendor might only restart traceparent for authentication requests from or to external services.

Services may also define an algorithm and audit mechanism to validate the randomness of incoming or outgoing random numbers in the traceparent field. Note that this algorithm is services-specific and not a part of this specification. One example might be a temporal algorithm where a reversible hash function is applied to the current clock time. The receiver can validate that the time is within agreed upon boundaries, meaning the random number was generated with the required algorithm and in fact doesn't contain any personally identifiable information.

The tracestate field may contain any opaque value in any of the keys. The main purpose of this header is to provide additional vendor-specific trace-identification information across different distributed tracing systems.

Vendors MUST NOT include any personally identifiable information in the tracestate header.

Vendors extremely sensitive to personal information exposure MAY implement selective removal of values corresponding to the unknown keys. Vendors SHOULD NOT mutate the tracestate field, as it defeats the purpose of allowing multiple tracing systems to collaborate.

When vendors include traceparent and tracestate headers in responses, these values may inadvertently be passed to cross-origin callers. Vendors should ensure that they include only these response headers when responding to systems that participated in the trace.

As mentioned in the privacy section, information in the traceparent and tracestate headers may carry information that can be considered sensitive. For example, traceparent may allow one request to be correlated to the data sent with another request, or the tracestate header may imply the version of monitoring software used by the caller. This information could potentially be used to create a larger attack.

Application owners should either ensure that no proprietary or confidential information is stored in tracestate, or they should ensure that tracestate isn't present in requests to external systems.

When distributed tracing is enabled on a service with a public API and naively continues any trace with the sampled flag set, a malicious attacker could overwhelm an application with tracing overhead, forge trace-id collisions that make monitoring data unusable, or run up your tracing bill with your SaaS tracing vendor.

Tracing vendors and platforms should account for these situations and make sure that checks and balances are in place to protect denial of monitoring by malicious or badly authored callers.

One example of such protection may be different tracing behavior for authenticated and unauthenticated requests. Various rate limiters for data recording can also be implemented.

Application owners need to make sure to test all code paths leading to the sending of traceparent and tracestate headers. For example, in single page browser applications, it is typical to make cross-origin requests. If one of these code paths leads to traceparent and tracestate headers being sent by cross-origin calls that are restricted using Access-Control-Allow-Headers [FETCH], it may fail.