Copyright © 2018 W3C® (MIT, ERCIM, Keio, Beihang). W3C liability, trademark and permissive document license rules apply.
Distributed tracing is a set of tools and practices to monitor the health and reliability of a distributed application. A distributed application is an application that consists of multiple components that are deployed and operated separately. It is also known as micro-service.
The main concept behind distributed tracing is event correlation. Event correlation is a way to correlate events from one component to the events from another. It allows to find the cause-and-effect relationship between these events. For instance – find which user action in a browser caused a failure in the business logic layer.
To correlate events between components, these components need to exchange and store a piece of information called context. Typically context consists of an originating event identifier, an originating component identity and other event properties. Context has two parts. The first part is a trace context. Trace context consists of properties crucial for event correlation. The second part is correlation context. Correlation context carries user-defined properties. These properties may be helpful for correlation scenarios. But they are not required and components may choose to not carry or store them.
Unifying the format of distributed tracing context as well as aligning on semantic meaning of the values is the main objective of this working group. The goal is to share this with the community so that various tracing and diagnostics products can operate together.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at https://www.w3.org/TR/.
This is the First Public Working Draft (FPWD). There are a few implementations of this protocol available. Experimental interoperability scenarios were run and have demonstrated promising results. The specification will be progressed into Candidate Recommendation stage after that, drafts for binary, AMQP and MQTT protocols will be written to make sure the concepts and structure defined in this specification can be ported to other protocols.
This document was published by the Distributed Tracing Working Group as a Working Draft. This document is intended to become a W3C Recommendation.
GitHub
Issues
are preferred for
discussion of this
specification.
Alternatively,
you can
send
comments
to our
mailing
list.
Please send them
to
public-trace-context@w3.org
(archives)
with
trace-context
at the
start of
your
email's
subject
.
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document was produced by a group operating under the W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
This document is governed by the 1 February 2018 W3C Process Document.
Trace context consists of a set of properties crucial for event correlation in distributed applications. Trace context positions requests in a distributed trace. If you look at a distributed trace as a graph of correlated requests - a single request can be part of multiple graphs. For example, let's say component A
calls component B
which in turn calls C
. One distributed tracing system may see component A
calling component B
. Another may see all three components communicating. So the call from B
to C
will carry trace context about service A
as well as information about its direct communication upstream to B
.
The situation when a single request is a part of multiple graphs is becoming more common. One distributed application may have components monitored by different distributed tracing systems. Those distributed tracing systems may not be easily replaceable as they might be provided by cloud vendors or distributed as pre-built components.
The trace context is designed to allow extensibility for all distributed tracing systems. It requires them to respect context set by other systems.
Trace context is represented by a set of name/value pairs describing the identity of every http request. Two propagation fields carry the common and vendor-specific properties that make up the trace context.
traceparent
describes the position of the incoming request in its trace graph in a portable, fixed-length format. Its design focuses on fast parsing.tracestate
maps all graphs the incoming parent is a part of in potentially vendor-specific formats. For example, if a request crosses tracing systems, there will be one entry in tracestate
for each system.Notably, the tracestate
field is unreliant on data in the traceparent
.
Libraries and platforms MUST propagate traceparent
and tracestate
headers to guarantee that trace will not be broken.
This section describes the binding of the distributed trace context to traceparent
and tracestate
http headers.
The traceparent
header represents the incoming request in a tracing system in
a common format. The tracestate
header includes the parent in a potentially
vendor-specific format.
For example, a client traced in the congo system adds the following headers to an outbound http request.
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=BleGNlZWRzIHRohbCBwbGVhc3VyZS4
If the receiving server is traced in the rojo
tracing system, it carries
over the state it received and adds a new entry with the position in
its trace.
traceparent: 00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01
tracestate: rojo=00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01,congo=BleGNlZWRzIHRohbCBwbGVhc3VyZS4
You'll notice that the rojo
system reuses the value of traceparent
in its
entry in tracestate
. This means it is a generic tracing system. Otherwise,
tracestate
entries are opaque.
If the receiving server of the above is congo
again, it continues from its
last position, overwriting its entry with one representing the new parent.
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
tracestate: congo=lZWRzIHRoNhcm5hbCBwbGVhc3VyZS4,rojo=00-0af7651916cd43dd8448eb211c80319c-00f067aa0ba902b7-01
Notice when congo
wrote its traceparent
entry, it reuses the last trace ID
which helps in consistency for those doing correlation. However, the value of
its entry tracestate
is opaque and different. This is ok.
Finally, you'll see tracestate
retains an entry for rojo
exactly as it was,
except pushed to the right. The left-most position lets the next server know
which tracing system corresponds with traceparent
. In this case, since
congo
wrote traceparent
, its tracestate
entry should be left-most.
Field traceparent
identifies the request in a tracing system.
In order to increase interoperability across multiple protocols and encourage successful integration by default it is recommended to keep the header name lower case. Header name is a single word without any delimiters like hyphen (-
).
Header name: traceparent
Platforms and libraries MUST expect header name in any casing and SHOULD send header name in lower case.
This section uses the Augmented Backus-Naur Form (ABNF) notation of RFC5234, including the HEXDIG rules from that document.
value = version "-" version-format
version = 2HEXDIG ; this document assumes version 00. Version 255 is forbidden
The value is US-ASCII encoded (which is UTF-8 compliant). Character -
is
used as a delimiter between fields.
Version (version
) is a 1 byte representing an 8-bit unsigned integer. Version 255 is invalid. Current specification assumes the version
is set to 00
.
The following version-format
definition is used for version 00
.
version-format = trace-id "-" span-id "-" trace-flags
trace-id = 32HEXDIG ; 16 bytes array identifier. All zeroes forbidden
span-id = 16HEXDIG ; 8 bytes array identifier. All zeroes forbidden
trace-flags = 2HEXDIG ; 8 bit flags. Currently only one bit is used. See below for details
Is the ID of the whole trace forest. It is represented as a 16-bytes array, for example,
4bf92f3577b34da6a3ce929d0e0e4736
. All bytes 0
are considered invalid.
Trace-id
is used to uniquely identify a distributed trace. So implementation should generate globally unique
values. Many algorithms of unique identification generation are based on some constant part - time or host
based and a random values. There are systems that make random sampling decisions based on the value of trace-id
.
So to increase interoperability it is recommended to keep the random part on the right side of trace-id
value.
When a system operates with a shorter trace-id
- it is recommended to fill-in the extra bytes with random values rather
than zeroes. Let's say the system works with a 8-byte trace-id
like 3ce929d0e0e4736
. Instead of setting trace-id
value to 0000000000000003ce929d0e0e4736
it is recommended to generate a value like
4bf92f3577b34da6a3ce929d0e0e4736
where 4bf92f3577b34da6a
is a random value or a function of time & host value.
Note, even though a system may operate with a shorter trace-id
for distributed trace reporting - full trace-id
should
be propagated to conform to the specification.
Implementation HAVE TO ignore the traceparent
when the trace-id
is invalid. For instance, if it contains
non-allowed characters.
Is the ID of the caller span (parent). It is represented as an 8-byte array, for example,
00f067aa0ba902b7
. All bytes 0
is considered invalid.
Implementation HAVE TO ignore the traceparent
when the span-id
is invalid. For instance, if it contains
non-allowed characters.
An 8-bit field that controls tracing flags such as sampling, trace level etc. These flags are recommendations given by the caller rather than strict rules to follow for three reasons:
You can find more in security section of this specification.
Like other fields, trace-flags
is hex-encoded. For example, all 8
flags set
would be ff
and no flags set would be 00
.
As this is a bit field, you cannot interpret flags by decoding the hex value and looking at
the resulting number. For example, a flag 00000001
could be encoded as 01
in hex, or 09
in hex if present with the flag 00001000
. A common mistake in bit fields is forgetting to
mask when interpreting flags.
Here is an example of properly handing trace flags:
static final byte FLAG_RECORDED = 1; // 00000001
...
boolean recorded = (traceFlags & FLAG_RECORDED) == FLAG_RECORDED
Current version of specification only supports a single flag called recorded
.
When set, the least significant bit documents that the caller may have recorded trace data. A caller who does not record trace data out-of-band leaves this flag unset.
Many distributed tracing scenarios may be broken when only a subset of calls participated in a distributed trace were recorded. At certain load recording information about every incoming and outgoing request become prohibitively expensive. Making a random or component-specific decision for data collection will lead to fragmented data in every distributed trace. Thus it is typical for tracing vendors and platforms to pass recording decision for given distributed trace or information needed to make this decision.
There is no consensus on what is the best algorithm to make a recording decision. Various techniques include: probability sampling (sample 1 out of 100 distributed traced by flipping a coin), delayed decision (make collection decision based on duration or a result of a call), deferred sampling (let callee decide whether information about this request need to be collected). There are variations and customizations of every technique which can be tracing vendor specific or application defined.
Field tracestate
is designed to handle the variety of techniques for making
recording decision specific (along any other specific information) for a given
tracing system or a platform. Flag recorded
is introduced for better
interoperability between vendors. It allows to communicate recording decision
and enable better experience for the customer.
For example, when SaaS services participate in distributed trace - this service
has no knowledge of tracing system used by it's caller. But this service may
produce records of incoming requests for monitoring or troubleshooting purposes.
Flag recorded
can be used to ensure that information about requests that were
marked for recording by caller will also be recorded by SaaS service. So caller
can troubleshoot the behavior of every recorded request.
Flag recorded
has no restriction on it's mutations except that it can only be
mutated when span-id
was updated. See section "Mutating the traceparent
field". However there are set of suggestions that will increase vendors
interoperability.
recorded
flag.recorded
flag value. Security considerations should be applied to protect from
abusive or malicious use of this flag - see security section.recorded
should be propagated unchanged. And set to 0
as a default option when trace is initiated by this component. There are two
additional options:recorded
flag to 1
for a subset of
requests.recorded
to 1
for the subset of requests.The behavior of other flags, such as (00000100
) is not defined and reserved for
future use. Implementation MUST set those to zero.
Valid traceparent when caller recorded this request:
Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
base16(<version>) = 00
base16(<traceid>) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(<spanid>) = 00f067aa0ba902b7
base16(<traceflags>) = 01 // recorded
Valid traceparent when caller haven't recorded this request:
Value = 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00
base16(<version>) = 00
base16(<traceid>) = 4bf92f3577b34da6a3ce929d0e0e4736
base16(<spanid>) = 00f067aa0ba902b7
base16(<traceflags>) = 00 // not recorded
traceparent
Implementation is opinionated about future version of the specification. Current version of this specification assumes that the future
versions of traceparent
header will be additive to the current one.
Implementation should follow the following rules when parsing headers with an unexpected format:
-
)), implementation should restart the trace.trace-id
: from the first dash - next 32 characters.
Implementation MUST check 32 characters to be hex. Make sure they
are followed by dash.span-id
: from the second dash at 35th position - 16
characters. Implementation MUST check 16 characters to be hex.
Make sure this is followed by a dash.flags
: 2 characters from third dash.
Following with either end of string or a dash. If all three
values were parsed successfully - implementation should use them.
Implementation MUST NOT parse or assume anything about any fields
unknown for this version. Implementation MUST use these fields to
construct the new traceparent
field according to the highest
version of the specification known to the implementation (in this
specification it is 00
).The tracestate
HTTP header field conveys information about request
position in multiple distributed tracing graphs. This header is a
companion header for the traceparent
. If library or platform failed to
parse traceparent
- it MUST NOT attempt to parse the tracestate
.
Note, that opposite it not true - failure to parse tracestate
MUST NOT
affect the parsing of traceparent
.
In order to increase interoperability across multiple protocols and encourage successful integration by default it is recommended to keep the header name lower case. Header name is a single word without any delimiters like hyphen (-
).
Header name: tracestate
Platforms and libraries MUST expect header name in any casing and SHOULD send header name in lower case.
Multiple tracestate
headers are allowed. Values from multiple headers in
incoming requests SHOULD be combined in a single header according to the
RFC7230 and send as a single
header in outgoing request.
This section uses the Augmented Backus-Naur Form (ABNF) notation of RFC5234, including the DIGIT rule in appendix B.1 for RFC5234. It also includes the OWS rule from RFC7230 section 3.2.3.
DIGIT
rule defines number 0
-9
.
The OWS
rule defines an optional whitespace. It is used where zero or more whitespace characters might appear. When it is preferred to improve readability - a sender SHOULD generate the optional whitespace as a single space; otherwise, a sender SHOULD NOT generate optional whitespace. See details in corresponding RFC.
The tracestate
field value is a list
as defined below. The list
is a series of list-members
separated by commas ,
, and a list-member
is a key/value pair separated by an equals sign =
. Spaces and horizontal tabs surrounding list-member
s are ignored. There can be a maximum of 32 list-member
s in a list
.
Empty and whitespace-only list members are allowed. Libraries and
platforms MUST accept empty tracestate
headers, but SHOULD avoid
sending them. The reason for allowing of empty list members in
tracestate
is a difficulty for implementor to recognize the empty
value when multiple tracestate
headers were sent. Whitespace
characters are allowed for a similar reason as some frameworks will
inject whitespace after ,
separator automatically even in case of an
empty header.
A simple example of a list
with two list-member
s might look like: vendorname1=opaqueValue1,vendorname2=opaqueValue2
.
list = list-member 0*31( OWS "," OWS list-member )
list-member = key "=" value
list-member = OWS
Identifiers are short (up to 256 characters) textual identifiers.
key = lcalpha 0*255( lcalpha / DIGIT / "_" / "-"/ "*" / "/" )
key = lcalpha 0*240( lcalpha / DIGIT / "_" / "-"/ "*" / "/" ) "@" lcalpha 0*13( lcalpha / DIGIT / "_" / "-"/ "*" / "/" )
lcalpha = %x61-7A ; a-z
Note that identifiers MUST begin with a lowercase letter, and can only contain lowercase letters a
-z
, digits 0
-9
, underscores _
, dashes -
, asterisks *
, and forward slashes /
. For multi-tenant vendors scenarios @
sign can be used to prefix vendor name. Suggested use is to allow set tenant id in the beginning of key like fw529a3039@dt
- fw529a3039
is a tenant id and @dt
is a vendor name. Searching for @dt=
would be more robust for parsing (searching for all vendor's keys).
Value is opaque string up to 256 characters printable ASCII RFC0020 characters (i.e., the range 0x20 to 0x7E) except comma ,
and =
. Note that this also excludes tabs, newlines, carriage returns, etc.
value = 0*255(chr) nblk-chr nblk-chr = %x21-2B / %x2D-3C / %x3E-7E chr = %x20 / nblk-chr
The length of a combined header MUST be less than or equal to 512 bytes. If the length of a combined header is more than 512 bytes it SHOULD be ignored.
Example: vendorname1=opaqueValue1,vendorname2=opaqueValue2
The value of a concatenation of trace graph key-value pairs. Only one entry per key is allowed because the entry represents that last position in the trace. Hence implementors must overwrite their entry upon reentry to their tracing system.
For example, if tracing system name is congo
, and a trace started in their
system, went through a system named rojo
and later returned to congo
, the
tracestate
value would not be:
congo=congosFirstPosition,rojo=rojosFirstPosition,congo=congosSecondPosition
Rather, the entry would be rewritten to only include the most recent position:
congo=congosSecondPosition,rojo=rojosFirstPosition
Limits:
There might be multiple tracestate
headers in a single request according to RFC7230 section 3.2.2. Maximum length of a combined header MUST be less than 512 characters. This length includes commas required to separate list items. But SHOULD NOT include optional white space (OWA) characters.
tracestate
field contains essential information for request correlation. Platforms and tracing systems MUST propagate this header. Compliance with the specification will require storing of tracestate
as part of the request payload or associated metadata. Allowing the long field values can make compliance to the specification impossible. Thus, the aggressive limit of 512 characters was chosen.
If the tracestate
value has more than 512 characters, the tracer CAN decide to forward the tracestate
. When propagating tracestate
with the excessive length - the assumption SHOULD be that the receiver will drop this header.
Single tracing system (generic format):
tracestate: rojo=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
Multiple tracing systems (with different formatting):
tracestate: rojo=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01,congo=lZWRzIHRoNhcm5hbCBwbGVhc3VyZS4
tracestate
Version of tracestate
is defined by the version prefix of traceparent
header. Implementation needs to attempt parsing of tracestate
if a higher version is detected to the best of its ability. It is the implementor's decision whether to use partially-parsed tracestate
key-value pairs or not.
Library or platform receiving traceparent
request header MUST send it to outgoing requests. It MAY mutate the value of this header before passing to outgoing requests.
If the value of the traceparent
field wasn't changed before propagation - tracestate
MUST NOT be modified as well. Unmodified headers propagation is typically implemented in pass-thru services like proxies. This behavior may also be implemented in a service which currently does not collect distributed tracing information.
Here is the list of allowed mutations:
span-id
. The value of property span-id
can be regenerated. This is the most typical mutation and should be considered a default.recorded
flag of trace-flags
may be set to 1
if it had value 0
before or vice versa. span-id
MUST be
regenerated with the recorded
flag update. See details of recorded
flag
for more information on how this flag is recommended to be used.recorded
. The value of recorded
reflects the caller's recording behavior: either the trace data were dropped or may have been recorded out-of-band. This mutation gives the downstream tracer information about the likelihood its parent's information was recorded.trace-id
, span-id
, trace-flags
are regenerated. This mutation is used in the services defined as a front gate into secure networks and eliminates a potential denial of service attack surface.
Libraries and platforms MUST NOT make any other mutations to the traceparent
header.
Library or platform receiving tracestate
request header MUST send it to outgoing requests. It MAY mutate the value of this header before passing to outgoing requests. The main concept of tracestate
mutations is that the order of unmodified key-value pairs MUST be preserved. Modified keys MUST be moved to the beginning of the list.
Here is the list of allowed mutations:
tracestate
keys for privacy and security concerns. The second scenario is a truncation of long tracestate
's.
Requirements to propagate headers to downstream services as well as storing values of these headers opens up potential privacy concerns. Trace vendors MUST NOT use traceparent
and tracestate
fields for any personally identifiable or otherwise sensitive information. The only purpose of these fields is to enable telemetry correlation.
Trace vendors MUST assess the risk of header abuse. This section provides some considerations and initial assessment of the risk associated with storing and propagating these headers. Trace vendors may choose to inspect and remove sensitive information from the fields before allowing the platform or tracing system to execute code that potentially can propagate or store these fields. All mutations should, however, conform to the list of mutations defined in this specification.
traceparent
field has a predefined set of values. These values are randomly-generated numbers. If a random number generator has any logic of using user-identifiable information like IP address - this information may be exposed. Random number generators MUST NOT rely on any information that can potentially be user-identifiable.
Another privacy risk of the traceparent
field is an ability to correlate calls made as part of a single transaction. A downstream service may track and correlate two or more calls made in a single transaction and make assumptions about the identity of the caller of one call base on information in another call. Service initiating calls MAY choose to restart trace while making calls that might identify caller in the downstream service.
Note, both privacy concerns of traceparent
field are theoretical rather than practical.
The field tracestate
may contain any opaque value in any of the keys. The main purpose of this header is to provide additional information about the position of request in the multiple distributed tracing graphs.
Platforms and tracing systems MUST NOT include any personally identifiable information in the tracestate
header.
Platforms and tracing systems extremely sensitive to personal information exposure MAY implement selective removal of values corresponding to the unknown keys. This mutation of the tracestate
field is not forbidden, but highly discouraged. As it defeats the purpose of this field for allowing multiple tracing systems to collaborate.
There are two types of potential security risks associated with this specification: information exposure and denial of service attacks against the tracing system.
Services and platforms relying on traceparent
and tracestate
headers should also follow all the
best practices of parsing potentially malicious headers. Including checking for header length and content of header
values. These practices help to avoid buffer overflow and html injection attacks.
As mentioned in the privacy section, information in traceparent
and tracestate
headers may carry information that can be
considered sensitive. For example, traceparent
may allow one call to be correlated to the data sent with another call.
tracestate
may imply the version of monitoring software used by the caller. This information could potentially be used to
create a larger attack.
Application owners should either ensure that no proprietary or confidential information is stored in the tracestate
, or
they should ensure that tracestate
isn't present in requests to external systems.
When distributed tracing is enabled on a service with a public API and naively
continues any trace with the recorded
flag set, a malicious attacker could
overwhelm an application with tracing overhead, forge trace-id
collisions
that make monitoring data unusable, or run up your tracing bill with your SaaS
tracing vendor.
Tracing vendors and platforms should account for these situations and make sure that checks and balances are in place to protect denial of monitoring by malicious or badly authored callers.
One examples of such protection may be different tracing behavior for authenticated and unauthenticated requests. Various rate limiters for data recording can also be implemented.
Application owners need to make sure to test all code paths leading to the sending of traceparent
and tracestate
headers. For
example, in single page browser applications it is typical to make cross-origin calls. If one of these code path leads
to the sending of traceparent
and tracestate
headers - cross-origin calls restricted via Access-Control-Allow-Headers
header, it may fail.