I think that we should be able to multiplex any TCP, UDP, or IP protocol. Internet protocol numbers are 8 bit fields.
So we need 16 bits for TCP, one bit to distinguish TCP and UDP, and one bit more we can use for IP protocol numbers and address space we can allocate privately. This argues for an 18 bit length field to allow for this reuse.
Leaves one unused bit. The spec above reflects this.
Back to alignment. If we demand 4 byte alignment, for all requests that do not end up naturally aligned, we waste bytes. Two bytes are wasted on average. At 14.4Kbaud the overhead for protocols that do not pad up would on mean be 6 bytes or ~3ms, rather than 4 bytes or ~ 2 ms (presuming even distributions of length). Note that this DOES NOT effect initial request latency (time to get first URL), and is therefore less critical than elsewhere.
I have one related worry; it can sometimes be painful to get padding bytes at the end of a buffer; I've heard of people losing by having data right up to the end of a page, so implementations are living slightly dangerously if they presume they can send the padding bytes by sending the 1, 2 or 3 bytes after the buffer (rather than an independent write to the OS for padding bytes).
Alternatively, the buffer alignment requirement can be satisfied by implementations remembering how many pad bytes have to be sent, and adjusting the beginning address of the subsequent write by that many bytes before the buffer where the mux header has been put. Am I being unnecessarily paranoid?
Opinion: I believe alignment of fragments in general is a GOOD THING, and will simplify both the mux transport and protocols at higher levels if they can make this presumption in their implementations. So I believe this overhead is worth the cost; if you want to do better and save these bytes, then start building an application specific compression scheme. If not, please make your case.
I've ended back at the original bits specified, rather than the smaller set suggested by Bill Janssen. This enables full emulation of all the details of a socket interface, which would not otherwise be possible. See details around TCP and socket handling, discussed in books like "TCP/IP Illustrated," by W. Richard Stevens.
Question: do we want/need a short control message? Right now, the out for extensibility are control messages sent in the reserved (and as yet unspecified) control session. This requires a minimum of 8 bytes on the wire. We could steal the last available bit, and allow for a 4 byte short control message, that would have 18 bits of payload.
Opinion: leave for future, lets see if we need it.
the above allows for someone who just wants to mux a single protocol to entirely ignore protocol ID's.
It occurs to me that defining a control message to inform the other end of the connection which protocols you are able to handle may save a round trip in the HTTP related cases. For example, if a connection starts out using HTTP, then we might know that we can upgrade to NG as soon as the first reply is back from the server, if it tells the client that it is also NG capable.
Is this worth defining immediately, or should we wait for experience?
Wasting bytes in general, and in particular at connection establishment, for a multiplexing transport must be avoided
There are several reasons for this:
So there is more than the usual tension between generality vs. performance.
Human perception is about 30 milliseconds; if much more than this, the user perceives delay. At 14.4 K baud, one byte uncompressed costs .55 milliseconds (ignoring modem latencies). On an airplane via telephone today, you get a munificent 4800 baud, which is 3X slower. Cellular modems transmitting data (CDPD), as I understand it, will give us around 20Kbaud, when deployed.
So basic multiplexing @ 4 byte overhead costs ~ 2 milliseconds on common modems. This means basic overhead is small vs. human perception, for most low speed situations, a good position to be in.
On connection open, with above protocol we send 4 bytes in the setup message, and then must open a session, requiring at least 8 bytes more. 12 bytes == 7 milliseconds at 14.4K. Not 64 bit aligned, and 4 bytes costs of order 2 milliseconds. Ugh... Maybe a setup message isn't a good idea; other uses (e.g. security) can be dealt with by a control message.
Answer: no. More bits than we need. Current protocol doesn't use as many.
We want to mux multiple protocols simultaneously over the same transport connection, so we need to know what protocol is in use with each session, so the demultipexor can hand the data to the right person. (e.g. SUNRPC and DCERCP simultaneously).
There are two obvious ways I can see to do this:
a) send a control message when a session is first used, indicating the protocol.
Disadvantage: costs probably 8 bytes to do so (4 mux overhead, and 4 byte message), and destroys potential 64 bit alignment.
b) if syn is set indicating new session, then steal mux_length field to indicate protocol in use on that session. (overhead; 4 bytes for the mux header used just to establish the session.)
Opinions? Mine is that b) is better than a.
Answer: b) is the adopted strategy.
For a given stream, priority will affect which session is handled when multiplexing data; sending the priority on every block is unneeded, and would waste bytes. There is one case in which priority might be useful: at an intermediate proxy relaying sessions (and maybe remultiplexing them).
If so, it should be sent only when sessions are established or changed. Changes can be handled by a control message. Opinions?
A priority field can be hacked into the length field with the protocol field using b) above.
So the question is: is it important to send priority at all in this mux protocol? Or should priority control, if needed, be a control message?
Answer: Not in this protocol. Opens Pandora's box with remultiplexors, which could have denial of service attacks.
Is the setup message needed? I'm thinking more and more that it isn't. and initial bytes are precious (see performance discussion above), and it complicates trivial use. If we move the byte order flag to the mux header, and use control messages if other information needs to be sent, we can dispense with it, and the layer is simpler. This is my current position, and unless someone objects with reasons, I'll nuke it in the next version of this document.
Answer: Not needed. Nuked.
While higher layer protocols using host dependent byte order can be a performance win (when sending larger objects such as arrays of data), the overhead at this layer isn't much, and may not be worth bothering with. Worst case (naive code) would be four memory reads and 3 shift overhead/payload.
Opinions? I'm still leaning toward swapping bytes here, but there are other examples of byte load and shift (particularly slow on Alpha, but not much of an issue on other systems).
Answer: Not sufficient performance gain at mux level to be worth doing. Defined as LE byte order for mux headers.
there are several error conditions, probably best reported via control messages from server:
Any others? Any twists to worry about?
Answer: Only error that can occur is no such protocol, given no priority in the base protocol. May still be some unresolved issues here around "Christmas Tree" message (all bits turned on).
Any reason to believe that the 32 bit length field for a single payload is inadequate? I don't think so, and I live on an Alpha.
Answer: 32 bit extended length field for a single fragment is sufficient.
Does there need to be a bit saying the payload is compressed to avoid explosion of protocol types?
Answer: It seems best not to mix conceptual differences in the same protocol.