I find this pretty hard to read, but assuming I am reading it corectly, it basically says that if you read any packet or write any packet that has a frame other than ACK or padding, then you reset the timer. The text about "other packet has not been acknowledged or lost" doesn't make sense unless you have some very complicated timer scheme, because the packet won't be acknowledge or lost until after it is sent, at which point I have already reset the timer.
Yeah, this is opaque. I think that the logic is something like: when receiving a packet, restart the idle timer when sending an ACK-eliciting packet. if you haven't sent other ACK-eliciting packets since the idle timer was started, restart the idle timer That is, if you send 10 packets, only the first of those restarts the timer. This clearly needs to be reworded.
I'd suggest one change to the second bullet, Martin. "when sending an ACK-eliciting packet. if you haven't sent other ACK-eliciting packets since the idle timer was restarted due to a received packet, restart the idle timer" And yes, that text is not clear.
I like the look of the wording change in ac59e11. Is there something blocking it from being applied?
Nothing. I just neglected to open the pull request. That happens sometimes. Lots of changes in flight...
I think this looks good, but NAME has more familiarity with this code than I do, so I'm adding him as well.


In the recovery draft, RTT is briefly spelled out in section 5 but is used in several earlier sections. SRTT and PTO are not explained at all, it seems. TLP is spelled out. RTO has a section of its own but it referenced before that. The recovery doc cannot be a tutorial on transmission theory, but it would be helpful to list the key acronyms and possibly reference an RFC with more details. There are already sections summarising key variables in pseudocode. There are references at the end, but the reader would have to go through them all.
Yes, thanks for the note. I'm planning to do a cleanup of that doc soon.
I fixed defined RTT earlier in .


While we allow the consumer to start using new CIDs anytime, there are consequences of consuming them excessively, as discussed in starting from this comment. The text clarifies that excessive consumption is not what the issuer needs to support.
NAME For HTTP3, it is natural to assume that the connections would not survive for very long time. In such case, capping the number of total connection IDs is a practical countermeasure against excessive issuance of CIDs. Implementations need not to be as complicated as capping the frequency.
I don't want to put this in the draft. It tells endpoints not to change connection IDs based on some unspecified (scary!) constraints that may or may not be present in a peer. I realize that there might be a limited supply in some implementations, but artificial constraints on the rate of use is not the way I would address the problem. Refusing or slowing issuance of new connection IDs if someone retires too many is a better approach. In other words, implementations should deal with this on their own.
NAME If that's the concern, do you think clarifying that the issue can "refuse or slow issuance of new connection IDs" in section 5.1.1 would be fine? The problem I have with current text is that section 5.1.1 only says that the issuer SHOULD issue new CIDs as the consumer retires them. I think we need to clarify that doing that without limitation could be dangerous.
NAME Moved the clarification to the issuer side. PTAL.
One low-level observation: CID's need to be looked up in a hash table or similar. For a busy server multiplexing many connections, it want this table in fast cache, also to reject invalid CID's fast. You may be able to handle 5000 connections efficiently, but if you are required to handle 8x5000 this can degrade performance, especially on rejection. You can counter this by cryptographically mapping multiple CID's a single internal CID but that is not the simplest approach. Therefore, choosing a minimum number of CID's should be done with care. Would 2 or 3 not be sufficient for most use cases?
NAME > Unfortunately, this is a design change, and a new one. just to be clear, are you referring to my comment on a limit of eight, or something earlier?
NAME I think we should have a separate discussion around a sensible default. Part of the idea here was to make sure that we were very unlikely to run out, even around lost/retransmitted packets on different paths. Choosing a very low number brings with it other potential issues which probably should hold up this change?
NAME yes, I realised this relates to maxconnectionids.
I'm going with editorial then.
While I can see why you'd limit the frequency of issuing new connection IDs, I don't think you should limit the number of total connection IDs per connection.Unfortunately, this is a design change, and a new one. From an abundance of caution, we'll need an issue for this, and chair assessment of consensus.I'm not convinced this is a design change -- there was previously a SHOULD, and this adds additional text (including a MAY) around when you might choose to violate the SHOULD. However, no mandatory behavior is added, changed, or removed.A few nits, but this looks good to me. That said, this issue will need consensus.


I suspect these are uncontroversial.
All three Mart[ie]ns now agree, no further objections allowed.
LGTM, with the one change altered.


Another attempt to group related concepts in this section which hopefully (This is technically design, because previously-required server behavior is now only recommended.)
Why do I have to explicitly cancel after GOAWAY? Given that GOAWAY already tells the client this, what's the point?
GOAWAY tells the client that those requests aren't going to be serviced, but the streams will continue to exist. If the cleanup period is significant, having the streams open is a bad idea. However, attaching SHOULD to this is unnecessary, I think that we might instead just note that clients are free to do as they please with streams that won't be answered.
Discussed in Tokyo; this doesn't need to be required server behavior, but recommended (or RECOMMENDED) as a method to clean up state. Should have a reminder cross-reference to request rejection.
Lgtm now


I believe UNEXPECTED_FRAME is the preferred type.
I actually prefer the wording in duplicate push section and wonder if we should port that to PUSH_PROMISE too


Builds on .
Initial packets need to be in 1200 byte datagrams for DoS amplification reasons. We have an exception for the case where an ACK for a Handshake packet has been received from the peer. In that case, we remove the requirement to pad. That condition is extraordinarily obscure, especially if we are suggesting that endpoints not send any more Initial packets once they have Handshake keys. Removing this text is the right answer.
+1 to the proposed resolution.


I realise this may be a bit out of context for this PR, but I can't find any clear definition of when the handshake is complete other than this being magically spawned from the inner workings of TLS.
Previous discussions have established the CFIN is HoL blocking, partially because even though it's not strictly necessary, if client auth is in use, it's critical to receive the client's entire Handshake flight before the server processes 1RTT packets. I don't think the current TLS text is sufficiently clear(and normative) about that. Section 4.1.1 has: "Important: Until the handshake is reported as complete, the connection and key exchange are not properly authenticated at the server. Even though 1-RTT keys are available to a server after receiving the first handshake messages from a client, the server cannot consider the client to be authenticated until it receives and validates the client's Finished message. The requirement for the server to wait for the client Finished message creates a dependency on that message being delivered. A client can avoid the potential for head-of-line blocking that this implies by sending a copy of the CRYPTO frame that carries the Finished message in multiple packets. This enables immediate server processing for those packets." URL
I agree, perhaps this section needs to say something along the lines of:
How is this not a duplicate of ?
Nevermind, I can fix this easily.


This was always intended (and also written in the description of FRAMEENCODINGERROR), but didn't show up in the text somehow.
I agree with NAME about the MUSTs that you shuffled around are statements of fact, not interoperability requirements, but the addition is fine.


…are: require unlinkability by stating: "SHOULD NOT expose linkability" suggest use of encryption when embedding expiration time in the token
Thanks NAME !
My understanding is that tokens provided by NEW_TOKEN frames should not leak information that allows observers correlate the newly established connection with the previous connection. Such information include the time when the token was issued, RTT, server name (when SNI is not used or Encrypted SNI is used), or even the size of the token. However, the draft seems to be vague about the requirement. I think it might be worth clarifying the principle in the draft. "a token SHOULD include an expiration time" (section 8.1.2) - I think we should change this to "a token SHOULD be associated an expiration time", considering the fact that we are talking about "stateful" design (stateless design is mentioned briefly in one of the following paragraphs). In a stateful design, an opaque identifier is the only thing that a server should be allowed to send. "In a stateless design, a server can use encrypted and authenticated tokens to pass information to clients" - I think we might prefer using "SHOULD" rather than "can".
I think the current text is fine: "include" translates to "put into the token and encrypt" in my mind. Reading that a token is "associated" with an expiration time would give me pause.
Would you mind clarifying where the "and encrypt" comes from? My point is that that is the way it should be, however the text does not seem to clarify that.
I don't recall how I arrived at it -- perhaps some common sense. Now that I have implemented it, it seems only natural. Perhaps we should simply add "and encrypt" to the text instead of changing to "associated with," which is less clear?
I'm happy to know that we share the common sense :+1: As stated in the opening comment, my "editorial" preference goes to stating the principle that any information in an NEW_TOKEN token (other than the opaque lookup key used in a stateful design) SHOULD be encrypted. If that is to be met, I think adding "and encrypt" would be fine.
I agree with NAME -- I think stating this explicitly as a SHOULD is sensible.
(For the chairs, and triage purposes). This suggests a normative requirement that I don't think that we need, or that is already implied by existing text. There would be new normative text. I believe that is still editorial on the basis that the requirement is already implied, but will defer to your judgment.
FWIW, what I am shooting for is a change like URL
NAME and others, is kazuhoNAME kind of the change you were envisioning?
I was asked to move here. I miss the point, that the server SHOULD NOT construct the same token multiple times because this leads to additional scenarios where tracking by a network observer becomes feasible. The referenced PR does not fix this problem.
NAME I think we are in agreement that unlinkability between tokens is required. IMO the phrase "SHOULD NOT expose linkability" in the proposed text captures the concept, though I might agree that "i.e., ..." needs improvement.
NAME NAME change would be adequate, and I agree with NAME that this is effectively editorial. Do the chairs want to mark this issue as ?
NAME Yes, I agree with you on the phrase "Should NOT expose likability". To address my concern, I suggest to include "This includes that the server constructs each time a different token when issuing tokens to clients from the same source address." after line 1560 of #kazuho/base-draftsNAME
Marking this as editorial
Looks good, modulo the minor suggestions.


draft-ietf-quic-tls currently mentions AEADAES256CCM, however there is no way to use that AEAD in TLS 1.3 - TLSAES256CCMSHA256 does not appear in the . If I understand this correctly, the presence of AEADAES256CCM in the document was an editorial oversight.
Thanks. This was extremely confusing: you meant CCM, but wrote GCM in tons of places.


I must have missed this when making the PR to remove the version negotiation text. Thanks NAME !
I wasn’t aware that there was an issue. Sorry for that!
No worries, thanks for doing this!
We can probably remove big chunks of section "8.1. Protocol and Version Negotiation" since we don't do version negotiation anymore in v1.
Fixed with . Thanks NAME


As of , the client MUST use the same handshake message.
NAME That's quite complicated, and I'm not sure if we've chosen the right design here. I'd like to keep this PR editorial though. I opened to discuss this.
In URL, NAME points out that after a Retry, all 0-RTT packets sent in the first flight are still valid, although they will probably have been dropped by the server (unless there's significant reordering). That means that a client SHOULD retransmit all 0-RTT packets in the second flight. We now have two different scenarios: In the case of a HelloRetryRequest, the client is prohibited from sending 0-RTT, and has to cancel retransmissions for all outstanding 0-RTT packets. In the case of Retry, the 0-RTT packets from the first flight are declared lost (without any effect on the cwnd though) and retransmitted. Considering that Retry adds a roundtrip anyway (and thereby limits the usefulness of 0-RTT), and that we only expect it to be used by a server that's under attack (which makes it likely to refuse 0-RTT later in the handshake), does it really make sense to send 0-RTT in that case?
NAME I am not sure if I agree with the observation. The transport draft states that a server MAY send Retry packets in response to Initial and 0-RTT packets. A server can either discard or buffer 0-RTT packets that it receives (). Notice that the server is allowed to "buffer" 0-RTT packets. I am not sure if I like this ambiguity (because I think most if not all of us will be dropping 0-RTT packets that carry the original DCID when sending a Retry), but IMO the draft is clear.
NAME thanks for pointing that out. My point is that we should change the draft (i.e. this is a design issue, not an editorial issue). The reason a server (or a LB) performs a Retry is that it either has reasons to suspect that the client isn't reachable at the 2-tuple it sent the packet from, or that it is under load, and either uses the Retry to redirect traffic to a different backend server, or just to buy itself some time Under both circumstances, a server wouldn't want to allocate additional state for buffering 0-RTT packets. I'm therefore suggesting to change the draft such that: A server that performs a Retry MUST discard 0-RTT packets. A client MUST NOT use 0-RTT after a Retry was performed. Does that make sense to you?
Yeah, I can envision a server that considers buffering some number of packets to be cheaper than doing a TLS handshake. For such servers, delaying the TLS handshake until client proves it's existence but buffering the 0-RTT packets in the meantime would make sense. That said, I'd be fine with this, assuming that the discarded 0-RTT packets are the ones that carry the original DCID. I would prefer not having this requirement. Once the client is known to exist, a server needs to communicate with the client. Use of 0-RTT helps reduce the server load, because when used, average lifetime of connections becomes 1-RTT less. That leads us to having less concurrent connections.
When our implementation does Retry, it's because we've hit a memory pressure limit and we want to protect ourselves from attack. If a client supplies us with a valid token, we will accept 0-RTT still, but we won't buffer it from the previous Initial (without a token). So, if the client gets a Retry and still wants to do 0-RTT, they need to retransmit it, with the new CID, in a new packet (with a new packet number). I don't see why that design should be prevented.
Ok, I see that there's value in doing 0-RTT after a Retry, and furthermore, there seem to be good reasons (prevention of tampering by middle boxes) to disallow 0-RTT after a HelloRetryRequest in TLS 1.3, as NAME pointed out OOB. Since we can assume that most (all?) server implementation will drop 0-RTT packets when sending a Retry, the only reasonable thing to do for a client is to treat all 0-RTT packets as lost and retransmit them. Considering that, it would make sense to require servers to drop all Retry packets from the first flight, as NAME suggests.
NAME Thank you for suggesting a path forward. I think my weak preference goes to something slightly different. I think we are in agreement that most if not all the servers would not buffer the 0-RTT packets carrying the original DCIDs when they send Retrys. It makes sense to acknowledge the fact in the specification and suggest clients to retransmit 0-RTT data it has already sent. What I am not sure is if there is a reason to forbid servers from buffering such 0-RTT packets. From client's point of view, sending 0-RTT data once per connection is an optimization. Retransmitting 0-RTT data when the server sends a Retry is further optimization. If we think that way, servers that buffer 0-RTT packets when sending a Retry has increased chance of utilizing 0-RTT data. Compared to that, there is no reason to mandate servers to drop 0-RTT packets. To summarize, I think all we need to clarify is that servers responding with Retries are likely to discard 0-RTT packets that carry the original DCIDs and therefore that retransmitting 0-RTT data makes sense. But nothing more.
What if client takes action on Retry, and the old 0-RTT state lingers - wouldn't that make servers more vulnerable to DDoS attacks from packet sniffers?
This also bears on . If the DCID doesn't change, then the server won't be able to differentiate the first flight of 0-RTT packets, which you're proposing to drop, from the second flight of 0-RTT packets which it accepts. The current text says that the server MAY buffer and process them later; you're saying that servers probably won't. I think that's fine, but some might -- hence, I favor NAME suggestion to retain the permission to buffer but the caution to the client that the packets were quite possibly lost/dropped. In terms of actual spec change, I think that means this text in : ...becomes a SHOULD instead of a MAY.
It seems like you have sorted this out. I have no problem with using "SHOULD" over "MAY" here. Note that the reason we have this distinction is logical separation of the mechanism of Retry and 0-RTT acceptance. Retry strictly precedes the connection attempt. No point in blocking 0-RTT if you have to Retry. HelloRetryRequest is different in that certain characteristics it might cause to reset affect whether the 0-RTT works. Retry is required to use the same ClientHello, so there is no need to have the two interact; you can treat a Retry almost as a loss event for the first Initial and everything works fine.
So, the effect of not changing the handshake message is that all previously-sent 0-RTT data is still valid, but has potentially/probably been lost rather than queued. Clients obviously MUST NOT reuse the packet number for different data, but in their not-resetting, they also can't change the data that they've already sent in those packets. They'll need to retransmit it, because otherwise there's a potential DoS attack by delaying/replaying a 0-RTT packet by one RTT against a server that does Retries.This PR looks good to me as an editorial improvement. Regarding NAME point, the transport draft explicitly states that a server sending a Retry can buffer 0-RTT packets that carried the original DCID, and process them after receiving a response to the Retry. To put it differently, I think we are clear on the semantics and that there are ways to implement this without having security concerns. That said, I understand that NAME has opened a design issue () to discuss if that is the design we want to have.LGTM


This
The transport draft talks about QUIC being resilient to NAT rebindings. However, if a QUIC server is behind an L3 load balancer which simply routes based on 5-tuple, then connections to this server will (likely) not survive rebinding. I couldn't find any language which addressed whether such a deployment was "OK" or not. I think that since this load balance does not support NAT binding resilience, it is implicitly "bad" according to the draft, but others might disagree. In any case, I think there should be text to address this. Note, if the server advertised a preferredaddress which routed around the load balancer, and if clients were required to use this address, then that could obviously work. But preferredaddress support is a SHOULD, not a MUST.
We did discuss this issue. Use of empty connection IDs and demultiplexing based on source addresses. The conclusion there was that this was OK: people could do that, as long as it was clear that what they were getting was not different than TCP. That is, we made it clear that if you don't use the identifiers that you control (your addresses, the connection ID), then you accept that you can't handle migration at all and connections will drop (well, unless you do something like trial decryption, which has some obvious scaling issues). The outcome of that discussion was captured in . Assuming that Ryan finds this answer satisfactory (and I haven't missed anything), then I suggest we just close this as a duplicate of .
I don't think this captures Ryan's concern: when the server is using a non-zero CID and also is behind 4-tuple routing. I think he's looking for a section of text to coherently describe what you have to do here: send either disablemigration or preferredaddress; or forward packets between servers; or either don't use a common stateless_reset key, or put client address/port in the reset token. This is sort of ops-drafty but there real requirements on servers to behave in a secure way. I think someone with full command of the transport draft would figure this out, but it is not clearly stated anywhere.
I agree with Martin Duke and the language he suggests makes sense to me.
This does feel a bit ops-drafty to me, but I'd be happy to review a PR.
NAME notes that: We need to normatively reference 7540 for this capability.
See also discussion thread starting at URL Hi, RFC7540 allows crumbling cookies for compression efficiency with HPACK. However, neither quic-http nor quic-qpack drafts mention cookies or crumbling. Is it considered safe for a QPACK implementation to crumble cookies before compression and concatenate them after decompression? If so, should the draft include clarification on this? Thanks, Bence


This just writes down what was proposed in .
Looks fine, but what about some guidance on stream vs connection errors? And also the principle of nearly always choosing a hard protocol violation error, or a more granular error with same effect?
Stream vs. connection is something that application protocols (like HTTP) need to worry about. The transport only concerns itself with connection errors. The idea that you pick the most helpful error is not something that we need to codify (in my view).
Isn't that worth mentioning? Or do we already have such text?
This is already discussed in Section 11, which makes me wonder if it would make more sense for text on this PR to be placed there too.
This I do not disagree with. I was merely suggesting that we explain why, as a general rule, a hard error is preferred as a design principle, not the nature of the error message which I do think should have a large degree of freedom.
Maybe there is a followup needed for the HTTP draft. If the principles are generally the same, we can copy the text and then add some more to cover the idea that we make most errors fatal to the connection rather than the stream.
I like Lucas suggestion of adding this text to the existing and already fairly complete text in section 11.
I moved the second paragraph up to the error handling section, but the text about how codes are selected really belongs with the codes themselves.
We don't really have any real principles that we agree on for deciding what error codes we are describing. Proposal: if the error carries distinct semantics (like stream rejection in HTTP), then it gets a new error code if the error is frequent or particularly significant, then it gets a new error code otherwise, target a more generic error code that identifies the broad area of the protocol finally, if there is no more specific applicable error code, use PROTOCOL_VIOLATION
is example of an issue discussed where it would have been useful to have principles.


As discussed, this is a) hard to do in all cases, b) impossible to do in many, and c) not that useful.
good
When multiple connections share the same network path or multiple connection IDs are used for the same path, using the spin bit is complicated by the difficulty in correlating forward and reverse flows. There is currently text that uses RFC 6919 language in the draft recommending coordination of the spin to avoid this problem. However, as , this coordination seems difficult and therefore unlikely. It also creates a strong signal that indicates that all the connections on the path are terminated at the same endpoint on both ends. Maybe we don't want to be generating that signal. Also, because this coordination can't be guaranteed, the on-path observer is forced to build code for correlating/segregating measurements by connection ID in the case where there is no coordination. Maybe it's best just to remove this recommendation entirely.
+1 to removing the recommendation, or changing the statement to just a caution (e.g., if you are to coalesce connections that spin, then you'd better be aware of ...). FWIW I am not sure if I agree with this observation. IIUC, the connections sharing a 5-tuple will be terminated by the same endpoint on both ends; it is impossible for a middlebox to coalesce multiple QUIC connections. It is impossible because CIDs can change mid-connection, and a middlebox cannot determine to where the packets containing the new CID must be sent.
There definitely needs to be a degree of coordination between the middlebox and the endpoints, but that doesn't mean that you can't use something like the quic-lb stuff on either end (or both ends).
Yes, I think it is better to remove that recommendation, but perhaps replace it with a note to observers.


Use more uniform and more specific language in description of instructions, like "follows" instead of "includes" or implicit mention of a field. Add references to integer and string encodings where missing. Prefer internal references to external ones (also in Section 1.2). Explicitly allow a header block to not have any instructions after the mandatory prefix. Refer to Literal Header Field With Name Reference section in Literal Header Field Without Name Reference instead of verbatim repeating text on 'N' bit. Reshuffle sentences, insert references instead, remove sentences where references suffice.
Mostly good improvements; a few things to fix up further.


Is that not implied by the text here?
Kind of, but this text talks about the sender's perspective. Furthermore, checking for duplicates is inherently racy if you're using tokens for new connections while the old connection is still alive.
You know, I think that I put some proposed fixes for this in c016639e39c with rather than this branch. Moved it over here. NAME you approved , so I hope that moving that commit over here works for you.
Sec 13.2 of QUIC Transport explains what to do for the loss of a packet for every stream type except NEW_TOKEN. IMO it's nice but not critical to repair this loss, but I care less about how we resolve it than that we write it up.
Somewhat related question: Does this section fit better in the recovery draft?
I don't think so. This talks about the repair semantics of these frames, not the process of detecting loss. Recovery doesn't care about the distinction between PING and STREAM when it comes to repair.
LGTM, but do we need some text saying that when receiving a NEW_TOKEN frame, you MUST make sure that it's not a duplicate? Otherwise, a client might just save the token twice, and effectively reuse the token.


The initial request was simply to add sub-headings. I still feel like Security Considerations is a bit of a hodge-podge, and would welcome concrete suggestions or PRs for tightening it up.
Can you add a sub-heading(##) for each of these considerations, like in the transport draft? Originally posted by NAME in URL


I assume the intention was to forbid the same setting identifier occurring multiple times in a SETTINGS frame, regardless of the values. However, since a parameter is defined as an identifier-value pair, the current text does not forbid duplicate identifiers if the values are different.
An alternative below. But this is good either way.Looks fine to me, modulo the line length.


The nonce can just be opaque bytes. The block counter is tricky, as noted in the issue. The "obvious" choice is little-endian, as that is consistent with the philosophy of the designer (as I understand it), but that is not anything more than a guess. Absent strong evidence that big-endian is a better choice, I'm going to err on the side of not making a substantive change here. But I think that we need a consensus call to support that viewpoint.
Here's what 5.4.4 says However, RFC 8439 says: This is a little confusing, but I think the correct way to read this is that the ChaCha function takes the nonce as a 96-bit opaque quantity, and the fact that it's little-endian is an internal detail irrelevant to our purposes, so we should just replace the second sentence with "The remaining 12 bytes are used as the 96-bit ChaCha nonce". This is reinforced by the fact that the test vectors have the nonce as a byte block. The situation with the counter is more complicated because we are told it is both a counter and 32-bits (which might make you think it's an integer parameter), but it is also treated as a little-endian integer, which makes one think it's a bitstring. And then the test vectors treat it as a value, rather than as a bitstring. If you look at code it typically thinks of this as an integer: See, for instance: URL So, I think the right answer here is to think of ChaCha as having the following API: Note that because the value on the wire is in big-endian format, this would invert the nonce, but I think it's the right answer anyway. Otherwise, you're having to pass in the counter in the wrong format on big-endian platforms.
As a completely general note, I don't think it is right to change endianness of crypto wire formats. AES-GCM has a completely backwards neither little nor big endian encoding (byte and bit swapped so no platform wins) this remains so on the wire. Especially for HW processing this is significant. But little endian is also more performant on nearly all relevant platforms today so there is no need to go over board forcing big endian if it can be avoided without insulting RFC's.
Yeah, this should simplify that text some.
I like the proposed change, the text had somewhat confused me when implementing this in a previous life.
I don't think that the reinterpretation of the little-endian thing is safe, as per NAME comment. The takes a byte buffer as input rather than a . That is to allow for different block counter sizes. I'll make the editorial change, but leave the suggestion there alone.
I have a question out to the editor of the PKCS spec. It appears like there are no solid rules in the spec to support any particular position here. I'm going with my best guess at WWDJBD here, but I'm happy to make a change if there is a strong case made for something else.
NAME NAME can we flip this to please? I don't think that we need to change the substance here, but I want to confirm the little-endian/big-endian choice here. I hope to have more information to share on the question soon, but that might not be definitive. We might just have to make a call ourselves. Absent better information, I'm going to suggest that we continue with little-endian, but I this is far from an obvious conclusion.
The API I'm using for ChaCha20 takes a const uint8_t nonce[12], and I currently pass in a pointer (offset by 4 bytes) to the sample as the nonce, without doing any endianness conversions. I'm in favor of clarifying the language for this input. (Given that this has the design label instead of the editorial label, my design opinion is to keep with the simple thing of passing the bytes directly from the sample to the nonce input of ChaCha20 and not do any endianness conversions. I'm assuming that other crypto libraries have similar APIs though.)
The API I have takes . PKCS takes pointers just like the API NAME refers to (but it fails to specify the endianness, which is a critical oversight, compounded by the fact that the size of the counter can vary). I suspect that - as NAME says - the right idea is to note that ChaCha20 will interpret the nonce as three little-endian integers and that the counter might be expressed as a byte sequence that is then (probably) interpreted as a little-endian integer.
I understand and appreciate this being marked as “design”, though I consider the text that landed in master as editorial, as it (in my view) ended as adding practical advice while retaining the definition we have had.
Yes, but as a design issue, we want to confirm that "no action" is the correct outcome. It's cheap enough to do so.
I like the updated text. It reads good as it gives the practical advice first, then amends that with the formal definition.


Invariants text was unclear on where the Destination Connection ID field was and how big it could be.
Section 4.2 of the Invariant Short header are not explicit that the Destionation CID follows the first byte: "A QUIC packet with a short header includes a Destination Connection ID. The short header does not include the Connection ID Lengths, Source Connection ID, or Version fields." As see it just says that the destination CID is part of the short header, not where to find it.
Also the length of the DCID in the short header is unclear. With the long header's restriction on both source and destination CID I don't think the CID is of arbitrary length as stated in Section 4.3. I think it is 0 to 18 bytes long as set agreed by the connection.
Yes, DCID follows the first octet, and we should say that. As for the length, 0/4-18 is a restriction that we agreed applies only to long headers. You can - if the QUIC version allows it - negotiate other lengths for connection IDs using other methods.
Ok, the spec is in some sense clear that this can be of arbitary length. However, that this is not required to be consistent with the long header is far from obvious and probably should be explicitly remarked. An realization I didn't have before over this design is that a given destination IP address and UDP port that deals with multiple QUIC connections will be foreced to use one commond DCID length across all QUIC connections using that port, otherwise it will have to use prefixes for DCIDs to be able to determine how long the actual DCID field are to enable decoding the header protection. It might have been a misstake to not include an DCID length field.
The endpoint can decide on its own DCID length encoding, so that is fine.
One editorial suggestion, but LG


This changes HTTPUNEXPECTEDFRAME, which is used only once, to HTTPFRAMEUNEXPECTED that is used multiple times.
During the London interim, it was noted that error codes refer to things being "malformed," "bad," "invalid," and "unexpected." Pick one and use it consistently, or establish precise definitions for each. This applies to both the current document and .
The three words are not equivalent: while both "malformed" and "invalid" are "bad," "malformed" is not the same as "invalid."
Minor correction here, "bad" is only used in . The only instance of "bad" in current document is a substring of example "0x1abadaba". I'll fix the PR before it lands, whatever order we end up doing things in.
The different value-judgement terms are corrected by the flurry of settings PRs that NAME already did. HTTPFRAMEERROR has replaced malformed, and UNEXPECTEDFRAME and WRONGSTREAM are reasonably descriptive (though see , ). We do still have some variation between HTTPTHINGDESCRIPTOR, HTTPTHINGERROR, and HTTPDESCRIPTORTHING (e.g. HTTPUNEXPECTEDFRAME versus HTTPREQUESTREJECTED) that might be worth reconciling. Is it worth turning into HTTPFRAMEUNEXPECTED?
Resolved by and something prior to it.
oops, thanks


Fixing
changed the rules regarding what kind of packet an endpoint is allowed to send after receiving a non-ack-eliciting packet. The intention of that text used to be to prevent peers from getting stuck in an ACK-loop, but now the text says: Among others, this means that once an endpoint receives a non-ack-eliciting packet, it's not allowed to send a packet containing only PADDING. This would be valuable for defeating traffic analysis. While it would be possible to hack your way out of this by sending a PING + PADDING packet, I'm not sure if this was the intention of .
To me, the key phrase here is 'in response to'. If you want to send PADDING for some other reason and the congestion controller allows it, then you can send it.
This feels editorial...
I don't intend to do anything about this one. As Ian notes, you can use other reasons to justify sending of anything, just don't justify the sending on the basis of having received a non-ack-eliciting packet or you will get into the sort trouble that doesn't end.
I think the text is clear that sending PADDING is allowed at any time. I'm closing this with no action.
quic-transport draft 13.2.1: An endpoint MUST NOT send a packet containing only an ACK frame in response to a non-ACK-eliciting packet (one containing only ACK and/or PADDING frames), NAME points out that this allows one to send an ACK+PADDING packet, which seems to violate the spirit of the rule. This loophole breaks . I believe this may be Editorial if we all know what we meant.
Yes, it was intended that ACK should not elicit an ACK+PADDING packet.
I agree with this change, but I disagree that this is a satisfactory fix for .
This issue should have been
LGTM. Though all the "non-ACK-eliciting packets" is a bit of a mouth full.


Came from the discussion of
NAME : I realized after we did this PR yesterday that this does not I It reduces burstiness in general, which is still a good thing, but it doesn't change slow start increase. I'll send a PR or ask Vidhi if she wants to.
When an ACK arrives that acknowledges N packets (where N could be very large), QUIC will increase the congestion window by more than 1MSS per congestion window during Congestion Avoidance causing bursty send. This could also affect the slow start. if (congestionwindow < ssthresh): // Slow start. congestionwindow += URL else: // Congestion avoidance. congestionwindow += kMaxDatagramSize * URL / congestionwindow One could argue that QUIC ACKs every other packet, but there ACK packet could get lost or ACK for ACK packet could get lost. The draft is probably assuming packet pacing but not all implementations support that. I discussed this with Jana and he thinks that we should add something in pseudo code or some specific text in the draft.
PR added more specific text on avoiding bursts into the network, FYI.
Please note that PR addresses the issue of app-limited or pacing-limited cases to not increase cwnd when it is under utilized. My concern is cwnd being inflated due to an ACK frame containing ACKs for large number of packets.
NAME : The problem here is that we don't talk about burstiness that can occur when pacing is not in place, and a stretch-ack is received. PRR handles this, but we don't do PRR in the draft.
No we need to spend lots of time on Reno as everybody appears to be doing Cubic or BBR?
Cubic has this problem. BBR cannot be assumed.
Except Cubic opens the window based on time spent, not just ack received
I don't get the scenario. It seems that NAME is worried about receiving an ACK for a large number of packets. This is not a congestion window problem, because it happens even if the congestion windows size remains constant. The root cause is that the ACK of N packets now reduces the number of packet in flow by N. So this is strictly about pacing, not about congestion control.
NAME yes, and it can and still does burst because of stretch acks
Not just stretch Acks -- deliberate Ack compression by the network happens too. Very few alternative to pacing.
URL describes
Discussed in Cupertino. NAME to prepare a PR that hopefully can incorporate some ideas from RFC3465 to add safety features for non-pacing stacks.
But 3465 does not solve the burst problem. It limits the growth of the window, but does not do pacing of any kind.
I think we can change to something like this
NAME I think we need to do 2MSS clamping per ACK received and not per packet acked. I am thinking that we can remove OnPacketAckedCC and instead do OnAckReceivedCC. Pseudocode change something like this:
I think we can keep but just move cwnd update to as you propose.
Yup, that's an important missing fix, though it needs to be applied only when not pacing. Additionally, helps with burstiness in general. NAME : I would make a variable, and apply the min here when not pacing.
The proposed change has the effect of reducing the effective Window if ACKs are compressed. Say you have a window of 12 packets, acks are delayed, and a single ACK acknowledges 12 packets. Now you have a window of 2 packets, your transmission rate is 6 times slower than before, and it will take you 10 RTT to recover. Is that really what you want?
NAME This is what ABC (RFC 3465) does for TCP. ACK compression has side-effects, and the way to deal with it is, as you say, to pace packets out. This reduction in performance is only when the endpoint does not pace, and it is the only way to limit bursts in that case.
Also, ACK compression does not necessarily imply ACK pruning. I have traces in which 1 RTT worth of ACKs is compressed and delivered back to back. In that case, the proposed algorithm will do nothing: you will get 2 MSS added for each ACK, back to back within less than 1 ms.
FYI. FreeBSD implements rfc3465 (and enabled by default) URL Linux removed tcpabc implementation but saying it does in a different way: URL Considering FreeBSD doesn't have pacing by default, I think current proposed algorithm is not worse (or aggressive) than what tcp implementation does.
NAME I am working on the PR for pseudocode changes.
I agree that when not pacing, using an ABC limit is appropriate. FWIW Windows TCP was using a limit of 4 and not 2 for a long time and now uses 8 which is pretty close to IW10 burst in slow start phase.
For the WG: The proposed PR attempts to include a mechanism to limit CWND increase during slow start when not pacing. I'll note that by itself, it does not guarantee we follow the existing MUST: "Implementations MUST either use pacing or limit such bursts to the initial congestion window, which is recommended to be the minimum of 10 maxdatagramsize and max(2 maxdatagramsize, 14720)), where maxdatagramsize is the current maximum size of a datagram for the connection, not including UDP or IP overhead." I don't think we should add this to the pseudocode, because we say SHOULD pace packets and the pseudocode currently only covers SHOULDs, not MAYs. Adding all MAYs to the pseudocode is too complex. I don't have a concern about adding a reference to the existing RFC as a mechanism that MAY help limit bursts during slow start. I'd like a decision from the WG on whether to include this change and if so, some criteria on what should be included going forward.
I agree with the comment that burst from ACK compression or Stretch-ACKs result in bursts. I think the ways to change that is implement a maximum burst size or use pacing. The bursts do not appear a CC function to me.
Bringing conversation in from the list: With regards to , one of the things we found in testing was that it could (a) still allow an implementation to be vulnerable to too much bursting (the original problem), but more importantly (b) locked receivers into an ack pattern of acking every other packet. My main concern is that this causes problems if receivers want to experiment with and change acking strategies going forward in way that otherwise would be fine. One way to look at the problem described by this issue is that the existing text has a minor underspecification, in which it has the notion of a burst limit but says nothing about the frequency of such bursts. Two "bursts" of 10MSS that are sent within a couple nanoseconds are really one burst that's too big. Another way of fixing this is to remove even the specification of the 10MSS limit if we really don't want too much text about the non-pacing cases. Essentially, what this comes down to is: pace, or even if you don't pace, you need to not send bursts above some frequency; and if you don't set some pacing timers, you'll likely underutilize the link.
I'm going to put this out there: the answer is MUST pace. You might formalize this by saying the number of packets sent in any interval cannot exceed by more than the initial cwnd (i.e., the 10MSS). That allows for bursting, but requires a period of quiet either preceding or following any burst.
NAME Yes, it does largely reduce to "implementations MUST pace" (even if the pacing algorithm isn't very nuanced, in which case you'll perform poorly). The formalization of the rate, one option for which was specified in , I think is the heart of what should be done here.
There are two routes that we can take: Specify clearly what 'delay` refers to in the below statement, which is done in OR Change MUST pace to SHOULD and remove the 10 MSS burst size limit and provide general recommendation if an implementation doesn't pace. Something like:
I'm sympathetic to saying MUST pace, and I like the formatlization that NAME suggests. I've heard some pushback on saying MUST pace in the past, but it might be worth giving it a shot again. NAME NAME NAME : What do you think?
That said... I will note that NAME 's formalization allows for a burst at the end of an RTT. Consider for example when a single ACK is received that acknowledges an entire window. That would be legal by this definition. I think that's ok.
Discussed in ZRH. Proposed resolution is to define and REQUIRE pacing.
NAME I think requiring pacing makes good sense, with some formalization of what that means.
To clarify the status here, my understanding is that the proposal is to close this design issue with no action. The discussions have identified editorial improvements that will be addressed independently.
Let's continue the editorial discussion on .
Closing this issue and continuing discussion on
This needs to run through the consensus call procedure, so we'll keep it open for another week.
Let's get this in -- it addresses some burstiness anyway.LGTM


If the Retry token is known to be invalid by the server, then the server can close the connection with INVALID_TOKEN instead of waiting for a timeout. Came out of discussion of Issue Related to
You need to update the "Transport Error Codes" section too.
Thanks NAME I did that in a subsequent update. Refresh and tell me if it looks correct.
NAME you're missing the section that gives some text to each error code .
NAME do you have a suggestion about what the client should do? Clearly it could start a new connection, but it would't want to do that an unlimited number of times, and it may depend upon whether a TCP fallback is available. Given that, it wasn't clear I should recommend something. And I agree making them distinguishable should be a MUST, as stated in
I think that the best thing we can say is "the connection attempt failed". The first version of the PR strongly suggested that making another connection attempt was the right thing to do, but that gets into lots of difficult questions about identifying why. Given that this is an error that only happens if a Retry is spoofed, then I think that all we can say is: !
When the server receives a Token that is corrupted, it may want to close the connection quickly. This issue suggests adding a new error code to indicate the token is corrupted, INVALID_TOKEN.
This is - I believe - a design issue. The PR seems to have approval, we just need the appropriate process run.
We currently say that these tokens SHOULD be distinguishable by a server on the basis that an error in a NEW_TOKEN token is recoverable, but an error in a Retry token is fatal. I don't see why this can't be MUST. If we make it "MUST", then we can rely on this property in the design.
Agreed, that'd be a useful property to rely on.
Does that property need to hold if there are bit flips in the token? If you’re encrypting the token, this would be easy to ensure: put the distinguishing bit under crypto cover. If you’re using a map lookup, you’ll probably have to work a little bit harder.
Please ignore my last post, I mixed up Retry packets and Initials. Tokens in the client’s Initial are part of the AAD, so we don’t have to worry about bit flips.
Depending how the server distinguishes them, a change of method / confusion about what server you're sending it to could make you appear to be presenting a Retry token instead of a regular token. This error code enables the client to recover and retry quickly, but there's no language about what the client should do when it receives the code.Rereading the changes, I have the following editorial suggestion.


This was inspired by the comment by afrind at URL For each representation, uniformly use the word "representation" in the first sentence; use the word "identifies" if it's a verbatim entry; "encodes" otherwise; identify relative versus post-base indexing in the first sentence; move drawing right after first sentence; use consistent language describing binary format.


In , we say that one of the required operations on a connection is "enable 0-RTT". NAME : The current text implies that implementations are required to support 0-RTT, which is not the case. It would be more correct to say that applications must be able to prevent early data from being offered on a connection, even if the implementation supports it.
Alternately, the server section prefaces certain bullets with "If Early Data is supported, ...." That formulation might be used here as well. Also, this section uses "0-RTT" in the client piece but "Early Data" in the server piece. Should be consistent.


It is permissible for a client to do 0-RTT resumption with a NST from one connection and a token from another. This clarifies that a client only needs to associate a token (from a NEW_TOKEN frame) with the server it was from and no additional state, and that a server can't require that a token be from the same connection as the NST in use.
The token is used for deciding whether the server believes the packet it's receiving actually comes from the address it says it's coming from. I could believe a server might trust a client only if it's doing a 1-RTT handshake, but not trust it if it's doing a 0-RTT handshake, so might encode that bit in the token. I don't think the server should be putting more specific bits like "this is the NST that should be used" or "here are the SETTINGS from the previous connection" in the token. If the server doesn't like a token it receives with a 0-RTT handshake, it can send a Retry packet but still continue with the 0-RTT crypto handshake (even though it's now 1-RTT from a transport perspective). I don't see the need for there to be fate sharing between accepting a token and accepting 0-RTT - a server can reject one without rejecting the other. I've created to express the problem statement as an issue.
If you're going to continue with 0-RTT, but not trust the token, why not just limit the amount sent by the amplification factor? The receipt of a Handshake packet constitutes path validation. Also, a Retry doesn't allow you to acknowledge the client's Initial or any 0-RTT packets.
For a 0-RTT resumption in H3, a client needs 4 pieces of state previously received from a server: 1) a NewSessionTicket 2) Transport Parameters 3) SETTINGS 4) a token (from a NEW_TOKEN frame) The last isn't strictly necessary, but without it the server is likely to burn a round trip with a Retry packet to prove the client controls the source address. On a given connection, there is only one server SETTINGS and Transport Parameters, though there may be multiple NewSessionTickets and tokens. The first 3 pieces of state are necessarily required to have come from the same connection. A NST (from any connection) is needed since without it resumption can't happen. Transport Parameters and SETTINGS must come from somewhere for 0-RTT to be useful; the obvious and currently specified place is the SETTINGS and Transport Parameters that were sent on the same connection the NST came from. There's no such need that the 4th piece of state come from the same connection as the others. The token is used to prove that the client controls the address the packet is purportedly coming from. This can be done separately from any crypto state (in the NST), application state (SETTINGS), or the rest of the transport state (Transport Parameters). For a client to store the state needed for 0-RTT resumption, one possible implementation is to create cache entries containing an NST, TP, and SETTINGS. It's easy to do that where the TP and SETTINGS get duplicated N times, assuming N NSTs were received on a connection. If M tokens were received on that connection, and tokens need to be added to those cache entries, either the M tokens need to be paired to the N NSTs somehow, or all M tokens are in all N cache entries. This introduces noticeable complexity to a client's session cache implementation to correlate unrelated things. It should be clarified that the client doesn't need to associate the token with the rest of the state needed for 0-RTT. has suggested wording for that clarification.
As pointed out in NAME in URL, the proposed change essentially forbids a server from storing SETTINGS in the token. I think I'm persuaded by NAME argument above that we'd better forbid such coordination, as it would be a burden to certain client implementations. OTOH, I think that untying the relationship between session tickets and tokens does open a new issue, see .
I have some editiorial comments, but the design looks fine to me. While we might embed some information related to 0-RTT configuration in QUIC tokens, they would be decryptable and be usable, so we do not think there would be a problem for us (thanks to NAME for checking).A couple of editorial suggestions.


, requires transport stacks to support abrupt closure of a stream even after the stream has been cleanly closed. This reflects the possibility to transition from "Data Sent" (stream has ended) to "Reset Sent" by sending a RESETSTREAM. Based on the state diagram in the transport doc, if an implementation was already in the "Data Recvd" state, they wouldn't actually send a RESETSTREAM, as they're already in a terminal state; a reset call would be a no-op or an error of some kind. However, NAME points out that in a handle-based API, this essentially means that handles for closed streams have to be recognizable as such for the lifetime of the connection. This might be burdensome for some implementations. Should we reword the requirement here to say that it's permitted if the stream hasn't yet reached the "Data Recvd" state.
This was clearly just a miss on the wording. I don't think that the intent was ever to require that streams be kept around indefinitely. Sure, some stacks might be able to send RESET_STREAM at a whim, but it's not necessary


I didn't notice this change in NAME PR , but I think the older text is more correct. I added a bit more detail about why it cannot be treated as an acknowledgement. URL
NAME I updated the text to reflect both the lack of processing and packet number.


I was trying to come up with text for this but this PR looks great.
I believe there's an extra "is".
Can you write a PR? Removing either 'is' doesn't read well to me, but I suspect a sentence rewrite could be an improvement. ie: "A loss epoch ends when a packet sent after loss is first declared is acknowledged."?
I agree a rewrite would be better.


Lest some readers get the impression that each request consumes a single QUIC stream and each response consumes a single QUIC stream.


From Gorry's review.
Gorry suggested that the QUIC recovery draft should specifically recommend against packet reordering thresholds less than 3 because it will not be sufficiently tolerant of real-world reordering. Pull already created.
FWIW, I think this is likely design because it adds a SHOULD NOT, but it also doesn't substantively change the existing recommendation.


This turned out to be fairly simple, but I think that it's worth writing down. I don't have concrete recommendations, but I could add them if people thought that they help. See the commented-out bit.
Thanks both for the reviews, I've taken some of the suggestions and moved this to the appendix.
After discussing with editors, we agreed that this was a reasonable interpretation of existing text. We do agree that it would be worth discussing Mirja's views on ECN reliability and possibly changing the algorithm to reflect that, but it would be a design change.
In , NAME suggested that a different algorithm might be better to recommend enabling ECN and only disabling it if a failure is detected. That algorithm adopts the same posture as the draft: that ECN is optional and that endpoints are permitted to be cautious in enabling it. If we want to adopt the view that ECN works and endpoints are required to use it unless it breaks, that's a bigger change. I open this issue, while being aware that this is re-opening something we discussed and agreed previously, without there being new information to present. Chairs, if you do think we should discuss this, it's a design issue.
I would not be in favor of reopening this.
Just to clarify the proposal was that if you decide to enable ECN, you should just do it and only disable if you notice a failure, instead of having the more complicated probing logic that we have right now (in the appendix I think)
The shows that out of 14 implementations doing active tests, only 3 demonstrate ECN support. No ill will, but major places like AWS or the Windows OS do not support ECN, so it is not clear what more text in the spec would change.
This is just about this one paragraph: The concrete proposal would be to remove this paragraph.
Discussed in ZRH. Proposed resolution is to close with no action.
I will make a proposal for an editorial PR for the paragraph mention above to make it sounds less scary that this risk exists.
ECN-capable is not a constant condition, but we don't have any algorithm that governs it. I don't think that we can just say "mark until validation of ECN counts fails". Maybe the following algorithm is OK: When starting a new path: set ECN state to "testing" after N packets or M round trips, set ECN state to "unknown" on every ACK, check ECN counts if ECN validation fails set ECN state to "failed". if ECN validation passes and ECN state is "uncertain", set it to "capable". when sending packets, mark with ECT(0) if state is "testing" or "capable" (I know that this is unchanged text, but I just realized that we could be clearer.) Gorry pointed out that we can add an extra thing: if a path is marked as "failed", you MAY choose to occasionally switch back to "testing". Mirja pointed out that we need to deal with bleaching more than we need to worry about black holes. We should negotiate that. Originally posted by NAME in URL
I also would suggest one more intro sentence could please some to say "When a path supports ECN, this enables ECN marking and uses ECN for a new path" - to effectively say that the validation is about how to detect and react to problems with a path that is not ECN capable.
Discussed in Cupertino.
Presumably this is just another section in Section 13.4? That would be useful.


Note also the editorial changes in , which are great and still needed: this adds a little bit more to the reference from those changes to do the key clarifying for . The remaining clarification about is coming in the SPA PRs, but I wanted to look at this paragraph in one place.
Section 12.2 makes two quasi-conflicting statements about coalesced packets. In a connection where the sender has multiple CIDs in hand, using two different CIDs in the two coalesced packets is entirely consistent with the guidance to the sender, but would be dropped based on the guidance to the receiver. We should align these two statements.
Fixing this would require a change to normative language. I'm of the opinion that the first should change to different connection IDs rather than connections; the second might change to a MUST NOT, but it could remain a SHOULD and achieve the same end. The main reason for this is that routing will be based on the first packet. But there is also a secondary benefit in not creating a strong correlation between different connection IDs for the same connection.
But this complicates the coalescing logic, because you no longer only need to look at packet types but now also CID fields. At least in my stack, that is an ugly change to make at this stage.
The alternative, which I can now see is probably a smaller change is:
What motivated this was two observations: we don't coalesce except during the handshake, where we are not migrating some stacks generate packets for coalescing while processing, and so they might generate a datagram with multiple connection IDs
NAME writes: This is much simpler alternative than what is being proposed. Now an implementation can only drop a packet after doing a connection lookup, whereas before it could just compare the CID to the previous value. If this is just a matter of personal preference, I vote for this approach, so now the score is 1:1 (with NAME being on the opposite side). To take a step back: why is recommendation to drop packets exist in the first place? If a sender wants to coalesce packets from different connections, why not let it?
I think it should not allow having multiple packets for different connections in a single UDP datagram - please. Currently I see implementations (including mine) have something like: Receive UDP packet. Parse packet header. Check CID, find the connection handler. Pass the packet to the respective connection (thread) to handle the packet. I think it makes implementations much more complicated (already). Also I wonder it may cause security issue if having bad assumption?
The classic example would be coalescing Handshake packets, whose DCID is the SCID used by the server, and Initial or Zero-RTT packets, whose DCID is the client-chosen Initial DCID. The client MAY switch to using the server chosen CID for Initial packets. Maybe we should say that it MUST do that if it is going to coalesce Initial and Handshake packets. Not so clear about 0-RTT packets. Is it actually allowed to send them with a different DCID than the Initial DCID?
NAME : Our assumption has been that load balancers, for instance, can simply route on the CID in the first header they see in the UDP datagram. If we allow different connections to be coalesced, that complicates things. It may also start to leak some things about the way load is distributed within the server network: if some CIDs when combined cause loss and some don't, that reveals some things about how the server load is distributed. Is there a strong argument for changing this behavior? This new language in was always the intent. I suspect it didn't change when we started allowing multiple CIDs within a connection. Unless there's an argument for changing behavior, this is really an editorial PR.
NAME If I understand correctly, every time you change the connection ID, you change the connection ID for all packet number spaces. Assuming that my reading of the spec is correct in this regard, it seems like there's no compelling reason for making this change?
NAME the spec actually says (7.2): The first flight of 0-RTT packets use the same Destination Connection ID and Source Connection ID values as the client's first Initial packet. Upon first receiving an Initial or Retry packet from the server, the client uses the Source Connection ID supplied by the server as the Destination Connection ID for subsequent packets, including any 0-RTT packets. SO you are correct, there is no ambiguity.
NAME Thank you for looking this up. Then I'm opposed to making any change here. Not only is it unnecessary (that alone should be enough), it would also make it more difficult to evolve the way we deal with coalesced packets in the future: I can imagine a future QUIC version introducing a way to authenticate the entire coalesced packet, while at the same time compressing the header. This will be easier if we keep the explicit property that all connection IDs of the coalesced packets match.
NAME I'm not sure if I agree. Coalescing of QUIC packets is a feature specific to v1, entities that operate without the knowledge of the connection (e.g., load balancers, datagram dispatchers) are not expected to consult the CID values of QUIC packets that are appended. Therefore, what we are talking here does not affect what we can do in the future.
That argument was based on the thought that it will be easier in a future QUIC version because the diff to v1 would be smaller. My main point is that this change is unnecessary because when endpoints change connection IDs, they change it for all packer types at the same time.
NAME Thank you for the clarification. I think I do not agree to this, too. IIUC, the consensus we reached with was that the CID of a Handshake packet and that of the 1-RTT packet can be different. While it is true that the initial DCID set by the client is expected to change when the client receives the first "active" connection ID supplied by the server, the fact does not mean that the CIDs of all QUIC packets within a datagram has to be the same in all the cases.
I think we need to make a change, in one direction or the other. It's unfortunate to mix CIDs, since it clearly links them to any observer, but doesn't seem like a big deal. For some implementations, it sounds like it's a big deal to avoid mixing them. Therefore, while I'm content with either resolution, it seems like it makes more sense to resolve toward being more permissive; e.g. . Mixing packets from different connections seems like a generally bad idea, since it assumes an implementation structure on the far end for it to work. The more likely outcome is that coalesced packets from other connections will simply be dropped as invalid. (Of course, this very property could be used for padding / misleading observers.)
Hi Mike, Can you say more about why you don't see this as an issue? Is it because this use case is most likely during the handshake? Or is there some other linkability already present that makes this less of an issue? regards, Ted
Some implementations try to change the CID as soon as possible, to avoid linking 1RTT traffic with the CID visible during the handshake and to improve privacy. There will be a brief window when they still need to send both some handshake packets and some 1RTT packets. But coalescing a Handshake packet with the old CID and a 1RTT packet with the new CID links old and new CID and thus negates the effort. And just in case someone asks, no, you don't really need to decrypt the packets to access the second CID. You just need the coalesced packet and another 1RTT packet. Not hard.
NAME If that is the concern, wouldn't it be better for endpoints to migrate to a new CID after learning that the peer has dropped the Handshake keys? If an endpoint switches to the new CID before the handshake keys are dropped by the peer, there's chance that the peer might use a new CID in the Handshake packets that the peer sends. That at least leaks when the connection was established for that CID.
NAME yes, the simplest way to not run into coalescing problems is to not start migrations or CID changes before Handshake Done.
Apologies, I'm getting lost in the weeds here. I agree with NAME that we need to make a change, and that is really meant to fix the text to what the intent always was. NAME NAME NAME Are we all agreeing on this point?
NAME That is my view. I think to be an editorial amendment of . Regarding linkability problem, the simple and safe way of mitigating the problem (as proposed in URL) does not interfere with the proposed resolution.
NAME I personally prefer the old version, force single CID in coalesced packet. Loosening the text to allow multiple CID requires a code change in my implementation.
I'm with NAME Single CID.
Both, actually. This occurs during the handshake, when addresses are required to be stable. Unless you're running a large number of parallel connections, all CIDs used on that 4-tuple during the handshake belong to the same connection with very high probability. You can break that linkage by deliberately changing CID and port at the same time, but you can't do that until the handshake is confirmed. If you want the linkage broken, you need to do that jump post-handshake anyway.
Hi Mike, I think "high probability" is probably app-dependent, because there may be cases where there are significant numbers of parallel connections. In the case of proxied queries, for example, the use of the same 4-tuple but a different CID could turn out to be common, because the proxy may prefer to keep traffic from its different clients separate even if they could be served by the same upstream server. A DNS example: if you have an DNS resolver talking to an upstream proxy or a common authoritative server, that resolver may want to set a different Client Subnet for different query streams, so that input can be used to determine the preferred reply. In the presence of a NAT or CGNAT, there could be interesting interaction with the dwell timers in the state table of the middlebox. If you change it too late, the middlebox may recycle the most recently freed port, which could result in you getting the same 4-tuple to an observer on that part of the path. Speaking strictly as an individual, I think using a single CID per coalesced packet as Christian prefers is cleaner and leaves open app behaviors we may not have yet identified. regards Ted I'm not a big fan of Client Subnet, and these particular parallel connections may or may not pose a risk; it's just a proxy situation where the parallelism might occur.
The discussion is active, I'm marking this as design.
NAME I might be missing something, but if an endpoint wants to cut linkability between the packets used during the handshake and the packets used in 1-RTT, that endpoint has to switch to a new CID after the peer has dropped its Initial and Handshake keys, regardless of the outcome of this issue. That is because if the endpoint switches to a new CID while the peer is still sending Initial or Handshake packets, we allow the peer to use that new CID in the long header packets sent in response. Therefore, I do not think that requiring endpoints to use one DCID in all the coalesced QUIC packets help us in practice.
In my implementation, ngtcp2, it processes packet in the way that PR proposes; which means I just over looked existing text, but anyway, I think that processing a packet atomically is cleaner approach in implementation wise rather than passing a some kind of pointer to the very first packet header.
NAME Once a non-coalesced packet is passed to a connection, in my implementation I don't have to do any checking. The server already makes sure to pass packets to the correct connection based on the DCID. So there's additional validation logic required for coalesced packets. And the easiest way is to just check that all DCIDs match. Sure, it would be possible to somehow query which other DCIDs would also be valid for the same session, but it's more complicated. And given that nobody has come up with any good reason to coalesce packets with different DCIDs in the first place, I'm wondering how this additional complexitiy would be justified.
I don't foresee a large problem with the proposed change, but I also don't think it's necessary.
Do we have a proposed resolution for this one?
I think we settled on requiring connection IDs to be the same, but I haven't had the time to write that up, so not just yet.
NAME FTR, discussion happened on the mailing list . My sense is that it's still not clear to some (including me) why the current fix in (requiring "connections" to be the same) doesn't work, but nobody is arguing strongly against what NAME said above (requiring "connection IDs" to be the same). So that might be the least common denominator.
added the text " Connection IDs that are issued and not retired are considered active; any active connection ID is valid for use at any time, in any packet type. " As far as I can tell there is no text specifying that an issued connection ID cannot be used with subsequent connections. The introduction to connection IDs state that there must not be correlation across connections, but if an issued connection ID was never consumed, there would be no correlation because it was transmitted encrypted.
We're currently not specifying which DCID is supposed to be used on retransmissions of Handshake packets. A client might receive a NEWCONNECTIONID frame in the first couple of 1-RTT packets it receives, before receiving either a HANDSHAKEDONE or an acknowledgement for a 1-RTT packet is sent itself. Therefore, it's still running loss recovery on the Handshake packets it sent out earlier. If there's an acknowledgement outstanding for one of those Handshake packets, the client will have to retransmit that packet. Which DCID is it supposed to use on the retransmission? Is it the DCID that was used on the original packet? Or is it the CID provided in the NEWCONNECTIONID frame? Note that the NEWCONNECTION_ID frame might already have requested the retiring of the CID used during the Handshake.
As I can see, there are three options: a) let the receiver of NCID to send Handshake packets using the new CIDs b) require (or recommend) receiver of NCID to continue using the original DCID until the handshake is confirmed c) require (or recommend) the sender to withhold the emission NCID (with a non-zero RTP) to until it sees an ACK for handshake confirmation I prefer (c) (or maybe (b)), because (a) has corner cases even though it might seem easy. Consider the case where a server sends TP.preferredaddress then as the first packet, sends the NCID frame with RTP set to 2. If we adopt (a), this would mean that the client would have only one CID that it can use (the one being embedded in the NCID frame) even though it needs two (one for the original path, and another for the path specified by TP.preferredaddress). As you can see, the design of the NCID has been based on the assumption that providing one new CID alongside a retirement request guarantees that the receiver would have enough active CIDs. But in this particular case, that is not the case. Based on this observation, and based on my assumption that a non-zero RTP would be sent by only some of the endpoints (the necessity of CID retirement is questionable for HTTP/3, the scope of QUIC v1, as the HTTP connections are typcially short-lived and can most likely be gracefully terminated to induce a new connection when retirement of CIDs is necessary), my preference goes to (c): an endpoint that uses RTP should cover the cost.
NAME (I think you mean RPT, not RTP) Thank you for the analysis. I prefer (c) as well, except that I do not want to condition it on non-zero RPT, to keep things simpler. Otherwise we're still left with what to do at the receiver. Ensuring that the handshake confirmation is acked means that there are no HS packets pending.
I'll also note that this issue was found through a resilience test we've built for the interop runner -- high loss handshake test. We're excited to see how everyone's handshake machinery performs under 50% loss :-)
At the time of this writing, ngtcp2 does (b).
NAME for you problem with (a) isn't that just a general problem with having multiple paths and then getting a NEWCONNECTIONID with RPT for all previous CIDs? You end up in a scenario where you only have 1 CID (the new one) and multiple paths that just had all their CIDs retired. If you can get away with using that one CID for the active path, you can still make forward progress. If not, the connection dies.
I believe we specify (b) already: "A client MUST only change the value it sends in the Destination Connection ID in response to the first packet of each type it receives from the server (Retry or Initial); a server MUST set its value based on the Initial packet. Any additional changes are not permitted; if subsequent packets of those types include a different Source Connection ID, they MUST be discarded. This avoids problems that might arise from stateless processing of multiple Initial packets producing different connection IDs." URL
NAME that is just indicating when a client should change DCID in response to receiving a Initial/Retry packet with a new CID. I don't think anything restricts either side from sending NEWCONNECTIONID frames in their first 1-RTT packets and their peers immediately using them for any packets (including handshake) they send. Personally, I don't have a problem with (a). I think that no matter what receivers will have to handle the multiple paths but single CID scenario, so I would prefer not to special case the handshake. I also don't think we have handshake confirmation problems because an endpoint should only send a NEWCONNECTIONID out when it's able to accept it. Then, if the peer can decrypt the packet with the new CID, I see no reason why it shouldn't be allowed to use it immediately.
NAME my reading of the text I quoted is that you cannot use any DCIDs in long header packets that weren't either the original client DCID/SCID or from a server Initial or Retry.
NAME There's a fourth option: d) require that the sender withholds the NCID frame until it has confirmed the handshake. As soon as an endpoint confirms the handshake, it drops the Handshake keys (I'm assuming here). As a result, the endpoint obviously doesn't care about Handshake packets any longer (and especially doesn't care which DCID was used to send the packet). Therefore, there's no problem if the receiver of the NCID frame immediately retires the connection ID used during the handshake, even if the the NCID and the HANDSHAKE_DONE frame were reordered.
NAME The part about what to do with DCID changes is underspecified. From the part of the spec you quote -- "Any additional changes are not permitted; if subsequent packets of those types include a different Source Connection ID, they MUST be discarded." -- it's not clear what these additional changes are limited to. I think the intent was for all long-form packets, but that's certainly not clear here. In any case, the problem with that approach is that the client will have to carefully construct HS packets with the old DCID while sending everything else with the new DCID.
NAME : (d) works too, but an endpoint still has to decide which DCID to use for sending HS packets in the case that the HANDSHAKE_DONE frame is lost. I'd rather not leave that unspecified.
"of those types" => Retry or Initial, since that's what it was talking about. That's trying to avoid the case where the server statelessly generates a new Retry (or Initial, in the rare case that would be stateless) with a second SCID upon receiving a retransmission or duplication of the client's Initial. But it doesn't say anything about the server issuing new CIDs via an NCID frame in 1-RTT before the handshake is confirmed. I think (a) or (b) would be fine; the server is clearly intending to accept the other CID if it issued it, but could conceivably use separate logic for long-header versus short-header routing.
Can someone explain why using the new connection ID is a problem?
NAME I think that that is a good way at looking into the issue. I agree that the endpoints can start using the new CIDs during handshake, if the peer sends NCID frames. The only problem that I can think of is related to retirement (see URL). When the server provides TP.preferred_address, and also decides to retire the CIDs immediately after sending all the handshake transcript, there's going to be a very small time window in which it has to send two CIDs (one for the current path and another for the path using the preferred address). Unless the QUIC stack allows bundling of two NCID frames when building a QUIC packet, the client might end up not having sufficient amount of CIDs, in which case it might terminate the connection. It is my view that this is very different from the ordinary case of retiring CIDs, in which case there would be enough time window to guarantee that a handful number of new generation CIDs are delivered before the older ones are delivered (cc NAME That said, I now tend to think that the most we might have to do is the following two: clarify that CIDs received using NCID frames can be used for Handshake packets have a cautionary text stating that a server needs to make sure that the client has enough number of new generation CIDs when it retires an older generation of CIDs
NAME Wouldn't we be able to solve this by using my proposed option d?
Artificial constraints on when the frame can be sent would be terrible to enforce and would introduce a performance cost for using NEWCONNECTIONID. In a great many cases, the 0.5RTT from the server is idle, which makes it a great way to get things like NEWCONNECTIONID sent without taking capacity from real work. Asking the server to defer sending would mean that NEWCONNECTIONID could compete with HTTP responses and the like. I see no reason not to allow use of connection IDs when they are available. They apply to the connection as a whole. You will note that we allow changing them with the Initial/Retry mini-protocol in ways that don't correlate with other connection-level events, so this isn't any different in that regard. Forcing the connection ID state to synchronize with other state changes seems more likely to complicate things than help. The caution NAME mentions is always true. Whenever an endpoint is forced to retire connection IDs, a short supply is always a risk to the connection.
I think QUIC allows more than 1 NCID frames in a single QUIC packet. Now my preference is (a). If server utilizes retire-prior-to, it should know the consequences and should provide sufficient backup connection IDs. Otherwise, it would be misconfiguration of server software.
NAME The problem is that the client might end up receiving only some of the NCID frames, unless the server always sends (and retransmits) the set of NCID frames as a whole. But anyways, I agree that (a) is probably fine. That is because it is trivial to design a server that never requests CID retirement during the handshake. All you need to do is start issuing the new generation of CIDs at least N seconds before sending NCID frames that asks for the retirement of the previous generation, where N is your handshake timeout. Knowing that this design is why I initially preferred (c), not realizing that it works equally well with (a). But know that I understand that, I prefer (a) as it is simpler.
Given a server has to sent NCID to have the CID change, I'm fine with (a), but I thought it was not what we'd previously agreed to and specified, so I believed (b) was the status quo.
I don't think this is a serious problem, because if the server doesn't want the long header DCID to change, it can wait before sending NEWCONNECTIONID. That said, we do have an issue here in that different readers of the draft had different interpretations of what was allowed. All that to say I don't much care if this is allowed or prohibited, but we should at least make an editorial change to clarify things.
I agree with NAME here. Any alternative is ok, but the draft should say whether servers can expect that the client might switch DCID and whether the clients can expect the server to send NCID w/ RPT>0 before the handshake is complete.
Let's keep it simple: no migration before Handshake Done, period. That would include prohibit NCID in 0-RTT Also state that implementations MAY drop any packet received with NCID different from original before Handshake Done.
NAME The problem is not that simple. Consider the case where the server receives ClientFinished and sends two 1-RTT packets, the first packet containing HANDSHAKE_DONE, and the second packet containing an NCID with RPT set to 1. If the first 1-RTT packet gets lost, the client receives a request to retire the connection ID before it confirms the handshake.
I think I'd prefer a design that meets the two requirements stated below: A server should not be required to withhold NCID frames until it receives an ACK for HANDSHAKEDONE. A client should be allowed to use the original CID until the handshake is confirmed. The intention is to minimize additional complexity to existing (and future) QUIC stacks. One solution that meets these requirements is to state that a NCID frame confirms the handshake, exactly the same way as HANDSHAKEDONE frame does. That would be a trivial change to any of the QUIC stacks, and there would be no ambiguity regarding the state changes.
Discussed in ZRH. Proposed resolution is to clarify that issuing a CID means being prepared to seeing it used at any time. Additional Editorial issue on packetization issues.
Editorial PR on packetization challenges with retransmitted frames is .
Ugh, editorial changes for a design issue. Reopened for consensus calling.
This was fixed by . is related to this issue, but that simply clarifies things. To keep things sane, let's just close this when is merged.
Looks good. URL discovered a minor "discrepancy" in our requirements for coalescing packets. We required that: senders not coalesce packets for different connections in the same datagram. receivers reject packets with different connection IDs in the same datagram. URL was the result of a discussion where I was convinced that looser constraints here weren't harmful. The requirements would identify connections, not connection IDs. Originally, my inclination was to fix the requirement on the sender to use connection IDs, which would remove some linkability. But then I was reminded that linkability is already defeated for several reasons. You can't coalesce outside of the handshake and we require a stable path for the handshake AND long headers include a source connection ID that can be used for linkability. So the linkability concern is basically lost. We also learned that some implementations were generating packets while processing incoming packets. Later they would coalesce these into a datagram that might then have different connection IDs on the packets it contains. There has been some opposition to the proposed resolution in PR 3870. Apparently, for some, having multiple connection IDs in the same datagram complicates processing. I don't understand this objection. It seems to me more difficult to retain state across packets than it is to process each atomically. I was hoping that Christian or Nick can explain more about how this affects them. Marten also indicated that there was a reason not to allow this, but if I understand that, this is based on some potential for future optimization, which could be conditional on having consistent connection IDs. If we moved to a scheme that required consistent connection IDs, anyone using that scheme could be required to avoid coalescing different connection IDs. So this seems more of a theoretical concern. And - in anticipation of maybe choosing to make senders use a consistent connection ID - if you might be generating coalesced datagrams with different connection IDs in packets, how hard would it be to ensure that these are consistent? Lars and Ian, I think both of you indicated that you might do this.


From an interop discussion with NAME and NAME HTTP/1.1 the presence of a Host header: If no Host header is present, the section on describes how you determine what host is being asked for, ending with "guess" if it's an HTTP/1.0 request. HTTP/2 says that you generate an pseudo-header if sending direct-to-H2 and carry through the Host header if you're relaying HTTP/1.1. There is no discussion about what to do with an HTTP/1.0 (or 0.9) request being relayed over HTTP/2, which wouldn't have a Host header; you're neither required to generate or send one, even an empty one, nor are you given guidance about how to figure it out. HTTP/3 carries forward the H2 language (currently by reference, and explicitly after ). Do we want to have some guidance here? Or does this issue instead belong in URL to move some of the existing guidance to be HTTP version-independent? (NAME
NAME Thank you for looking into the problem. Regarding if (or how) we should handle the issue, I think my weak preference goes to having one rule that applies to both HTTP/2 and HTTP/3. That would be less surprising to the users, and would also be helpful to implementations. It could well be the case for an HTTP server to have a common logic of handing pseudo headers (or lack of) between HTTP/2 and HTTP/3.
Host is an HTTP/1.x header field (it was just late to the party), so HTTP/1.0 includes it as an extension. And that requirement is specific to HTTP/1.1, so would theoretically not apply to HTTP/1.2 (and that was intended): it was a short term requirement imposed by the IESG because they wanted to require deployment and didn't understand that it was already deployed. In general, I don't understand why ":authority" exists (as opposed to always using ":host"), but I think the HTTP requirement is simply that the identifying data must be present somewhere in the request.
":authority" exists for the purpose of always transporting the equivalent of the , with ":scheme" and ":path" transporting the rest. It's mostly a consistency thing, I think. If you have a "Host" header instead, that's acceptable, too. So should we say that the target authority MUST be present in either "Host" or ":authority", or do we leave it alone?
Well, assuming we still allow requests for URIs that do not have an authority component (e.g, a URN or similar), I think it is better to just leave it alone. OTOH, folks are still going to ask what needs to be sent when the request target doesn't have an authority component, so that ought to be defined somewhere here more sensibly than it was for 1.1.
Discussed in ZRH. Proposed resolution is that if a URI scheme has a mandatory component, it better have one of those two things, preferably a :authority. Alert httpbis.
Doesn't this beg the question "what if the request is for a URI that does not have a mandatory authority component?" All of the other cases are described. I would add something like: Thanks!


A sender can intentionally skip a packet number, causing the receiver to treat a packet as "Out-of-order". From NAME comment on


Addresses . I'm flexible on wording and where exactly it lives in the document. I put it in Security Considerations.
I just completely rearranged the new section based on your input. See how it reads now. Editors, I don't seem to have permission to request review. Can someone add NAME
Unfortunately, some of the reviews are hidden as obsolete. I want to respond to this by NAME about disablemigration I tried to not recommend anything here except to send disablemigration. The client is going to decide what to do with regard to NAT rebinding. If wired, I probably send more PINGs or something. If wireless, maybe not. If I'm quite sure I'm not behind a NAT, I probably do nothing. There's nothing normative here but to send the TP, so if people hate this I can just delete the explanation.
I moved the whole thing to Section 5 as NAME suggested. I've taken most of the other suggestions, but am not sure if the intro needs to change, or what exactly people don't like about the disable_migration explanatory text.
I'm actually fine with this. We were considering keep alives on the server side when we can't support migration (or NAT rebinding) based on the current infrastructure. So having the spec suggest keep alives to the client would be better. It would allow the client to choose if it would prefer keep alives or just to close the connection.
After taking all suggestions from NAME I simply deleted the explanatory text about sending disablemigration. I didn't see the stuff about NAT rebinding as prescriptive or changing the semantics, but it's non-normative, clients are gonna do what they gonna do, and it's not worth holding up the rest of this over. If people want to add NAT rebinding considerations to disablemigration we can fight that out in another issue.
FWIW, we have a number of cases when the destination IP and port + CID would route correctly, but the original IP(being anycast) would not always route correctly. It's likely there are millions of connections to the destination IP being migrated to, but yes it's a smaller number than the anycast IP.
So NAME do you want new or revised language here, or will the other issue address the linkability stuff? I would like to close this out if it's good enough.
Given how hard this has been to get right, I want NAME to take a look. But I'm happy with this.
A reminder for NAME to either take a look or say that this is OK to proceed without another review.
I could not stand leaving this unmerged any longer. Anyone with more comments can make a new PR. Thanks to everyone who helped out here.
The transport draft talks about QUIC being resilient to NAT rebindings. However, if a QUIC server is behind an L3 load balancer which simply routes based on 5-tuple, then connections to this server will (likely) not survive rebinding. I couldn't find any language which addressed whether such a deployment was "OK" or not. I think that since this load balance does not support NAT binding resilience, it is implicitly "bad" according to the draft, but others might disagree. In any case, I think there should be text to address this. Note, if the server advertised a preferredaddress which routed around the load balancer, and if clients were required to use this address, then that could obviously work. But preferredaddress support is a SHOULD, not a MUST.
We did discuss this issue. Use of empty connection IDs and demultiplexing based on source addresses. The conclusion there was that this was OK: people could do that, as long as it was clear that what they were getting was not different than TCP. That is, we made it clear that if you don't use the identifiers that you control (your addresses, the connection ID), then you accept that you can't handle migration at all and connections will drop (well, unless you do something like trial decryption, which has some obvious scaling issues). The outcome of that discussion was captured in . Assuming that Ryan finds this answer satisfactory (and I haven't missed anything), then I suggest we just close this as a duplicate of .
I don't think this captures Ryan's concern: when the server is using a non-zero CID and also is behind 4-tuple routing. I think he's looking for a section of text to coherently describe what you have to do here: send either disablemigration or preferredaddress; or forward packets between servers; or either don't use a common stateless_reset key, or put client address/port in the reset token. This is sort of ops-drafty but there real requirements on servers to behave in a secure way. I think someone with full command of the transport draft would figure this out, but it is not clearly stated anywhere.
I agree with Martin Duke and the language he suggests makes sense to me.
This does feel a bit ops-drafty to me, but I'd be happy to review a PR.
I'm sure that this is now basically unrecognizable...Some more suggestions, but overall this LG