TCP vs UDP

The question

A client sends each request to a server 0.1 ms away as two writes, a header and a body, and every request takes 44 ms. A 1 MB download on a fresh connection to a server 100 ms away takes about 700 ms on a 50 Mbit/s line and about 700 ms on a 1 Gbit/s line. A voice call over TCP on a link that drops 4% of packets misses its deadline on 17% of its audio frames. And a service that opens 500 new connections a second to one backend starts failing with “cannot assign requested address” after about a minute.

None of these is a bug. Each is one of TCP’s guarantees doing its job, and every guarantee is enforced the same way: somebody waits, for a round trip, a timer or a missing packet. This page finds each of those waits, puts a number on it, and works out what you take on when you choose UDP to avoid them.

What each one actually promises

UDP (RFC 768) adds three things to IP: ports, a 16-bit checksum, and message boundaries. A datagram arrives whole, at most once, or not at all, in any order. Nothing in UDP knows how fast the receiver reads or how full the network is, and sendto() returning tells you only that your own kernel took the bytes.

TCP (RFC 9293) promises one thing, and the exact wording matters: a reliable, ordered stream of bytes between two kernels, or an error. Read it slowly and four consequences fall out, each of which this page measures:

1. A stream of bytes, not messages. Your writes are not preserved as units. Where one message ends is your problem.
2. Ordered means a hole blocks everything behind it. Bytes that arrived are held until the missing ones are resent.
3. Reliable means retried, by timers. Those timers are measured in hundreds of milliseconds, and TCP keeps retrying for minutes before it reports the error.
4. The sender is throttled by two windows: the receiver’s free buffer (flow control) and the sender’s estimate of what the network can take (congestion control). On a new connection that estimate starts small.

“Between two kernels” is the part people miss. When the peer’s kernel acknowledges your bytes, your side is done; if the peer’s process crashes before it reads them, they are gone and you were told “success”. Only an application-level reply can confirm the application acted, which is why payment APIs add their own acknowledgements and idempotency keys on top of TCP.

The pieces every later section is built from. TCP cuts your bytes into segments, each one IP packet. It numbers every byte (the sequence number), and the receiver answers with an acknowledgement (ACK) that names the next byte it expects. The time from sending a segment to getting its ACK back is the round-trip time (RTT). Here are 4,000 bytes sent at once, numbered from 1 for readability (real connections start from a random number):

Segment	Carries bytes	Receiver answers	Meaning
1	1–1,460	ACK 1461	“I have everything up to 1,460”
2	1,461–2,920	ACK 2921	“everything up to 2,920”
3	2,921–4,000	ACK 4001	“everything up to 4,000; 4,001 next”

ACKs are cumulative: ACK 4001 also confirms segments 1 and 2, so a lost ACK costs nothing once a later one arrives. A byte that stays unacknowledged for too long is assumed lost and sent again; much of this page is about how long “too long” is.

Before any data, the two kernels agree on starting numbers in the handshake: the client sends a SYN (“my bytes start at x”), the server answers SYN-ACK (“mine start at y, and I have yours”), and the client’s ACK can carry the first request. That is one RTT before the server sees a byte of data. At the end, each side sends a FIN (“I have nothing more to send”) and the other ACKs it; an RST (reset) instead aborts the connection at once.

What people assume first, and how it breaks

Two beliefs cause most transport bugs. The first is that one send on one side is one receive on the other. Here is a server that parses each read on a TCP socket as one JSON message. The client sends five small score updates and one 220 KB leaderboard, while the server is busy for 50 ms:

Python

naive_framing.py

# runnable version: examples/tcp-vs-udp/python/naive_framing.py
while True:
    chunk = conn.recv(65536)  # "one recv = one message" -- the naive assumption
    if not chunk:
        break
    reads += 1
    start = chunk[:24].decode(errors="replace")
    try:
        message = json.loads(chunk)
        print(f"  recv #{reads}: {len(chunk):>6} bytes, starts {start!r} -> parsed {message['type']}")
    except json.JSONDecodeError as err:
        print(f"  recv #{reads}: {len(chunk):>6} bytes, starts {start!r} -> {err.msg}")

JavaScript

naive-framing.mjs

// runnable version: examples/tcp-vs-udp/js/naive-framing.mjs
socket.on('data', (chunk) => {
  // "one 'data' event = one message" -- the naive assumption
  reads++;
  const start = `'${chunk.subarray(0, 24)}'`;
  try {
    const message = JSON.parse(chunk);
    console.log(`  data #${reads}: ${String(chunk.length).padStart(6)} bytes, starts ${start} -> parsed ${message.type}`);
  } catch (err) {
    console.log(`  data #${reads}: ${String(chunk.length).padStart(6)} bytes, starts ${start} -> ${err.message.split(/ at position|, "/)[0]}`);
  }
});

python naive_framing.py (Linux)

  recv #1:  65536 bytes, starts '{"type": "score", "playe' -> Extra data
  recv #2:  62699 bytes, starts '{"player": "p1824", "poi' -> Extra data
  recv #3:  65536 bytes, starts 'points": 3518}, {"player' -> Expecting value
  recv #4:  26277 bytes, starts '5289}, {"player": "p5290' -> Extra data

client: 6 send() calls of [47, 47, 47, 47, 47, 219813] bytes
server: the reads above. Message boundaries did not survive the trip.

Six messages went in, four reads came out, and none parsed. The first read holds all five small messages and the start of the leaderboard. On a quiet test machine each recv often does return one message, which is why this code passes tests and fails in production under load.

The second belief is that UDP is TCP minus the overhead. udp_loss.py sends 5,000 numbered 1,000-byte datagrams from one process to another on the same machine, no network involved, to a receiver that spends 0.2 ms on each:

python udp_loss.py (Linux)

full speed:
sender:   5000 datagrams sent in 14 ms, 0 send errors
receiver: 104 received, 4896 lost (98%), 4 gaps, 0 out of order

paced to 3,000 per second:
sender:   5000 datagrams sent in 1667 ms, 0 send errors
receiver: 5000 received, 0 lost (0%), 0 gaps, 0 out of order

98% vanished, with no error on either side: the receiving socket’s buffer filled and the kernel dropped the rest. TCP would have slowed the sender with its receive window. UDP did not remove that overhead; it handed you the job.

The core idea

Every TCP guarantee is a wait with a known size. The table is the map; each row has its own section below, which builds the mechanism up from the pieces above.

Guarantee	Enforced by waiting for	Typical cost
Both sides agree to talk	the handshake	1 round trip per new connection
Don’t flood the network	the congestion window to grow	1 round trip per doubling, from 14.6 KB
Nothing lost	a duplicate ACK or a timer	1 round trip, or 200 ms to seconds
In order	the missing segment	everything behind a loss waits too
Fewer tiny packets	an ACK (Nagle) that the peer delays	up to 40 ms on Linux, 200 ms elsewhere
No stale packets in a new connection	TIME-WAIT	the port, for 60 s on Linux

UDP waits for none of these and guarantees none of them. So the real choice is never “reliable or fast”. It is: which of these waits can your data afford, and which guarantees are you willing to rebuild yourself?

Here is the bill for one ordinary request: a 100 KB HTTPS response on a new connection, 100 ms away, on a 50 Mbit/s link.

Enlarge figure

One 100 KB response on a new connection takes about 509 ms. The bytes need 16 ms of the link; the rest is two handshake round trips and three slow-start round trips, each waiting for acknowledgements before the window may grow.

Two of the five round trips are handshakes (TCP, then TLS), and three are slow start sending 10, 20 and then the last 39 segments. The link could have carried the whole response in one round trip plus 16 ms. Nothing in this picture is lost or broken; it is TCP being careful.

How it works, piece by piece

A byte stream, so you frame your own messages

TCP’s specification says outright that there is no link between how the application writes and how segments are cut (RFC 9293 §3.7). The sender’s kernel may merge small writes into one segment and must split a large write into segments of about 1,460 bytes (the MSS, maximum segment size: a 1,500-byte Ethernet packet minus 40 bytes of IP and TCP headers). The receiver’s kernel appends every arriving byte to one buffer, and a read returns whatever is there.

Enlarge figure

The application writes three messages. TCP delivers the same bytes in whatever chunks happen to be ready, so a read can hold one and a half messages. UDP keeps each datagram whole. With a length field before each message, a TCP reader can find the boundaries again.

Every protocol on TCP marks its own boundaries, in one of three ways:

• A length prefix: DNS over TCP uses 2 bytes, HTTP/2 a 24-bit length on every frame (RFC 9113). The most robust choice; we build it below.
• A delimiter: a byte that cannot appear inside a message, such as the newline in newline-delimited JSON or the blank line that ends HTTP/1.1 headers.
• A library that does it: WebSocket, gRPC and database drivers frame for you. The bug appears when someone opens a raw socket beside them.

A length prefix needs one more defence: the length comes from the network. Without a maximum, one corrupt or hostile header claiming 4 GB makes your server try to allocate 4 GB.

UDP has the opposite problem. A datagram keeps its boundary, but it has a size limit: 65,507 bytes of payload over IPv4, and in practice much less, because anything larger than the path’s MTU (1,500 bytes on most links) is split into IP fragments, and losing any fragment loses the whole datagram. Many firewalls drop fragments outright. The largest unfragmented payload on Ethernet is 1,500 − 20 − 8 = 1,472 bytes, and QUIC stays at or below about 1,200, the size it assumes every path carries.

Loss repair: fast retransmit, timers, and the one-second spike

When a segment in the middle of a busy stream is lost, the segments after it still arrive. The receiver cannot acknowledge past the hole, so it repeats its last ACK for each of them. Five 1,000-byte segments, the second one lost:

Arrives at the receiver	ACK it sends	What the sender concludes
bytes 1–1,000	ACK 1001	all fine
(1,001–2,000 is lost)
bytes 2,001–3,000	ACK 1001, duplicate 1	maybe two packets swapped places
bytes 3,001–4,000	ACK 1001, duplicate 2	still maybe reordering
bytes 4,001–5,000	ACK 1001, duplicate 3	lost: resend 1,001–2,000 now

Networks reorder a packet or two often enough that one duplicate ACK proves nothing; three are taken as “lost, not late”. The sender resends without waiting for any timer: fast retransmit, roughly one round trip after the loss. With SACK (selective acknowledgement, on by default everywhere) each duplicate ACK also lists the later ranges that did arrive (“SACK 2001–5000”), so only the hole is resent.

Fast retransmit needs later segments to produce those duplicate ACKs. When the lost segment is the last of a burst, or the connection is quiet, nothing comes back, and only the retransmission timeout (RTO) is left: a timer on the oldest unacknowledged segment that resends it when it fires. Too short, and it resends segments that were only late; too long, and every tail loss waits for nothing. So the sender sets it from what it measures.

Each ACK gives one RTT sample: the time from sending a segment to its ACK. One sample is noisy, so RFC 6298 keeps two running averages: the smoothed RTT (SRTT) moves 1/8 of the way towards each new sample, and the RTT variation (RTTVAR) moves 1/4 of the way towards how far that sample was from SRTT. The timer is the average plus four times the typical deviation:

RTO = SRTT + 4 × RTTVAR

The first sample sets SRTT to itself and RTTVAR to half of it. Three steady samples, then a spike:

Sample	SRTT	RTTVAR	SRTT + 4 × RTTVAR
100 ms	100	50	300 ms
100 ms	100	37.5	250 ms
100 ms	100	28.1	213 ms
180 ms	110	41.1	274 ms

A steady path pulls the timer down towards the RTT; jitter pushes it up, so a merely slow ACK is not mistaken for a loss.

Three rules surround it. Before any measurement, the RTO is 1 second. It has a floor: the RFC asks for at least 1 s, which would lift every row above to 1 s; Linux instead never lets the 4 × RTTVAR term count for less than 200 ms, so its RTO is at least SRTT + 200 ms. And it doubles after every expiry, so a dead path is not flooded. A retransmitted segment is never used to measure the round trip, because you cannot tell which copy the ACK was for (Karn’s rule).

A worked example. On a steady 100 ms path, Linux’s RTO is therefore about 100 + 200 = 300 ms. The last segment of a response is lost: the sender waits 300 ms and resends. If that copy is lost too, it waits 600 ms, then 1.2 s. One unlucky segment has cost over two seconds on a path where a round trip is a tenth of a second. Linux softens the first case with a tail loss probe that resends after about two round trips instead of the full RTO (RFC 8985), but repeated losses still back off.

A lost SYN is worse, because there is no measurement yet: the client waits the initial 1 second. On older Linux kernels the SYN retries then double (1, 2, 4 s), which is why latency graphs so often show a cluster of requests at almost exactly 1 s and 3 s above normal. Since 2023 (Linux 6.5), the first four SYN retries are spaced 1 s apart instead (net.ipv4.tcp_syn_linear_timeouts = 4, as of October 2026, kernel docs), so the clusters sit at 1, 2, 3 and 4 s. Either way, a latency that is a suspiciously round number of seconds is almost always a timer.

TCP is also patient past the point of usefulness. Linux keeps retrying a connect() for about 2 minutes and data on an open connection for about 15 minutes (tcp_retries2 = 15) before it reports an error. Your application needs its own, much shorter timeouts; TCP_USER_TIMEOUT caps the kernel’s patience per socket.

Head-of-line blocking: what “in order” costs

When a segment is lost, the segments behind it arrive and wait in the receiver’s buffer. The application cannot have byte 2,001 until it has byte 2,000. This is head-of-line blocking, and it turns a single loss into a stall for everything sent in the next round trip or so.

For a file download that stall is a rounding error. For anything with a deadline it is the whole problem. The simulation sends 50 small messages a second, one every 20 ms like voice frames, over a link with a 100 ms round trip and 4% loss. Each bar is one message’s delay from send to the application; the dashed line is 150 ms, roughly the most one-way delay a conversation tolerates before it feels awkward (ITU-T G.114). Press “Lose the next packet” to make a loss happen on demand.

SimulationTCP on a lossy link: nothing lost, everything waits

PlayStepReset

What to watch for. When one packet is lost, the packets behind it still arrive, but the status line shows them held in the buffer until the retransmission fills the gap; then they reach the app all at once. One loss becomes a staircase of amber bars over the deadline. Raise the round-trip time and the staircase gets taller and wider; lower it to 40 ms and most losses stay under the line.

Every loss looks the same: one tall bar for the lost message, then a descending staircase of bars for messages that had already arrived and were held behind the gap, all released at the same instant. That staircase is the “freeze, then fast-forward” of a voice call over TCP.

The staircase in numbers. Message n is lost at time 0. Messages n+1, n+2 and n+3 are sent at 20, 40 and 60 ms and arrive 50 ms later; the third duplicate ACK reaches the sender at 60 + 100 = 160 ms, and the resend arrives at 210 ms. Messages n+1 to n+7 arrived between 70 and 190 ms and are all released at 210 ms. So n is 210 ms late, n+1 190 ms, n+2 170 ms and n+3 exactly 150 ms: one loss pushes three messages past the deadline and drags four more towards it. Over a long run at 4% loss, the model shows 17% of messages late, four times the loss rate, because now and then a resend is lost too and only the timer recovers it. At a 200 ms round trip it is 42%.

UDP on the same link has no staircase: every message that arrives is 50 ms old, and 4% never arrive. For voice that is the better trade, because a codec can hide a missing 20 ms frame and nobody can hide a 200 ms freeze.

HTTP/2 inherits this. HTTP/2 multiplexes many requests as streams over one TCP connection, which fixed HTTP/1.1’s queueing at the HTTP layer and moved it down a level. The streams are independent to HTTP, but they share one TCP byte sequence, so one lost segment stalls every stream on the connection, including ones whose data all arrived. On a lossy mobile link, a browser’s six HTTP/1.1 connections can beat one HTTP/2 connection, because a loss on one of six stalls only a sixth of the traffic. That is the problem QUIC was built to remove, and we come back to it with the same simulation below.

Flow control: the receive window and the bandwidth-delay product

Every ACK also carries the receive window: how many more bytes the receiver’s buffer can take right now. The sender may have at most that many bytes in flight, sent but not yet acknowledged. Picture the stream as a row of bytes with the window as a frame laid over it:

Enlarge figure

With a 4,000-byte window, after ACK 2001 the sender may have bytes 2,001 to 6,000 out; byte 6,001 must wait. ACK 3001 moves the frame 1,000 bytes to the right and lets 1,000 more bytes go. This is the sliding window.

Each ACK slides the frame forward by the bytes it confirms. If the receiving application stops reading, its buffer fills, the window it advertises shrinks to zero and the sender stops without losing a byte. That is TCP’s built-in backpressure, and it is why the UDP demo above lost 98% while a TCP version would have lost nothing.

The window also caps throughput at one window per round trip, however fast the link: the sender sends a full window, then waits a round trip for the ACKs that let it send more. To keep a 100 Mbit/s path 100 ms long busy, you need 100,000,000 ÷ 8 × 0.1 = 1.25 MB in flight at all times. That product of bandwidth and round-trip time is the bandwidth-delay product (BDP), the size of the “pipe”. The original 16-bit window field allows 65,535 bytes, which caps that path at 65,535 × 8 ÷ 0.1 ≈ 5.2 Mbit/s. The window scale option (RFC 7323) lifts the limit to about 1 GB, and Linux grows socket buffers automatically up to tcp_rmem’s maximum. When a transfer between two distant data centres is slow on a fast link and nothing is lost, a buffer that an application or a sysctl capped is the first suspect.

Congestion control: slow start in numbers

The receive window protects the receiver. Nothing in it protects the routers in between, and in October 1986 that nearly stopped the internet: throughput between two Berkeley sites fell by a factor of about a thousand because the network was full of retransmissions of packets dropped because the network was full (Jacobson and Karels, 1988). Their fix is still the shape of every TCP.

The sender keeps a second limit, the congestion window (cwnd): its own guess at how much the path can hold. It may have in flight only the smaller of cwnd and the receive window: with a 256 KB receive window and a cwnd of 14.6 KB, it sends 14.6 KB; once cwnd grows past 256 KB, the receiver is the limit.

Why a guess. The sender cannot see the routers. A router sends packets out of each link at that link’s speed and holds any excess in a queue; when the queue is full, it drops new arrivals. A 50 Mbit/s link forwards about 4,170 full-size packets a second. If senders push 5,000 a second, its queue grows by about 830 packets a second, a 1,000-packet queue is full in 1.2 s, and from then on every extra packet is dropped. The sender sees only the effects: ACKs come back later as the queue grows, and one goes missing when a packet is dropped. So a classic TCP probes: it raises cwnd while everything is acknowledged and cuts it when a loss shows that a queue overflowed.

Slow start. A new connection starts at 10 segments, about 14.6 KB (RFC 6928). Every segment acknowledged raises cwnd by one segment, so 10 segments acknowledged make a window of 20: the window doubles every round trip.

Round trip	1	2	3	4
cwnd (segments)	10	20	40	80
Sent so far	14.6 KB	43.8 KB	102 KB	219 KB

After k round trips it has sent 10 × (2^k − 1) segments, so a response of S bytes on a new connection needs about log₂(S ÷ 14.6 KB + 1) round trips: 3 for 100 KB, 7 for 1 MB, 10 for 10 MB, whatever the bandwidth. The rest of the algorithm:

• A loss means congestion. The sender assumes a queue overflowed and cuts cwnd: to 70% under CUBIC (RFC 9438), the default in Linux, Windows and Apple’s stacks, or to 50% under the older Reno.
• Then it probes again, more carefully than in slow start (below).
• Idle resets it. If a connection sits idle for longer than one RTO, Linux shrinks cwnd back towards 10 segments before sending again, because what it learned about the path may be stale (tcp_slow_start_after_idle, on by default, from RFC 2861).

How it climbs back. Reno adds one segment per round trip: after a cut from 100 segments to 50 on a 100 ms path, it needs 50 round trips, 5 s, to get back. CUBIC grows by elapsed time instead, along a cubic curve centred on Wmax, the window at which the last loss happened. With RFC 9438’s constants and Wmax = 100 segments:

Seconds after the loss	0	2	4.2	6	8
cwnd (segments)	70	96	100	102	122

Fast while well below the old limit, nearly flat around it, where the last loss happened, then faster again to look for more.

Here is one connection sending the same response again and again. Each bar is one round trip: how many segments the sender had in flight. The dashed line is the pipe, the BDP; the bottleneck can queue one more pipe’s worth, and anything beyond that is dropped.

SimulationWhy a fast link does not make a new connection fast

PlayStepReset

What to watch for. The first response climbs 10, 20, 40, 80, 160, 320 and takes seven round trips; the next one, on the now warm connection, takes two. Raise the bandwidth to 1000 Mbit/s: the first response barely gets faster. Set the idle gap to 1000 ms, longer than the retransmission timeout, and the warm window collapses before every response. Then set the size to 5000 KB and loss to 1%: the window never gets near the pipe, and throughput sits at a few Mbit/s on a 50 Mbit/s link.

The defaults are a 1 MB response, 100 ms away, on 50 Mbit/s. The first response climbs 10, 20, 40, 80, 160, 320 and finishes in the seventh round trip, 713 ms. The link alone needs 260 ms (one round trip plus 164 ms of bytes), which is what the second response, on the now warm connection, takes. Slide the bandwidth to 1,000 Mbit/s and the first response takes 701 ms: twenty times the bandwidth bought 12 ms. On a new connection, round-trip time is the only number that matters.

Then slide the idle gap to 1,000 ms, longer than the 300 ms RTO. The window halves three times during the gap, falls from 640 segments to 80, and every warm response takes 429 ms instead of 260. This is the trap of “we reuse connections, so slow start is not our problem”: a connection that idles for a second between bursts pays part of slow start again on every burst. Servers that hold long-lived connections (HTTP/2, gRPC, database proxies) often turn tcp_slow_start_after_idle off for exactly this reason.

Loss caps throughput, not bandwidth. Set the response to 5,000 KB and random loss to 1%. The window never gets near the pipe: each loss cuts it, and it climbs back by only a few segments per round trip. The sim settles at about 1.6 Mbit/s on a 50 Mbit/s link.

The classic model of this is the Mathis formula (Mathis et al., 1997), and it follows from Reno’s sawtooth. The window climbs by one segment per round trip to a peak W, one loss halves it to W/2, and it climbs again. One cycle lasts W/2 round trips and sends about (3/8)W² segments with one loss, so the loss rate is p = 1 ÷ ((3/8)W²). At p = 1%, (3/8)W² = 100 gives W ≈ 16: the window swings between 8 and 16 segments and averages about 12 per 100 ms round trip, 12 × 1,460 × 8 ÷ 0.1 ≈ 1.4 Mbit/s, whatever the link. Solved in general, the long-run rate is about (MSS ÷ RTT) × 1.22 ÷ √p: halve the round trip and the rate doubles; cut the loss to 0.1% and it rises about three times. This is why a long-distance transfer over a slightly lossy link is slow even on a huge pipe.

Two refinements matter in practice. Linux’s CUBIC uses HyStart to leave slow start before the first loss, by watching round-trip times rise as the queue fills, so it rarely overshoots as hard as the sim’s 5 MB run does (netdev 0x1A, 2026). And Google’s BBR stops treating loss as the signal at all: it measures the bottleneck bandwidth and the minimum round trip and paces packets to match, which holds up far better on links with random loss (ACM Queue, 2016). BBR’s third version is still an IETF draft as of October 2026 (draft-ietf-ccwg-bbr).

UDP has none of this, which is fine for one DNS question and harmful for a program that sends megabytes as fast as the socket accepts them. RFC 8085, the IETF’s rules for UDP applications, says anything sending more than a little over UDP must control its rate in a way comparable to TCP.

Nagle and delayed ACK: the 44 ms request

Back to the first puzzle. It comes from two features that are each sensible alone. IP and TCP headers take 40 bytes, so a 1-byte write, a keystroke over SSH, would cost a 41-byte packet. Nagle’s algorithm (RFC 896), on by default, holds back small writes while any sent data is unacknowledged, and sends them together when the ACK comes. A bare ACK is also a 40-byte packet, so delayed ACK makes the receiver wait before acknowledging, hoping to piggyback the ACK on its reply: at least 40 ms on Linux, up to 200 ms on Windows. Now the request sent as two writes:

Time	Client	Server
0 ms	writes the header; it goes out at once	has the header, cannot reply yet, so holds the ACK
0 ms	writes the body; Nagle holds it, the header is unacknowledged	waits for the body
~40 ms		delayed-ACK timer fires; sends the bare ACK
~40 ms	ACK arrives; sends the body	has the body; replies

Each side waits for the other until the timer fires. nagle.py and nagle.mjs time 50 such requests on loopback:

python nagle.py (Linux; node nagle.mjs gave the same)

nagle on, two writes     median  44.01 ms   worst  48.20 ms
TCP_NODELAY, two writes  median   0.06 ms   worst   0.16 ms
nagle on, one write      median   0.06 ms   worst   0.10 ms

Seven hundred times slower, from one extra write. The better fix is the last line: build the whole message in memory and write it once, as the framing code below does. TCP_NODELAY turns Nagle off for the socket. Most HTTP and database libraries set it; Node’s http.createServer does by default since version 18, but a plain net.Socket does not (Node docs). Windows’ loopback path does not show the stall, so measure on the operating system you deploy to.

Closing: TIME-WAIT, CLOSE_WAIT and running out of ports

A TCP connection is named by four numbers, the four-tuple: source address, source port, destination address, destination port. The kernel uses it to find which socket an arriving segment belongs to, so no two live connections may share one. For a client calling one database, three of the four never change:

Source address	Source port	Destination address	Destination port
10.0.1.7	51000	10.0.2.9	5432
10.0.1.7	51001	10.0.2.9	5432

Only the source port tells these two connections apart, and the kernel picks it for you from a range of temporary ephemeral ports. That is the limit this section ends on.

The side that closes first does not free its four-tuple at once; it enters TIME-WAIT for twice the maximum segment lifetime (MSL), the longest a segment is assumed to survive in the network before a router discards it. That wait does two jobs: if the final ACK is lost, the peer resends its FIN and still gets an answer, and a delayed duplicate segment from this connection cannot be mistaken for data in a new connection with the same four numbers. The RFC’s MSL is 2 minutes, so TIME-WAIT lasts 4; Linux uses a fixed 60 seconds.

Which side holds it is decided by who calls close() first. time_wait.py makes 500 requests three ways and counts the TIME-WAIT sockets left behind:

python time_wait.py (Linux; node time-wait.mjs gave the same)

client closes first, 500 connections     TIME_WAIT {'client side': 500, 'server side': 0}
server closes first, 500 connections     TIME_WAIT {'client side': 0, 'server side': 500}
one connection reused for 500 requests   TIME_WAIT {'client side': 1, 'server side': 0}

ephemeral ports 32768-60999: 28232. Linux keeps TIME_WAIT for 60 s, so one client
can open at most about 470 new connections a second to one server address and port.

TIME-WAIT on a server costs a little memory and nothing else, because the server’s port is fixed and each four-tuple differs by the client’s address and port. On a client it is a real limit. Each outgoing connection takes an ephemeral port from 32,768–60,999 on Linux by default (28,232 ports, as of October 2026), and a port stays taken for 60 s after the client closes. Per destination address and port:

maximum new connections per second ≈ 28,232 ÷ (connection lifetime + 60 s) ≈ 470

So the service from the opening, at 500 new connections a second, needs 500 × 60 = 30,000 ports, runs out after about 56 seconds, and connect() fails with EADDRNOTAVAIL, “cannot assign requested address”, from code that never chose a port. The typical culprit is an HTTP client created per request, or a proxy calling one backend without keep-alive.

The fixes, best first:

1. Reuse connections. A pool with keep-alive turns 500 connections a second into a handful of long-lived ones; the third line of the output is the whole argument.
2. Spread over more four-tuples: more backend addresses or ports, or more source addresses (Cloudflare describes the socket options for this: Cloudflare, 2022).
3. Widen ip_local_port_range or enable tcp_tw_reuse (default 2: loopback only), which lets a new outgoing connection take a port in TIME-WAIT, relying on TCP timestamps to reject old segments. Its cousin tcp_tw_recycle broke clients behind NAT and was removed in Linux 4.12.

Do not “fix” TIME-WAIT by closing with SO_LINGER set to 0: that sends an RST instead of a FIN, skips the safety TIME-WAIT exists for, and can throw away data the peer has not read yet.

CLOSE_WAIT is the opposite state, and it is always your bug. It means the peer closed and your application never called close(), usually on an error path. The kernel will wait for ever, and a growing count in ss -tan is a file-descriptor leak in progress.

Idle connections: the middleboxes that forget them

Between your two kernels sit boxes that keep a table entry for every flow they pass. The commonest is a NAT (network address translation): many machines with private addresses share one public address, so the NAT rewrites the source of each outgoing packet and remembers the swap:

Inside	Rewritten to	Talking to
10.0.1.7:51000	198.51.100.7:40001	203.0.113.5:5432

A reply addressed to 198.51.100.7:40001 reaches 10.0.1.7:51000 only because that row exists. Cloud NAT gateways, load balancers, stateful firewalls and Linux’s conntrack (the connection-tracking table behind iptables NAT and Kubernetes service routing) all keep rows like this one. Each row has an idle timer, and when it expires the row is deleted without telling either end. The next packet on that connection is dropped silently, or answered with an RST. TCP’s own keep-alive, which could keep the row fresh, is off unless you enable it, and once enabled Linux waits 2 hours before its first probe. Every one of these timeouts is shorter (as of October 2026):

Box	Idle timeout	What happens next
AWS NAT gateway	350 s	RST on the next packet (AWS)
AWS Network Load Balancer	TCP 350 s (60–6,000), UDP 120 s	silent drop (AWS)
Azure Load Balancer	4 min (up to 100)	silent drop unless “TCP reset on idle” is on (Microsoft)
Google Cloud NAT	TCP 20 min, UDP 30 s	port mapping freed (Google Cloud)
Linux conntrack	TCP 5 days, UDP 30 s (120 s once two-way)	entry deleted (kernel docs)

The classic symptom: a service keeps a pool of database connections through a NAT gateway; traffic is near zero overnight; the first queries every morning fail with “connection reset”, then everything works. Make the pool discard connections idle for longer than the shortest timeout on the path, or send keep-alives below it (TCP_KEEPIDLE, or an application ping). Retry idempotent requests once on a reset on a pooled connection, because that is the common case of a stale entry. The mirror image, a backend that closes idle connections before its load balancer does and causes 502s, is on the proxy and load balancer page.

UDP has no FIN, so these boxes can only guess when a UDP flow is over, and they guess short: RFC 4787 asks NATs to keep a UDP mapping at least 2 minutes, but QUIC’s designers found that something every 30 seconds is needed in practice (RFC 9000 §10.1.2).

Enlarge figure

A NAT keeps a row for each UDP flow and deletes it after a silence. With a small keep-alive every 25 seconds the server's reply gets back in; after 30 seconds of silence the row is gone and the reply is dropped without a word.

That is why voice and game clients send a tiny packet every 15 to 30 seconds even when nobody speaks; RFC 8085 asks for no more than one every 15 seconds.

Why QUIC moved to UDP

If TCP’s waits hurt, why not change TCP? Because TCP lives in two places that are very hard to change. It is in the operating system kernel, so a new feature reaches users only as fast as phones and servers update their kernels. And it is inspected by millions of firewalls, NATs and load balancers that parse its headers and drop what they do not recognise. TCP Fast Open, which lets data ride on the SYN (RFC 7413), is the cautionary tale: boxes that drop SYNs carrying data kept it from wide use. The network has frozen around what TCP looked like; engineers call this ossification.

UDP is the one other transport every network passes. QUIC (RFC 9000, 2021) runs inside the application on top of UDP, rebuilds TCP’s machinery there, and changes the parts that hurt:

• Independent streams. One connection carries many streams, each ordered on its own. A loss stalls only its own stream: head-of-line blocking moves from the connection to the stream.
• Encryption built in. TLS 1.3 is part of the handshake, so transport and encryption are set up in one round trip; a returning client can send data in its very first packet (0-RTT).
• Connection IDs. A connection is not named by the four-tuple, so it survives a phone moving from Wi-Fi to mobile data, and a NAT rebinding its port.
• Nearly every header byte encrypted, so middleboxes cannot learn to depend on it and freeze it again.

Enlarge figure

Round trips spent before the first byte of a request can reach the server, at 100 ms each. QUIC sets up transport and encryption together, and a returning client can send data in its first packet.

Here are TCP, QUIC and UDP on identical traffic and identical losses. The TCP lane is also what HTTP/2’s streams get, since they share one TCP connection.

SimulationTCP, QUIC and UDP on identical losses

PlayStepReset

What to watch for. One loss makes several TCP bars late, because everything behind the gap waits. QUIC repairs the same loss, but only the lost message’s own stream waits. Set QUIC streams to 1 and its chart becomes identical to TCP’s: streams, not UDP, are what removes the blocking. UDP is never late and loses exactly the dropped messages.

At the defaults, TCP settles at about 17% of messages late, QUIC with four streams at about 6%, and UDP loses exactly the 4% the network dropped. Now set QUIC streams to 1: its chart becomes identical to TCP’s, 17%. QUIC’s gain comes entirely from splitting independent data across streams. A single ordered stream over QUIC (one video, one file, one voice channel sent as a reliable stream) blocks behind a loss exactly as TCP does; voice over QUIC uses unreliable datagrams for that reason.

QUIC has real costs. It runs in user space, without the decades of kernel and network-card offloads TCP has, so it uses more CPU per byte: Google’s 2017 paper put it at about 3.5 times TLS over TCP at the time (Google, SIGCOMM 2017). Some corporate networks block UDP, so every HTTP/3 client must be able to fall back to TCP. And 0-RTT data can be replayed by an attacker, so it is only safe for idempotent requests. Even so, HTTP/3 carried about 21% of the requests Cloudflare saw in 2025 (Cloudflare Radar).

What you must rebuild if you choose UDP

Choosing UDP directly, for a game server, a media path or a metrics pipe, means taking on the duties TCP would have done. RFC 8085 is the checklist; here are the parts that bite:

• Size: stay at or under about 1,200 bytes per datagram unless you discover the path MTU.
• Rate: pace your sends and back off when you see loss. The loopback demo above needed nothing more than pacing to go from 98% loss to none.
• Only the reliability you need: sequence numbers to detect loss and reordering, and acknowledgements with timers only for data that must arrive. A game resends inputs but never old position snapshots, because the next snapshot replaces them.
• Duplicates: a retransmission whose original was only late arrives twice; make applying a message twice harmless.
• NAT keep-alives every 15 to 30 seconds, as above.
• Spoofing and amplification. With no handshake, the source address on a datagram is a claim, not a fact. If your reply is much larger than the request, anyone can aim your server at a victim by forging the victim’s address. QUIC’s rule: before an address is verified, send at most three times the bytes received from it.

A real outage: 1.35 Tbps from a UDP amplifier

On 28 February 2018, GitHub received what was then the largest attack publicly reported, 1.35 Tbps at its peak. Attackers sent small requests to tens of thousands of memcached servers left open on UDP port 11211, forging GitHub’s address as the source, and each server sent its much larger reply to GitHub: amplification of up to 51,000 times. GitHub was unreachable for about five minutes and intermittently for nine, until its traffic moved behind DDoS protection (GitHub incident report).

One more rule: never carry TCP inside a TCP tunnel. When a packet is lost, the inner and outer connections both retransmit and both back off, with timers that fight each other. This is why WireGuard runs only over UDP (WireGuard), and why VPNs that fall back to TCP feel so much worse on a lossy network.

Choosing between them

Read the table as “what you get without writing code”.

	TCP	UDP	QUIC (on UDP)
Setup before data	1 round trip; 2 with TLS 1.3	none	1 with encryption; 0 when resuming
A loss delays	everything behind it	nothing	only its own stream
Message boundaries	none: a byte stream	kept, up to ~1,200 bytes in practice	none within a stream
Rate control	in the kernel	yours to build	in the library
Survives a network change	no	if your protocol does	yes, with connection IDs
State in middleboxes	until close, or idle timeout	idle timeout only, often 30 s	idle timeout only
Typical uses	APIs, databases, HTTP/1.1 and HTTP/2, SSH	DNS, voice and video media, games, VPN tunnels, metrics	HTTP/3, mobile apps on lossy networks

In practice you rarely pick TCP or UDP directly; you pick a protocol that sits on one of them.

1. Every byte must arrive, in order: payments, API calls, queries. TCP, through HTTP, gRPC or your driver, over pooled connections. This is the default for almost everything.
2. Late data is worthless: voice, video, game state, live sensor readings. UDP, through a library that adds what you need (WebRTC for media, a game networking library).
3. One tiny question, one tiny answer: DNS, time sync. UDP, with a retry, and TCP for answers too big for one datagram.
4. TCP’s guarantees, but lossy networks, many parallel streams, or roaming clients: HTTP/3 or QUIC through a library or your CDN. Do not write QUIC yourself.

Build it yourself

Two pieces: framing on a TCP stream, and reliability on UDP. Standard library only; every file is in examples/tcp-vs-udp, and every output on this page came from running them in October 2026.

Step 1: a length prefix on a TCP stream

Four bytes of length, then the message. The reader reads exactly 4 bytes, then exactly that many more, however TCP happens to chunk them:

Python

framing.py

# runnable version: examples/tcp-vs-udp/python/framing.py
HEADER = struct.Struct(">I")  # 4-byte unsigned length, network byte order
MAX_MESSAGE = 1 << 20  # refuse anything over 1 MB: a bad length must not make us allocate 4 GB


def send_message(sock: socket.socket, message: dict) -> None:
    body = json.dumps(message).encode()
    sock.sendall(HEADER.pack(len(body)) + body)  # one buffer: header and body leave together


def recv_exactly(sock: socket.socket, n: int) -> bytes:
    """Read exactly n bytes. recv() may return fewer than asked for, so loop."""
    buf = bytearray()
    while len(buf) < n:
        chunk = sock.recv(min(n - len(buf), 65536))
        if not chunk:  # the peer closed: clean only if we were between messages
            raise ConnectionClosed("peer closed mid-message" if buf else "peer closed")
        buf += chunk
    return bytes(buf)


def recv_message(sock: socket.socket) -> dict:
    (length,) = HEADER.unpack(recv_exactly(sock, HEADER.size))
    if length > MAX_MESSAGE:
        raise ValueError(f"message of {length} bytes is over the {MAX_MESSAGE}-byte limit")
    return json.loads(recv_exactly(sock, length))

JavaScript

framing.mjs

// runnable version: examples/tcp-vs-udp/js/framing.mjs
const MAX_MESSAGE = 1 << 20; // refuse anything over 1 MB: a bad length must not make us buffer 4 GB

function encode(message) {
  const body = Buffer.from(JSON.stringify(message));
  const header = Buffer.alloc(4);
  header.writeUInt32BE(body.length);
  return Buffer.concat([header, body]); // one buffer: header and body leave together
}

/** Turns a stream of chunks into whole messages. Call push() with each chunk. */
class MessageReader {
  buffer = Buffer.alloc(0);

  constructor(onMessage) {
    this.onMessage = onMessage;
  }

  push(chunk) {
    this.buffer = Buffer.concat([this.buffer, chunk]);
    while (this.buffer.length >= 4) {
      const length = this.buffer.readUInt32BE(0);
      if (length > MAX_MESSAGE) throw new Error(`message of ${length} bytes is over the ${MAX_MESSAGE}-byte limit`);
      if (this.buffer.length < 4 + length) return; // not all here yet: wait for more chunks
      this.onMessage(JSON.parse(this.buffer.subarray(4, 4 + length)));
      this.buffer = this.buffer.subarray(4 + length);
    }
  }
}

With the same traffic and the same busy server, all six messages parse, the leaderboard included. Note the single sendall of header plus body: two writes would invite the Nagle stall above.

Step 2: reliability on top of UDP

Now the other direction: enough of TCP’s machinery on UDP to deliver 100 messages in order over a bad link. A wrapper drops 10% of datagrams each way and delays the rest by 20 to 30 ms, so they also reorder. The receiver buffers early arrivals and delivers in order; the sender keeps a timer per message and doubles it on each resend. The full file also sets the timeout with RFC 6298’s formula and Karn’s rule.

Python

reliable_udp.py

# runnable version: examples/tcp-vs-udp/python/reliable_udp.py
# Receiver.datagram_received(), after acking every copy:
        if seq < self.expected or seq in self.buffer:
            return  # a duplicate, from a retransmission whose original was not lost after all
        self.buffer[seq] = time.perf_counter()
        while self.expected in self.buffer:  # deliver in order; a gap holds everything behind it
            self.waits.append(time.perf_counter() - self.buffer.pop(self.expected))
            self.delivered.append(self.expected)
            self.expected += 1

# Sender:
    def transmit(self, seq: int, retries: int) -> None:
        self.link.send(DATA.pack(seq) + bytes(200), self.peer)
        if not self.reliable:
            return
        timeout = self.rto * 2**retries  # back off: each timeout doubles the wait
        timer = asyncio.get_running_loop().call_later(timeout, self.on_timeout, seq)
        self.in_flight[seq] = {"sent": time.perf_counter(), "retries": retries, "timer": timer}

JavaScript

reliable-udp.mjs

// runnable version: examples/tcp-vs-udp/js/reliable-udp.mjs
// receiver, after acking every copy:
    if (seq < state.expected || state.buffer.has(seq)) return; // a duplicate from a retransmission
    state.buffer.set(seq, performance.now());
    while (state.buffer.has(state.expected)) {
      // deliver in order; a gap holds everything behind it
      state.waits.push(performance.now() - state.buffer.get(state.expected));
      state.buffer.delete(state.expected);
      state.delivered.push(state.expected++);
    }

// sender:
  function transmit(seq, retries) {
    const data = Buffer.alloc(204);
    data.writeUInt32BE(seq);
    linkSend(socket, data, peerPort);
    if (!reliable) return;
    const timer = setTimeout(() => { // back off: each timeout doubles the wait
      s.retransmissions++;
      transmit(seq, retries + 1);
    }, s.rto * 2 ** retries);
    inFlight.set(seq, { sent: performance.now(), retries, timer });
  }

The same code runs three ways: plain UDP, stop-and-wait (a window of 1: send one message, wait for its ACK), and a sliding window of 16:

python reliable_udp.py (Linux)

link: 10% loss each way, 20-30 ms delay each way

1. plain UDP, no acks
    91/100 arrived, 9 lost, 50 arrived after a later one
2. stop-and-wait (window of 1)
    100/100 delivered, in order: True, in 6.71 s
    18 retransmissions; final RTO 66 ms
3. sliding window of 16
    100/100 delivered, in order: True, in 0.89 s
    28 retransmissions; final RTO 65 ms
    76 messages arrived but waited for an earlier one; longest wait 206 ms

Plain UDP lost 9 messages and delivered half out of order. Stop-and-wait paid one round trip per message, so 100 messages took almost 7 seconds: throughput is one window per round trip, with a window of one. The sliding window was seven times faster. The last line is head-of-line blocking again, now in our own code: 76 of 100 messages arrived and waited for an earlier one. This is roughly TCP without congestion control or decades of tuning, which makes it a good way to understand TCP and a bad thing to ship.

See it on your own machine

ss -tan state time-wait | wc -l counts TIME-WAIT sockets on Linux, and ss -tan state close-wait lists the leaks. ss -ti shows each connection’s cwnd, rtt and retransmissions. On a test box, tc qdisc add dev eth0 root netem loss 3% delay 100ms turns a good link into a bad mobile one in one command.

How real systems do it

Voice and games ride on UDP, with only the reliability they need. Discord described in 2018 how its native clients send voice over UDP to its own servers, with periodic pings to keep NATs and firewalls open, serving 2.6 million concurrent voice users at the time (Discord engineering). Fast multiplayer games send frequent snapshots of the world and never resend an old one, because the next snapshot replaces it (Gaffer on Games).

HTTP/3 pays off where loss and distance are high. Google’s 2017 QUIC paper reported 15–18% fewer YouTube rebuffers and Search latency down 3.6–8%, with the biggest gains on slow, distant paths, exactly where head-of-line blocking and handshake round trips cost most (Google Research). Uber moved its rider and driver apps to QUIC in 2019 and reported tail latency down 10–30% on mobile networks (Uber engineering).

Gotchas and war stories

The five-second DNS lookups in Kubernetes

Around 2017 to 2019, many Kubernetes clusters saw DNS lookups that occasionally took exactly 5 seconds. The C library sends the A and AAAA queries in parallel from one UDP socket, and a race in Linux’s connection tracking sometimes dropped one of the pair. Nothing retries a lost UDP packet except the application, so the client waited out the resolver’s 5-second timeout. The kernel races were fixed in Linux 4.19 and 5.0 (kubernetes/kubernetes#56903).

TCP’s own machinery is attack surface. In June 2019 Netflix disclosed “SACK Panic” (CVE-2019-11477): a peer that announced a tiny MSS of 48 bytes and then sent crafted SACKs made Linux split its retransmission queue into so many pieces that a 16-bit counter overflowed and the kernel panicked, on every version since 2.6.29. The mitigations, before patched kernels shipped, were to disable SACK or drop connections with a very small MSS (Red Hat). Every feature in the kernel’s TCP is code an untrusted peer can drive.

From the trenches

Teams usually discover their transport behaviour during an incident, not a design review. Three numbers worth knowing for every service you own: how many new connections it opens per second to each dependency, the shortest idle timeout on each path, and the RTO it would face on its longest path. Each one predicts an outage described on this page.

Trade-offs and when not to use it

TCP costs round trips before and during every new connection, a stall behind every loss, kernel timers your application must cut short, and a port held for a minute after every close. Avoid it when late data is useless (live voice and video, game state), when the whole exchange is one tiny question, or when you need broadcast or multicast on a local network.

UDP costs every guarantee you need, rebuilt by you, plus a TCP fallback for networks that block it. Avoid it when all your data must arrive in order: you would end up rebuilding TCP, and the first home-made TCP is always worse than the real one. If you need TCP’s guarantees without its head-of-line blocking or handshakes, use QUIC through a well-tested library.

Interview corner

A weak answer recites “reliable versus fast”. A strong one names the mechanism behind each guarantee and what it costs when packets are lost.

1. “TCP or UDP for a voice chat app?” UDP for audio, because head-of-line blocking turns each loss into a stall of at least one round trip and late audio is useless; then list what you must add (sequence numbers, a jitter buffer, rate control, NAT keep-alives, encryption) and say you would use WebRTC. Chat and settings stay on TCP.
2. “A 1 MB response is slow for distant users though bandwidth is fine. Why?” Slow start: about seven round trips on a new connection, plus two handshakes. Reuse and warm connections, serve closer to the user.
3. “Why does HTTP/3 run on UDP?” TCP is ossified by kernels and middleboxes; QUIC rebuilds reliability and congestion control in user space with independent streams (no cross-stream head-of-line blocking), a combined transport and TLS handshake, and connection IDs for migration. Bonus: on a single stream it blocks exactly like TCP.
4. “Your service logs EADDRNOTAVAIL under load.” Ephemeral ports held in client-side TIME-WAIT; about 470 new connections a second per destination. Pool connections first.

Check yourself

1. A 300 KB response, a new HTTPS connection with TLS 1.3, 100 ms round trip, a 1 Gbit/s link, no loss. Roughly how long until the last byte arrives?

300 KB is about 206 segments. Slow start sends 10, 20, 40, 80 (150 so far) and the remaining 56 in the fifth round trip, so five round trips of data. Add one for TCP and one for TLS: about seven round trips, 700 ms. The 1 Gbit/s link needs about 2.4 ms for the bytes; it barely matters.

2. A voice app sends 50 packets a second over TCP, the round trip is 120 ms, and one packet in the middle of the stream is lost. How late is the lost packet, and how many packets miss a 150 ms deadline?

The three packets after it are sent at 20, 40 and 60 ms and arrive 60 ms later; the third duplicate ACK reaches the sender at 60 + 120 = 180 ms, and the resend arrives at 240 ms. Packets sent at 20, 40, 60 and 80 ms wait until then too, with delays of 220, 200, 180 and 160 ms. Five packets miss the deadline from one loss; UDP would have lost one and delivered the rest on time.

3. A gRPC service holds one HTTP/2 connection to a backend 80 ms away. Requests come in bursts every 2 seconds and each burst is slow, though the connection is never closed. Why, and what would you change?

Between bursts the connection is idle for longer than the RTO (about 280 ms here), so Linux’s tcp_slow_start_after_idle shrinks the congestion window back towards 10 segments, and each burst pays part of slow start again. Turn tcp_slow_start_after_idle off on that host, or keep a little traffic flowing; also check that the receive window is not the limit.

4. A client makes 800 requests a second to one backend address and port, opening a new connection for each and closing it first. When does it start failing, and with what?

Each closed connection holds its ephemeral port in TIME-WAIT for 60 s, so the client needs about 800 × 60 = 48,000 ports and has 28,232. It runs out after about 28,232 ÷ 800 ≈ 35 seconds, and connect() fails with EADDRNOTAVAIL. Pool and reuse connections; spreading over more backend addresses or enabling tcp_tw_reuse only raises the ceiling.

5. Every morning the first few queries from your service fail with “connection reset”, then everything works. Its database connections go through an AWS NAT gateway and traffic is near zero overnight. What is happening?

The pooled connections sat idle for longer than the NAT gateway’s 350-second idle timeout, so it deleted their entries without telling either end, and answers the next packet on each with an RST. Make the pool discard connections idle for under 5 minutes, or enable keep-alives with an idle time below 350 s, and retry idempotent queries once on a reset.

Recap

• UDP adds ports, a checksum and message boundaries to IP. Rate, order and loss are yours.
• TCP is a reliable, ordered byte stream between kernels, or an error. Frame your messages, and confirm important work at the application level.
• Every guarantee is a wait: handshake round trips, slow start round trips, a stall behind every loss, RTO timers from 200 ms (1 s for a SYN), 40 ms Nagle stalls, 60 s of TIME-WAIT.
• On a new connection, round-trip time decides speed, not bandwidth: about log₂(size ÷ 14.6 KB) round trips of slow start. Under loss, rate ≈ (MSS ÷ RTT) × 1.22 ÷ √p.
• Reuse connections, keep them warm, and expire them before the shortest idle timeout on the path; a new connection per request runs out of ports at about 470 a second per destination.
• QUIC moves TCP’s machinery onto UDP with independent streams, a one-round-trip encrypted handshake and connection IDs. Its gain is from streams; one stream blocks like TCP.

Where to go next

• TLS and Encryption in Transit→

  The handshake that sits on top of TCP, and the round trip it adds to the timeline on this page.

  Networking & APIs
• HTTP/1.1 → HTTP/2 → HTTP/3coming soon

  How HTTP/1.1, HTTP/2 and HTTP/3 each work around TCP: parallel connections, multiplexed streams, then QUIC.

  Networking & APIs
• Connection poolingcoming soon

  The practical answer to handshakes, slow start and TIME-WAIT, and how to size and expire a pool.

  Databases
• Timeoutscoming soon

  TCP will wait minutes before giving up. How to choose application timeouts that match what users can tolerate.

  Resilience
• WebSocketscoming soon

  A long-lived, framed, two-way channel on top of TCP, the right tool for most real-time features that are not voice or video.

  Networking & APIs
• Backpressurecoming soon

  The receive window generalised: how any system tells a fast producer to slow down before something overflows.

  Messaging & async

Further reading

Specs & official docs

• RFC 768, UDP and RFC 8085, UDP usage guidelines — the whole of UDP, and the checklist for building on it.
• RFC 9293, TCP — the current TCP specification: the byte-stream rule, states and TIME-WAIT.
• RFC 6298, the retransmission timer and RFC 8985, RACK-TLP — RTO, backoff, and tail loss probes.
• RFC 6928, initial window, RFC 9438, CUBIC and RFC 2861, congestion window validation — slow start’s starting point, the default congestion control, and the restart after idle.
• RFC 7323, window scaling — windows beyond 64 KB, and the bandwidth-delay product.
• RFC 896, Nagle’s algorithm — the original small-packet problem.
• RFC 9000, QUIC and RFC 9002, QUIC loss detection — streams, connection IDs, anti-amplification, NAT keep-alive advice.
• RFC 7413, TCP Fast Open and RFC 4787, NAT behaviour for UDP — ossification in practice, and why NATs forget UDP flows.
• RFC 9113, HTTP/2 — length-prefixed frames and streams over one TCP connection.
• Linux kernel: IP sysctls and conntrack sysctls — ephemeral ports, SYN retries, RTO floor, slow start after idle, TIME-WAIT reuse, conntrack timeouts.
• AWS NAT gateway troubleshooting, AWS NLB idle timeout, Azure Load Balancer TCP reset and Google Cloud NAT tuning — the idle timeouts in the table.
• Node.js http module — the noDelay default.
• ITU-T G.114 — the 150 ms guideline for conversation.
• WireGuard: known limitations — why it does not tunnel over TCP.

Papers & engineering blogs

• Jacobson and Karels: Congestion Avoidance and Control (1988) — the 1986 congestion collapse and the ideas in every TCP since.
• Mathis, Semke, Mahdavi and Ott: The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm (1997) — why loss, not bandwidth, caps a long transfer.
• Cardwell et al.: BBR (ACM Queue, 2016) and the BBR draft — measuring the path instead of reacting to loss.
• Validation and evaluation of HyStart for Linux (netdev 0x1A, 2026) — leaving slow start before the first loss.
• Google: The QUIC transport protocol (SIGCOMM 2017) — why QUIC exists, what it improved and what it cost.
• Uber: Employing QUIC (2019) — tail latency on lossy mobile networks.
• Discord: 2.5 million concurrent voice users (2018) — voice over UDP at scale.
• GitHub: February 28th DDoS incident report (2018) — UDP amplification at 1.35 Tbps.
• Cloudflare: How to stop running out of ephemeral ports (2022) — port exhaustion and the socket options around it.
• kubernetes/kubernetes#56903 — the 5-second DNS delays from a conntrack race.
• Cloudflare Radar Year in Review 2025 — how much traffic uses HTTP/3.

Explainers

• Ilya Grigorik: High Performance Browser Networking, TCP and UDP — the classic treatment of handshakes, slow start and NAT traversal; written in 2013, before QUIC and BBR.
• Glenn Fiedler: UDP vs. TCP — the game developer’s case against TCP for real-time data.