Transport Protocols: QUIC Streams, HTTP/2 Framing, and Datagram Multiplexing

As we established in Part 2, the Supervisor's EdgeTunnelServer calls either serveQUIC or serveHTTP2 depending on the selected protocol. These two transport implementations share the same OriginProxy interface but differ fundamentally in how they multiplex traffic. QUIC gives cloudflared native stream multiplexing and datagram support; HTTP/2 piggybacks on Go's standard library but lacks datagram capabilities entirely.

This article takes you inside both transport implementations, traces the lifecycle of a QUIC stream from acceptance to origin dispatch, explains the control stream registration handshake, and then dives deep into the datagram v3 system — the sophisticated multiplexer that enables UDP and ICMP proxying over QUIC.

QUIC Connection Architecture

The quicConnection struct is the workhorse of cloudflared's transport layer. When Serve is called, it spawns three concurrent goroutine groups via an errgroup:

flowchart TD
    QC[quicConnection.Serve] --> CS[Control Stream<br/>Registration + Wait]
    QC --> AS[Accept Stream Loop<br/>HTTP/TCP requests]
    QC --> DH[Datagram Handler<br/>UDP/ICMP sessions]
    
    AS --> RS1[runStream goroutine]
    AS --> RS2[runStream goroutine]
    AS --> RS3[runStream goroutine]
    
    CS -.->|First stream opened| Edge[Edge Registration]
    
    style QC fill:#2563eb,color:#fff
    style CS fill:#dc2626,color:#fff
    style AS fill:#059669,color:#fff
    style DH fill:#f68b1f,color:#fff

The Serve method's design is elegant in its error handling: if any of the three goroutines returns, the errgroup cancels the context, which propagates shutdown to the other two. The defer q.Close() ensures the QUIC connection is cleanly closed regardless of which goroutine triggered the exit.

A subtle but important detail: the control stream is opened by cloudflared (not accepted), making it the first stream on the connection. This is a protocol convention — the edge expects stream 0 to be the control plane.

Stream Lifecycle and Cap'n Proto RPC Dispatch

When the edge sends a request to cloudflared, it opens a QUIC stream. The acceptStream loop accepts these and spawns a goroutine for each:

sequenceDiagram
    participant Edge as Cloudflare Edge
    participant Accept as acceptStream loop
    participant Run as runStream goroutine
    participant RPC as Cap'n Proto Server
    participant Handle as handleDataStream
    participant Dispatch as dispatchRequest

    Edge->>Accept: New QUIC stream
    Accept->>Run: go runStream(quicStream)
    Run->>Run: Wrap in SafeStreamCloser
    Run->>Run: Wrap in nopCloserReadWriter
    Run->>RPC: ss.Serve(ctx, noCloseStream)
    RPC->>Handle: handleDataStream(ctx, stream)
    Handle->>Handle: ReadConnectRequestData()
    Handle->>Dispatch: dispatchRequest(ctx, stream, request)
    Dispatch->>Dispatch: orchestrator.GetOriginProxy()
    Dispatch->>Dispatch: Switch on ConnectionType

In runStream, the raw QUIC stream is wrapped in two layers:

1. SafeStreamCloser — adds a write timeout and safe close semantics
2. nopCloserReadWriter — prevents the RPC handler from closing the write side of the stream prematurely

The nopCloserReadWriter is particularly clever. It uses atomic.StoreUint32/LoadUint32 to signal a close to the read side without actually closing the underlying stream. This is necessary because the fused reader/writer pattern means code in handleDataStream shouldn't be able to close the downstream write path — only runStream should do that via stream.Close().

The dispatchRequest performs the connection-type switch that routes HTTP/WebSocket requests to ProxyHTTP and TCP requests to ProxyTCP. The httpResponseAdapter translates Go's http.ResponseWriter semantics into QUIC metadata — status codes and headers are packed into Cap'n Proto metadata fields rather than being written as raw HTTP framing.

Tip: If you're debugging request failures in cloudflared, the connectResponseSent boolean tracked by both streamReadWriteAcker and httpResponseAdapter is your breadcrumb — it tells you whether the error happened before or after the response headers were sent to the edge.

Control Stream and Tunnel Registration

The control stream is cloudflared's registration channel with the edge. The controlStream struct manages the full lifecycle:

sequenceDiagram
    participant CFD as cloudflared
    participant RPC as Registration Client
    participant Edge as Cloudflare Edge

    CFD->>Edge: Open QUIC stream 0
    CFD->>RPC: RegisterConnection(auth, tunnelID, connOptions)
    RPC->>Edge: RPC call
    Edge-->>RPC: RegistrationDetails{UUID, Location}
    RPC-->>CFD: Success
    CFD->>CFD: connectedFuse.Connected()
    CFD->>CFD: SendLocalConfiguration (if conn 0 + local config)
    
    Note over CFD,Edge: Blocking wait...
    
    alt Graceful Shutdown
        CFD->>CFD: gracefulShutdownC closed
        CFD->>RPC: GracefulShutdown(gracePeriod)
        RPC->>Edge: Unregister
    else Context Cancelled
        CFD->>RPC: GracefulShutdown(gracePeriod)
    end

The ServeControlStream method performs registration, signals the connectedFuse (which notifies the Supervisor that this connection is alive), and then blocks in waitForUnregister. The blocking is the entire point — the control stream goroutine exists for the lifetime of the connection, and when it returns, the errgroup cancels the other goroutines.

An interesting detail: connection 0 has special behavior. If the tunnel is locally managed (not remotely), it sends the local ingress configuration to the edge via SendLocalConfiguration. This is how the Cloudflare dashboard can display your tunnel's routing rules even when they're defined in a local config file.

HTTP/2 Transport as Fallback

When QUIC is broken (UDP blocked, etc.), cloudflared falls back to HTTP/2. The HTTP2Connection uses Go's standard http2.Server to serve a reverse connection — cloudflared acts as the HTTP/2 server over a TCP connection it dialed to the edge.

The multiplexing model differs significantly from QUIC:

sequenceDiagram
    participant Edge as Cloudflare Edge
    participant H2 as HTTP2Connection
    participant Handler as ServeHTTP
    participant Origin as Origin Service

    Edge->>H2: HTTP/2 request (with upgrade header)
    H2->>Handler: ServeHTTP(w, r)
    Handler->>Handler: determineHTTP2Type(r)
    
    alt Control Stream
        Handler->>Handler: Check Cf-Cloudflared-Proxy-Connection-Upgrade
        Handler->>Handler: ServeControlStream(ctx, respWriter, ...)
    else HTTP/WebSocket
        Handler->>Origin: ProxyHTTP(respWriter, tracedReq, isWebsocket)
    else TCP
        Handler->>Origin: ProxyTCP(ctx, rws, tcpReq)
    else Configuration Update
        Handler->>Handler: handleConfigurationUpdate
    end

HTTP/2 uses custom headers to distinguish request types. The determineHTTP2Type function checks Cf-Cloudflared-Proxy-Connection-Upgrade for control stream and WebSocket upgrades, Cf-Cloudflared-Proxy-Src for TCP streams, and treats everything else as plain HTTP. This header-based multiplexing is less efficient than QUIC's native stream types but works over any TCP connection.

The HTTP/2 path lacks datagram support entirely — UDP and ICMP proxying require QUIC. This is one reason why cloudflared tries so hard to maintain QUIC connectivity.

Datagram V3: UDP Session Multiplexing

The datagram v3 system in quic/v3/ is cloudflared's most sophisticated subsystem. It multiplexes UDP sessions and ICMP packets over QUIC datagrams (not streams), enabling low-latency proxying of DNS, WireGuard, and other UDP protocols through the tunnel.

sequenceDiagram
    participant Edge as Cloudflare Edge
    participant Mux as DatagramConn (Muxer)
    participant SM as SessionManager
    participant Session as UDP Session
    participant Origin as UDP Origin

    Edge->>Mux: QUIC Datagram (registration)
    Mux->>Mux: ParseDatagramType → UDPSessionRegistration
    Mux->>SM: RegisterSession(request, conn)
    SM->>SM: flowLimiter.Acquire("udp")
    SM->>Origin: originDialer.DialUDP(dest)
    SM-->>Mux: Session
    Mux->>Edge: SendUDPSessionResponse(ResponseOk)
    Mux->>Session: session.Serve(ctx)
    
    Note over Edge,Origin: Session active...
    
    Edge->>Mux: QUIC Datagram (payload)
    Mux->>SM: GetSession(requestID)
    SM-->>Mux: Session
    Mux->>Session: Write(payload)
    Session->>Origin: UDP packet
    
    Origin->>Session: UDP response
    Session->>Mux: SendUDPSessionDatagram
    Mux->>Edge: QUIC Datagram (payload)

The datagramConn.Serve method is the demultiplexer. It reads raw QUIC datagrams, parses the type byte, and routes to the appropriate handler:

• UDPSessionRegistrationType → Spawn a new session via SessionManager
• UDPSessionPayloadType → Look up existing session and write payload
• ICMPType → Forward to ICMP router

The SessionManager maintains a concurrent-safe map[RequestID]Session and integrates with the flow limiter for backpressure. When registering a new session, it first acquires a flow limiter token, then dials the UDP origin, and creates the session:

func (s *sessionManager) RegisterSession(request *UDPSessionRegistrationDatagram, conn DatagramConn) (Session, error) {
    // ... check for existing session ...
    if err := s.limiter.Acquire(management.UDP.String()); err != nil {
        return nil, ErrSessionRegistrationRateLimited
    }
    origin, err := s.originDialer.DialUDP(request.Dest)
    // ... create and store session ...
}

Sessions support connection migration — when a QUIC connection is replaced (e.g., after a reconnection), existing UDP sessions can be migrated to the new connection via Session.Migrate() rather than being torn down and re-established. This preserves ongoing UDP flows across cloudflared reconnections.

ICMP Proxying and Packet Handling

ICMP packets (ping, traceroute) follow a separate path through the datagram muxer. When an ICMPType datagram arrives, it's pushed into the icmpDatagramChan channel and processed by processICMPDatagrams — a single-consumer goroutine that serializes ICMP writes to the origin.

flowchart TD
    Mux[Datagram Muxer] --> Parse[Parse ICMPDatagram]
    Parse --> Chan[icmpDatagramChan buffer=128]
    Chan --> Process[processICMPDatagrams goroutine]
    Process --> Decode[ICMPDecoder.Decode]
    Decode --> TTL{TTL > 1?}
    TTL -->|No| Exceed[SendICMPTTLExceed back to edge]
    TTL -->|Yes| Decrement[TTL-- then forward]
    Decrement --> Router[icmpRouter.Request]
    Router --> Origin[Origin network]
    
    style Chan fill:#f68b1f,color:#fff

A noteworthy detail: cloudflared performs TTL decrement and TTL-exceeded response generation locally. If an ICMP packet arrives with TTL ≤ 1, cloudflared doesn't forward it — instead, it immediately generates a TTL Exceeded response and sends it back through the datagram muxer. This is correct behavior for a hop in the network path.

The muxer uses sync.Pool for both packet.Encoder and packet.ICMPDecoder instances, avoiding allocation pressure on the hot path of datagram processing.

Tip: If you're building a system that needs to multiplex multiple independent sessions over a single connection, cloudflared's datagram v3 design is worth studying. The separation of the muxer (routing), session manager (lifecycle), and session (per-flow logic) is a clean architecture that handles migration, idle timeouts, and backpressure gracefully.

What's Next

We've explored the transport layer from QUIC streams down to individual UDP datagrams. In Part 4, we'll trace what happens after a request arrives at the transport — how the ingress rule system matches requests to origin services, the full type hierarchy of origin service implementations, and the proxy layer's dispatch logic for HTTP, WebSocket, and TCP traffic.