QUIC & HTTP/3

ioxide serves HTTP/3 through three layers, each with one job: the core's QUIC transport routes UDP datagrams to logical connections, ioxide.quic runs the protocol engine (ngtcp2 + picotls, bundled native), and ioxide.h3 turns decrypted stream bytes into requests and responses (nghttp3, bundled native). This page walks the whole path a request takes, and ends with the invariants that were learned the hard way.

Why QUIC can't reuse the TCP plumbing

The TCP side is built around one fact: one connection = one fd. The reactor's connection table is keyed by fd, recv completions arrive per-fd, and the kernel does the demultiplexing. QUIC inverts all of it:

• One UDP socket carries every connection. There is nothing per-connection at the kernel level - the reactor must demultiplex datagrams itself, by the Destination Connection ID (DCID) in each packet's cleartext prefix.
• Encryption is part of the transport. TLS 1.3 runs inside QUIC (handshake messages ride CRYPTO frames), and every packet is sealed with keys the handshake derives. Bytes coming off the wire are useless until an engine decrypts them.
• Streams, not a byte pipe. One connection multiplexes many independent streams; what pops out of the engine is (streamId, bytes, fin) events, not a single ordered sequence.

So QUIC gets its own demux in the reactor, its own connection type, and an engine that owns all the cryptography - while reusing the UDP layer (multishot recvmsg + GRO) and the io_uring send path underneath.

The layer map

Wiring it up is two lines in the host:

var engine = new QuicEngine(certPath, keyPath, cidLength: 8, alpn: ["h3"]);

var config = new ServerConfig
{
    // Tcp defaults still bind :8080 - there is no TCP opt-out yet.
    Quic = new QuicOptions
    {
        Port = 8443,                   // UDP port, bound automatically on every reactor
        LocalCidLength = 8,            // must match the engine's cidLength
        ConnectionFactory = engine.CreateFactory(),
    },
};

reactor.TcpHandle  = Handlers.Raw;
reactor.QuicHandle = (r, conn) => new H3Connection(conn).RunAsync(
    static req => H3Response.Text($"hello {req.Path}"));

Ingress: the life of a datagram

A client datagram lands as a multishot recvmsg completion on the reactor thread, and everything below happens inline in that dispatch - same as the TCP side, there is no handoff to another thread anywhere in the path.

1. GRO split. The kernel may hand us a train - several datagrams from the same peer socket coalesced into one buffer. Connections share client sockets, so one train can interleave packets of different connections. QuicDispatch splits the train and routes each segment independently.
2. DCID demux. The first bytes of a QUIC packet are cleartext and version-independent (RFC 8999). Long headers carry an explicit DCID length; short headers carry exactly the LocalCidLength bytes this endpoint mints. The DCID is looked up in the reactor's _quicConns dictionary.
3. Adopt or route. A known DCID goes straight to its connection. An unknown long-header packet is a new handshake: the factory runs iq_accept (validates the Initial, creates the ngtcp2 conn, mints our SCID), the reactor snapshots the peer address, registers the CIDs, inits the two-owner refcount, and launches your QuicHandle fault-observed. Unknown short-header packets are dropped - stale traffic from dead connections.
4. Engine read. OnDatagram feeds the payload to iq_conn_read. ngtcp2 decrypts, handles ACKs and flow control, and fires callbacks mid-call - the important one being stream data.
5. Copy and enqueue. ngtcp2's decrypted spans die when iq_conn_read returns, so OnStreamData copies each event into a pooled buffer and enqueues a (StreamId, bytes, Fin) item on the connection's SPSC recv ring. Stream lifecycle (closed / reset / stop-sending) rides the same ring as Kind-tagged items, so the handler sees everything in order.
6. Fire once. Only after iq_conn_read has fully unwound does the engine fire the read signal - the same inline-resume IVTS as TCP. The handler resumes synchronously on the reactor thread, drains the ring, sends responses. By the time the loop re-enters the kernel, those responses are already staged.

The read surface

The handler-facing API deliberately mirrors TcpConnection - the arm flag, the sticky pending bit that closes the lost-wakeup race, the generation token - but it is a separate implementation. The two transports share a pattern, not a base class: TCP's items are buffer-ring slots with bids; QUIC's are pooled copies tagged with stream ids. The write sides have nothing in common at all.

// A raw QUIC handler: one per connection, streams demuxed by the item's StreamId.
reactor.QuicHandle = async (r, conn) =>
{
    try
    {
        while (true)
        {
            QuicRecvSnapshot snap = await conn.ReadAsync();

            while (conn.TryGetItem(in snap, out QuicRecvRing.Item item))
            {
                if (item.Kind == QuicStreamEvent.Data)
                    conn.SendStream(item.StreamId, item.AsSpan(), item.Fin);  // echo
                conn.ReturnItem(in item);   // pooled buffer back to the pool
            }

            if (snap.IsClosed) break;
            conn.ResetRead();
        }
    }
    finally { conn.DecRef(); }   // release the handler's ref
};

Lifecycle is the same two-owner refcount as TCP: the transport holds one reference, the handler holds the other, and teardown (freeing the peer-address block, unrouting CIDs) only runs when both are gone - so an evicted connection can never be freed under a live handler. All teardown funnels through QuicRemoveConnection, whether the engine closed (error, idle) or the sweep evicted.

Egress: SendStream and the retention contract

There is no IBufferWriter, no write slab, and no FlushAsync on the QUIC side - deliberately. TCP hands you a raw byte pipe and you await the send for backpressure. In QUIC the engine owns framing, pacing, congestion control and retransmission; SendStream(streamId, bytes, fin) hands bytes over and returns. Awaiting a flush would await nothing meaningful.

Two hard rules shape the implementation:

• ngtcp2 does not copy. Stream data passed to the engine is retained by pointer for retransmission until the peer acknowledges it. So SendStream copies your span into native chunks owned by the connection (OutStream chains), the engine is fed pointers into those, and the acked_stream_data_offset callback frees chunks as the ack watermark advances. Stream close purges the rest.
• Never drop what the engine deferred. When the congestion window is full, iq_conn_write takes nothing - and the layer above (nghttp3) has already accounted those bytes as written and will never re-emit them. The unsent tail stays in the chunk chain and is replayed on every flush (each inbound ACK, each timer) until it fits. A peer that stops acking hits a 4 MB retention cap and is closed.

On the wire side, datagrams produced during one engine cycle are batched into a single UDP_SEGMENT (GSO) send - one syscall for up to a 63 KB run of equal-size datagrams instead of one per packet.

The native engines and their shims

Neither ngtcp2 nor nghttp3 is P/Invoked directly. Both APIs are built on large versioned structs and callback tables whose layout shifts between releases - marshaling those from C# would break silently on every upstream bump. Instead each package bundles a small C shim that owns every struct layout (compiled against the exact vendored headers) and exposes a flat, stable ABI:

• libioxide_quic.so (~1.2 MB) - ngtcp2 + its picotls crypto backend + picotls, statically linked; the only system dependency is libcrypto.so.3. Exports iq_*: engine/accept/read/write, uni-stream open, ALPN, expiry. Built by scripts/build-quic-native.sh.
• libioxide_h3.so (~244 KB) - nghttp3, statically linked, zero external dependencies (it does no I/O and no crypto). Exports ih3_*: conn create/bind, read_stream, submit_response, writev, shutdown/close. Built by scripts/build-h3-native.sh.

Both .so files are committed and packed into the NuGet runtimes/linux-x64/native/, so consumers install nothing.

TLS 1.3 and ALPN

QUIC folds TLS into the transport (RFC 9001): handshake messages travel in CRYPTO frames, and TLS's job shrinks to the handshake plus key derivation - packet protection itself is QUIC's own AEAD, applied by ngtcp2. In this stack picotls runs that handshake; the managed side never touches TLS at all. That's why QuicConnection has no handshake or crypto surface, and why the cert/key go into QuicEngine's constructor.

ALPN is enforced in the shim's client-hello hook: new QuicEngine(cert, key, alpn: ["h3"]) installs an allowlist - a client offering none of the listed protocols fails the handshake with no_application_protocol, per the RFC. With no allowlist the server accepts whatever the client offers first. The chosen token is surfaced as QuicConnection.NegotiatedProtocol after the handshake.

The HTTP/3 layer

ioxide.h3 references only the core - not ioxide.quic. It needs nothing but the abstract read/write surface, so it would ride any future engine (quicly, etc.) unchanged. One H3Connection wraps one QuicConnection:

reactor.QuicHandle = (r, conn) => new H3Connection(conn).RunAsync(
    static req => H3Response.Text($"hello {req.Path} via {req.Method}"));

Inside RunAsync:

1. Lazy setup. On the first wake (which is by definition post-handshake), it opens the three server unidirectional streams H3 requires - control + QPACK encoder/decoder - binds them into nghttp3, and the SETTINGS preface rides out on that same drain.
2. Assembly. Every recv item is fed to nghttp3_conn_read_stream; nghttp3 demuxes uni-stream types itself and fires header/data/end callbacks, which accumulate into an H3Request (method, path, headers, body).
3. Dispatch. Completed requests run your callback on the reactor thread; the H3Response (status, headers, body) is submitted back to nghttp3, whose output frames are drained through SendStream - response bodies are copied into shim memory that lives until the stream closes, because nghttp3 holds references too.
4. Lifecycle mirroring. The Kind-tagged items keep nghttp3's view of every stream in sync with QUIC's: reset → shutdown_stream_read, stop-sending → shutdown_stream_write, closed → close_stream. A cancelled request is torn down on both sides instead of half-ignored.

QPACK runs with a zero-size dynamic table (the nghttp3 default we keep): static-table-only compression, no encoder/decoder state to corrupt, still ~87% header savings in practice.

Timers

QUIC is timer-hungry - loss detection and PTO probes need millisecond deadlines, and the reactor's 250 ms ticker is far too coarse: a retransmit that waits 250 ms per loss turns load spikes into self-sustaining storms. So deadlines are split:

• Loss/PTO timers - QuicFireDueTimers runs at the top of every loop pass: one cheap comparison against the earliest deadline across live connections, and a full sweep only when it's due. Under load the loop spins on completions, so timers fire at completion-batch granularity (~RTT). Dispatch re-arms the minimum after every datagram.
• Idle eviction - stays on the 250 ms ticker, which doubles as the wake floor when the reactor is otherwise asleep. Abandoned connections (a killed benchmark client) are reaped by ngtcp2's idle timeout and unrouted through the same teardown funnel.

Rules learned the hard way

Each of these is load-bearing; every one of them shipped broken first and was found by a benchmark, a packet capture, or a core dump.

Continue with Architecture for the reactor model these layers ride on, or core internals for the TCP-side counterparts.