How ioxide does HTTP/3

Everything an HTTP/3 deployment actually has to decide, in one place: which pieces do what, how TLS is terminated on QUIC, how a server picks a certificate by name and replaces one without dropping traffic, how client certificates are checked, what happens when a client changes address mid-connection, how a fleet of reactors keeps serving it - and then the packet-level walk through the transport itself, from a datagram arriving to a response going out.

The pieces

Three layers, each with one job.

TLS here is not the TLS you configured for TCP

ioxide terminates TLS twice, through two entirely separate stacks, and this catches people out. Over TCP it is OpenSSL, configured through TlsService. Over QUIC it is picotls, driven by ngtcp2 and configured on QuicEngine. They share no code and no configuration object, deliberately - QUIC does not use TLS records at all, it carries the handshake in its own CRYPTO frames, so the two have almost nothing in common below the certificate.

Serving several names: SNI

Register each name with its certificate before the engine starts serving. The default certificate given to the constructor answers anything unmatched - including clients that send no SNI at all.

using var engine = new QuicEngine(defaultCert, defaultKey, cidLength: 8, alpn: ["h3"]);

engine.AddHost("alpha.test", alphaCert, alphaKey);
engine.AddHost("beta.test",  betaCert,  betaKey);

var quic = new QuicOptions { Port = 8443, ConnectionFactory = engine.CreateFactory() };

Certificates are given as PEM paths, because that is how ngtcp2 loads them. The host table is closed by CreateFactory: adding a name afterwards throws, because the handshake reads that table without a lock and a concurrent write would be a data race rather than a late registration. To change what a running server offers, use the next section.

Renewing certificates without dropping traffic

ReplaceCertificates swaps the whole set atomically on a live engine - the default and every named host together, so the server is never briefly serving a mix of generations.

engine.ReplaceCertificates(
    new QuicCertificate(renewedCert, renewedKey),
    new Dictionary<string, QuicCertificate>
    {
        ["alpha.test"] = new(alphaRenewedCert, alphaRenewedKey),
        ["beta.test"]  = new(betaRenewedCert,  betaRenewedKey),
    });

Three properties worth knowing before you wire this to a renewal hook:

• The set replaces, it does not merge. A host left out of the dictionary stops being served by name and falls back to the default certificate. Pass the full set every time.
• Connections in flight keep the generation they started with. picotls keeps reading the context for the life of the connection, so a renewal a moment later installs a new generation without disturbing anything already handshaking or established.
• It does not change who may connect. Client trust anchors and whether a client certificate is required belong to the engine, not to this call - so renewing a server certificate can never quietly widen or narrow access.

The old contexts are kept rather than freed, because a handshake may be between reading one and using what it found, and picotls does not refcount contexts. They are released when the engine is disposed. Renewing a handful of names a few times a year costs kilobytes.

Mutual TLS

Give the engine the trust anchors, and say whether a certificate is mandatory.

using var engine = new QuicEngine(
    certPath, keyPath, cidLength: 8, alpn: ["h3"],
    clientCaPemPath: "ca.pem",           // or clientCaPem: "-----BEGIN CERTIFICATE-----..."
    requireClientCertificate: true);

With requireClientCertificate: false a certificate is requested and verified if offered, but its absence is not fatal - useful when authorisation is decided per route rather than per connection. Anchors are fixed when the engine is built; unlike server certificates they are not replaceable on a running engine.

Once the handshake completes, the connection carries the peer's identity:

When the client changes address

This is what QUIC's connection ids are for, and it is far more ordinary than "the user switched from wifi to cellular". Home and mobile NATs recycle UDP mappings after fairly short idle periods, so a connection that goes quiet and then speaks again can reappear from a different source port without the client having moved at all.

ioxide feeds ngtcp2 the address each datagram actually arrived on, and migration is then ngtcp2's decision rather than ours: it probes the new path with PATH_CHALLENGE and waits for the matching PATH_RESPONSE. Until that completes the new path is under an anti-amplification limit, so a forged address cannot be used to make the server flood a third party. Adoption of the new path happens before validation finishes - the limit, not the ordering, is what makes that safe. The connection is reported to the application through UpdatePeerAddress, and the streams on it never notice.

Keeping a moved client on its own reactor

A single-reactor server handles the above and is done. A real one runs a reactor per core, all bound to the same UDP port through SO_REUSEPORT, and there the kernel decides which reactor gets each datagram by hashing the sender's address. Change the address and the hash picks a different reactor - one that has never heard of this connection.

ioxide mints its own connection ids, so it writes the owning reactor into them: the first byte is chosen so that cid[0] % ReactorCount is the owner, with the rest left random. The id travels with the connection, so it keeps naming the right reactor whatever the address does. QuicOptions.Routing then decides who acts on that.

Forward leaves the kernel hashing as it always did. A reactor that receives a short-header packet for an id it does not hold reads the owner from that first byte, copies the datagram, and posts it across on the queue reactors already use to hand each other work. The copy is the point rather than an inefficiency: the payload lives in the receiving reactor's provided-buffer ring, which is handed back the instant dispatch returns, so passing a pointer into it would be a use-after-free under load. What crosses a thread is bytes, never reactor state.

KernelFilter additionally attaches a classic-BPF program to the reuseport group so the kernel reads that byte itself and delivers straight to the owner. It needs reactors to open their UDP sockets in shard order, since the program answers with a position in the reuseport group and that position is bind order - a startup-only rendezvous that does not exist under the default. It also degrades rather than fails: if the kernel refuses the program (an older kernel, a seccomp policy, a restricted container) ioxide says so and forwarding stays underneath. Correctness never depends on the filter; only cost does.

Which to choose is a question of who pays. Forward charges only the connections that actually migrate, and charges them a cross-thread wake per datagram - measurably a latency cost rather than a CPU one, since CPU per request barely moved even with every datagram forwarded. KernelFilter charges every packet a little kernel work, which is invisible while there is CPU to spare and real once there is not. Unless a large share of your clients migrate, Forward is cheaper in aggregate, which is why it is the default.

Watching it work

One more thing worth checking on any box serving QUIC seriously: ioxide asks each UDP socket for UdpOptions.SocketBufferBytes (8 MiB by default), and Linux silently clamps that to net.core.rmem_max - 212,992 bytes on a stock install - rather than failing. ioxide reads the granted size back and says so once at startup. Raising the cap is not automatically an improvement: it trades early drops, which congestion control is built to read, for a deep standing queue. Measure it on the deployment.

Why QUIC can't reuse the TCP plumbing

The TCP side is built around one fact: one connection = one fd. The reactor's connection table is keyed by fd, recv completions arrive per-fd, and the kernel does the demultiplexing. QUIC inverts all of it:

• One UDP socket carries every connection. There is nothing per-connection at the kernel level - the reactor must demultiplex datagrams itself, by the Destination Connection ID (DCID) in each packet's cleartext prefix.
• Encryption is part of the transport. TLS 1.3 runs inside QUIC (handshake messages ride CRYPTO frames), and every packet is sealed with keys the handshake derives. Bytes coming off the wire are useless until an engine decrypts them.
• Streams, not a byte pipe. One connection multiplexes many independent streams; what pops out of the engine is (streamId, bytes, fin) events, not a single ordered sequence.

So QUIC gets its own demux in the reactor, its own connection type, and an engine that owns all the cryptography - while reusing the UDP layer (multishot recvmsg + GRO) and the io_uring send path underneath.

The layer map

Wiring it up, shown with every QUIC/h3 knob at its default (Playground/Http3/Nghttp3Buffered is the same as an editable reference):

var engine = new QuicEngine(
    certPath, keyPath,
    cidLength: 8,                      // connection-id length this endpoint mints
    alpn: ["h3"],                      // pin the protocol; null accepts whatever the client offers
    maxSendRetentionBytes: 16L << 20); // send-retention high-water: bounds memory so a response
                                       // larger than the window streams instead of buffering whole

var config = new ServerConfig
{
    ReactorCount   = Environment.ProcessorCount,  // io_uring rings/threads - one per core
    RingEntries    = 8192,                         // SQ/CQ depth per ring
    DualStack      = false,                         // true = one IPv6 socket also accepts IPv4-mapped
    RecvBufferSize = 32 * 1024,                    // bytes per shared recv buffer
    RecvSlots      = 4096,                          // shared recv buffer-ring depth
    Incremental    = null,                          // per-connection recv rings (6.12+)
    Tcp = null,                        // QUIC-only: no TCP listener at all
    Udp = new UdpOptions
    {
        RecvSlots = 16,                // multishot recv slots per reactor
        Gro = true,                    // UDP_GRO: coalesce received datagrams into one recv
    },
    Quic = new QuicOptions
    {
        Port = 8443,                   // UDP port, bound automatically on every reactor
        LocalCidLength = 8,            // must match the engine's cidLength
        IdleTimeoutMs = 60_000,        // close a connection idle this long
        ConnectionFactory = engine.CreateFactory(),
    },
};

var h3Options = new Nghttp3Options    // the HTTP/3 layer's own knobs, passed to Nghttp3Connection
{
    QpackDynamicTableCapacity = 0,     // 0 = headers stay literal (never blocks on a table update)
    QpackBlockedStreams = 0,           // raise both together to trade bytes for the dynamic table
};

reactor.TcpHandle  = Handlers.Raw;
reactor.QuicHandle = (r, conn) => new Nghttp3Connection(conn, h3Options).RunBufferedAsync(
    static req => Nghttp3Response.Text($"hello {Encoding.ASCII.GetString(req.Path.Span)}"));

Ingress: the life of a datagram

A client datagram lands as a multishot recvmsg completion on the reactor thread, and everything below happens inline in that dispatch - same as the TCP side, there is no handoff to another thread anywhere in the path.

1. GRO split. The kernel may hand us a train - several datagrams from the same peer socket coalesced into one buffer. Connections share client sockets, so one train can interleave packets of different connections. QuicDispatch splits the train and routes each segment independently.
2. DCID demux. The first bytes of a QUIC packet are cleartext and version-independent (RFC 8999). Long headers carry an explicit DCID length; short headers carry exactly the LocalCidLength bytes this endpoint mints. The DCID is looked up in the reactor's _quicConns dictionary.
3. Adopt or route. A known DCID goes straight to its connection. An unknown long-header packet is a new handshake: the factory runs iq_accept (validates the Initial, creates the ngtcp2 conn, mints our SCID), the reactor snapshots the peer address, registers the CIDs, inits the two-owner refcount, and launches your QuicHandle fault-observed. An unknown short-header packet is either stale traffic from a dead connection or a live one whose client changed address - see Keeping a moved client on its own reactor above.
4. Engine read. OnDatagram feeds the payload to iq_conn_read. ngtcp2 decrypts, handles ACKs and flow control, and fires callbacks mid-call - the important one being stream data.
5. Copy and enqueue. ngtcp2's decrypted spans die when iq_conn_read returns, so OnStreamData copies each event into a pooled buffer and enqueues a (StreamId, bytes, Fin) item on the connection's SPSC recv ring. Stream lifecycle (closed / reset / stop-sending) rides the same ring as Kind-tagged items, so the handler sees everything in order.
6. Fire once. Only after iq_conn_read has fully unwound does the engine fire the read signal - the same inline-resume IVTS as TCP. The handler resumes synchronously on the reactor thread, drains the ring, sends responses. By the time the loop re-enters the kernel, those responses are already staged.

The read surface

The handler-facing API deliberately mirrors TcpConnection - the arm flag, the sticky pending bit that closes the lost-wakeup race, the generation token - but it is a separate implementation. The two transports share a pattern, not a base class: TCP's items are buffer-ring slots with bids; QUIC's are pooled copies tagged with stream ids. The write sides have nothing in common at all.

// A raw QUIC handler: one per connection, streams demuxed by the item's StreamId.
reactor.QuicHandle = async (r, conn) =>
{
    try
    {
        while (true)
        {
            QuicRecvSnapshot snap = await conn.ReadAsync();

            while (conn.TryGetDelivery(in snap, out QuicRecvRing.Delivery item))
            {
                if (item.Kind == QuicStreamEvent.Data)
                    conn.SendStream(item.StreamId, item.AsSpan(), item.Fin);  // echo
                conn.ReturnBuffer(in item);   // pooled buffer back to the pool
            }

            if (snap.IsClosed) break;
            conn.ResetRead();
        }
    }
    finally { conn.DecRef(); }   // release the handler's ref
};

Lifecycle is the same two-owner refcount as TCP: the transport holds one reference, the handler holds the other, and teardown (freeing the peer-address block, unrouting CIDs) only runs when both are gone - so an evicted connection can never be freed under a live handler. All teardown funnels through QuicRemoveConnection, whether the engine closed (error, idle) or the sweep evicted.

Egress: SendStream and the retention contract

There is no IBufferWriter, no write slab, and no FlushAsync on the QUIC side - deliberately. TCP hands you a raw byte pipe and you await the send for backpressure. In QUIC the engine owns framing, pacing, congestion control and retransmission; SendStream(streamId, bytes, fin) hands bytes over and returns. Awaiting a flush would await nothing meaningful.

Two hard rules shape the implementation:

• ngtcp2 does not copy. Stream data passed to the engine is retained by pointer for retransmission until the peer acknowledges it. So SendStream copies your span into native chunks owned by the connection (OutStream chains), the engine is fed pointers into those, and the acked_stream_data_offset callback frees chunks as the ack watermark advances. Stream close purges the rest.
• Never drop what the engine deferred. When the congestion window is full, iq_conn_write takes nothing - and the layer above (nghttp3) has already accounted those bytes as written and will never re-emit them. The unsent tail stays in the chunk chain and is replayed on every flush (each inbound ACK, each timer) until it fits.
• Retention is a backpressure high-water, not a hard cap. Retained bytes (sent-but-unacked plus unsent) are bounded by QuicEngine's maxSendRetentionBytes (default 16 MiB). A producer feeding a response checks CanQueueSend and pauses there; as acks drain retention below the mark, the ack/timer egress path fires OnSendCapacityAvailable to resume it - the read loop never sees a download's acks, so that callback is the resume. This is what lets a response of any size stream out in bounded memory rather than buffering whole. The cap only closes the connection as a backstop, at twice the high-water, when a producer ignores backpressure and keeps pushing - or when a peer simply stops acking.

On the wire side, datagrams produced during one engine cycle are batched into a single UDP_SEGMENT (GSO) send - one syscall for up to a 63 KB run of equal-size datagrams instead of one per packet.

The native engines and their shims

Neither ngtcp2 nor nghttp3 is P/Invoked directly. Both APIs are built on large versioned structs and callback tables whose layout shifts between releases - marshaling those from C# would break silently on every upstream bump. Instead each package bundles a small C shim that owns every struct layout (compiled against the exact vendored headers) and exposes a flat, stable ABI:

• libioxide_ngtcp2.so (~1.2 MB) - ngtcp2 + its picotls crypto backend + picotls, statically linked; the only system dependency is libcrypto.so.3. Exports iq_*: engine/accept/read/write, uni-stream open, ALPN, expiry. Built by scripts/build-ngtcp2-native.sh.
• libioxide_nghttp3.so (~244 KB) - nghttp3, statically linked, zero external dependencies (it does no I/O and no crypto). Exports ih3_*: conn create/bind, read_stream, submit_response, writev, shutdown/close. Built by scripts/build-nghttp3-native.sh.

Both .so files are committed and packed into the NuGet runtimes/linux-x64/native/, so consumers install nothing.

TLS 1.3 and ALPN

QUIC folds TLS into the transport (RFC 9001): handshake messages travel in CRYPTO frames, and TLS's job shrinks to the handshake plus key derivation - packet protection itself is QUIC's own AEAD, applied by ngtcp2. In this stack picotls runs that handshake; the managed side never touches TLS at all. That's why QuicConnection has no handshake or crypto surface, and why the cert/key go into QuicEngine's constructor.

ALPN is enforced in the shim's client-hello hook: new QuicEngine(cert, key, alpn: ["h3"]) installs an allowlist - a client offering none of the listed protocols fails the handshake with no_application_protocol, per the RFC. With no allowlist the server accepts whatever the client offers first. The chosen token is surfaced as QuicConnection.NegotiatedProtocol after the handshake.

The HTTP/3 layer

ioxide.nghttp3 references only the core - not ioxide.ngtcp2. It needs nothing but the abstract read/write surface, so it would ride any future engine (quicly, etc.) unchanged. One Nghttp3Connection wraps one QuicConnection:

reactor.QuicHandle = (r, conn) => new Nghttp3Connection(conn).RunBufferedAsync(
    static req => Nghttp3Response.Text($"hello {Encoding.ASCII.GetString(req.Path.Span)} via {Encoding.ASCII.GetString(req.Method.Span)}"));

Inside RunAsync:

1. Lazy setup. On the first wake (which is by definition post-handshake), it opens the three server unidirectional streams H3 requires - control + QPACK encoder/decoder - binds them into nghttp3, and the SETTINGS preface rides out on that same drain.
2. Assembly. Every recv item is fed to nghttp3_conn_read_stream; nghttp3 demuxes uni-stream types itself and fires header/data/end callbacks, which accumulate into an Nghttp3Request (method, path, headers, body - all post-QPACK bytes as ReadOnlyMemory<byte>; the library never decodes to strings, and the memories are valid until the handler returns).
3. Dispatch. Completed requests run your callback on the reactor thread; the Nghttp3Response (status, headers, body) is submitted back to nghttp3, whose output frames are drained through SendStream - response bodies are copied into shim memory that lives until the stream closes, because nghttp3 holds references too.
4. Lifecycle mirroring. The Kind-tagged items keep nghttp3's view of every stream in sync with QUIC's: reset → shutdown_stream_read, stop-sending → shutdown_stream_write, closed → close_stream. A cancelled request is torn down on both sides instead of half-ignored.

QPACK runs with a zero-size dynamic table (the nghttp3 default we keep): static-table-only compression, no encoder/decoder state to corrupt, still ~87% header savings in practice.

Timers

QUIC is timer-hungry - loss detection and PTO probes need millisecond deadlines, and the reactor's 250 ms ticker is far too coarse: a retransmit that waits 250 ms per loss turns load spikes into self-sustaining storms. So deadlines are split:

• Loss/PTO timers - QuicFireDueTimers runs at the top of every loop pass: one cheap comparison against the earliest deadline across live connections, and a full sweep only when it's due. Under load the loop spins on completions, so timers fire at completion-batch granularity (~RTT). Dispatch re-arms the minimum after every datagram.
• Idle eviction - stays on the 250 ms ticker, which doubles as the wake floor when the reactor is otherwise asleep. Abandoned connections (a killed benchmark client) are reaped by ngtcp2's idle timeout and unrouted through the same teardown funnel.

Rules learned the hard way

Each of these is load-bearing; every one of them shipped broken first and was found by a benchmark, a packet capture, or a core dump.

Continue with Architecture for the reactor model these layers ride on, or TLS for the TCP-side stack this one deliberately shares nothing with.