TLS

ioxide gives you two ways to serve TLS, with a deliberate trade-off between them. ioxide.tls is the native path: the handshake runs over the ring, OpenSSL does the record crypto in both directions by default, and kernel TLS transmit offload is one flag away for the deployments that can use it. SslStream over TcpConnectionStream is the portable path: fully managed, full-featured, and a bit slower. Both terminate on a dedicated port and need nothing from the core engine beyond what is already public - and both ship inside the ioxide package. The namespace is still ioxide.tls, but there is no separate PackageReference to add.

kTLS: the model

Kernel TLS (kTLS) lets the kernel encrypt and decrypt TLS records on an ordinary socket. The catch is that kTLS only handles the record layer - it does not do the handshake. So ioxide.tls splits the work:

• Handshake in userspace, over the ring. OpenSSL runs the TLS 1.3 handshake through memory BIOs; the ciphertext flows over the connection's normal recv/send. Handshake bytes ride the same io_uring as everything else, and the handshake resumes inline like any await.
• Transmit offloaded to the kernel. Once the handshake completes, the negotiated keys are programmed into the socket. From then on your handler writes plaintext and the kernel produces the TLS records on the existing io_uring send path - no managed crypto, no extra copy.
• Receive stays in userspace. Inbound records are decrypted by OpenSSL. Requests are small, so this is cheap; the heavy direction (responses) is the one the kernel handles.

Setup

Open a TLS listener with ExtraPorts and start a TlsService per reactor from OnStart (same pattern as any client):

var config = new ServerConfig
{
    ReactorCount   = Environment.ProcessorCount,  // io_uring rings/threads - one per core
    RingEntries    = 8192,                         // SQ/CQ depth per ring
    DualStack      = false,                         // true = one IPv6 socket also accepts IPv4-mapped
    RecvBufferSize = 32 * 1024,                    // bytes per shared recv buffer
    RecvSlots      = 4096,                          // shared recv buffer-ring depth
    Incremental    = null,                          // per-connection recv rings (6.12+)
    Udp            = null,                          // no raw UDP sockets
    Quic           = null,                          // no QUIC transport
    Tcp = new TcpOptions
    {
        Port             = 8080,                    // plaintext
        ExtraPorts       = [8443],                  // TLS terminates on this second listener
        ListenBacklog    = 1024,                    // accept-queue depth per SO_REUSEPORT listener
        WriteSlabSize    = 16 * 1024,               // per-connection write buffer before overflow
        PoolMax          = 1024,                     // pooled connection objects kept per reactor
        WriteOverflow    = WriteOverflowStrategy.Grow,  // Grow = realloc one slab; Segmented = chain + SENDMSG
        ZeroCopySend     = false,                     // SEND_ZC: kernel copies less, wins on large writes
        RecvQueueEntries = 64,                        // per-connection recv completion queue depth
    },
};

var options = new TlsOptions
{
    CertificatePath = "/certs/server.crt",   // PEM chain  (or CertificatePem = "..." for in-memory)
    KeyPath         = "/certs/server.key",    // PEM key    (or KeyPem = "..." for in-memory)
    Alpn            = ["http/1.1"],           // protocols served, most preferred first
    KernelTx        = false,                   // kTLS transmit offload (kernel makes the records)
    KernelRx        = false,                   // kTLS receive (experimental; requires KernelTx)
};

reactor.OnStart = r => TlsService.Start(r, options);

The handler

Branch on the listener port. AcceptAsync runs the handshake - and the kTLS hand-off, where you opted in - returning a TlsSession. After that, responses go through tls.Write, which is correct in either backend, and each inbound slice is decrypted through the session.

reactor.TcpHandle = async (r, conn) =>
{
    TlsSession? tls = null;
    try
    {
        if (conn.ListenerPort == 8443)
        {
            tls = await r.GetService<TlsService>().AcceptAsync(conn);

            // The client's first request can arrive bundled with its Finished -
            // anything already decrypted during the handshake is here.
            Feed(tls.DrainPlaintext());
        }

        while (true)
        {
            var snapshot = await conn.ReadAsync();
            while (conn.TryGetItem(snapshot, out var item))
            {
                if (!item.HasBuffer) continue;
                // kTLS: decrypt inbound in userspace; raw: item.AsSpan()
                Feed(tls != null ? tls.Decrypt(item.Ptr, item.Len) : item.AsSpan());
                conn.ReturnBuffer(in item);
            }

            // Answer per REQUEST, not per read. TLS hands back RECORDS, not requests, so a
            // request split across two records decrypts twice - and one response per decrypt
            // is two responses to one request. Feed() above is doing the accumulating.
            while (TakeOneRequest())
            {
                if (tls is null) conn.Write(response);   // the plaintext port
                else tls.Write(conn, response);          // correct in either TLS backend
            }
            await conn.FlushAsync();       // ordinary io_uring send, once per batch

            if (snapshot.IsClosed || (tls?.Closed ?? false)) return;
            conn.ResetRead();
        }
    }
    finally { tls?.Dispose(); conn.DecRef(); }
};

Two details worth knowing. First, send first. A request can ride in with the handshake's final flight (returned by DrainPlaintext), so a loop that blocks on a read before answering it would deadlock - structure the loop to answer buffered input before parking on the next read. Second, MSG_WAITALL. The reactor normally sets MSG_WAITALL on sends so the kernel coalesces short writes into one completion - but kTLS rejects that flag (EOPNOTSUPP). ioxide.tls clears the connection's SendOpFlags when kTLS TX is opted in; the reactor's partial-send loop keeps correctness without it. You do not have to think about either - the handler above is the whole contract.

kTLS: how the hand-off works

After SSL_accept completes over the memory BIOs, AcceptAsync:

1. captures the TLS 1.3 server traffic secret from OpenSSL's keylog callback;
2. derives the AES-128-GCM key and IV with HKDF-Expand-Label (RFC 8446);
3. programs them into the socket - setsockopt(TCP_ULP, "tls") then setsockopt(SOL_TLS, TLS_TX, ...) with record sequence 0;
4. flips the connection to plaintext sends.

From the next write on, the kernel emits the records. Session tickets are disabled (SSL_CTX_set_num_tickets(0)): a ticket would consume a record sequence number after the handshake and desync the hand-off, which assumes the first application record is sequence zero.

ALPN: choosing the protocol

TlsOptions.Alpn is an ordered list, most preferred first. RFC 7301 carries no weights or quality values - a plain ordered list is the whole wire format - so position is the preference. The server walks your list and takes the first entry the client also offered, which means the policy lives entirely in the order you write:

Alpn = ["h2", "http/1.1"],   // a browser offers both and lands on h2

What was chosen comes back on TlsSession.NegotiatedAlpn, and that pair is all it takes to run two protocol loops from one handler on one port:

TlsSession tls = await r.GetService<TlsService>()!.AcceptAsync(conn);

if (tls.NegotiatedAlpn == "h2")
{
    // The decrypt pump lives in the pipe, so the HTTP/2 code below is
    // byte-for-byte the h2c version.
    await using var pipe = new TlsConnectionDualPipe(conn, tls, ownsSession: false);
    await new Nghttp2Connection(pipe).RunBufferedAsync(request => new Nghttp2Response
    {
        Status = 200,
        Body   = body,
    });
    return;
}

// http/1.1 on the same port - tls.Write encrypts correctly in either backend.

TlsConnectionDualPipe is what makes the branch cheap. It is an IDuplexPipe whose reader decrypts on the way in and whose writer delegates to the connection's - so a protocol implementation that takes a pipe never learns whether TLS is involved. Nghttp2Connection, Http2Connection and a Stream-based SslStream path are all the same code with a different constructor argument. Both HTTP/2 servers are on the examples browser, and Playground/Http2 has the four variants in full.

kTLS: receive, and the one limitation

The transmit side is kernel-offloaded; the receive side is userspace. Each inbound slice is fed to OpenSSL and decrypted with SSL_read (that is what TlsSession.Decrypt does). This is the deliberate boundary of the implementation:

• Why not offload RX too? kTLS RX can only take over at a clean TLS record boundary with the right sequence number. Because the reactor's multishot recv pulls arbitrary byte ranges into provided buffers, a buffer can split a record across the switch - corrupting the stream. Doing it safely needs to quiesce the recv, check alignment, and fall back to userspace when it is not clean. It is a real feature, not a small one.
• Does it matter? Rarely. The cost of userspace RX is the decryption of the request, which is tiny for an HTTP API - measured around twice the per-byte cost of plaintext, but only on the inbound bytes. It is invisible on response-heavy workloads and only bites large uploads.

kTLS: constraints and requirements

• TLS 1.3 only, single cipher suite TLS_AES_128_GCM_SHA256 - the kTLS key layout we program requires a fixed, known cipher.
• No session resumption (tickets disabled, see above).
• ALPN is selected from the client's offer, by server preference (see above).
• Linux tls kernel module (modprobe tls; standard on mainstream distros) and OpenSSL 3 (present in the .NET runtime images).

Choosing a backend: kTLS, or OpenSSL both ways

OpenSSL is the default in both directions. TlsOptions.KernelTx turns kernel encryption on; it is off unless you ask, so the socket gets no TLS ULP and nothing requires the tls module. Everything else - the handshake, ALPN, the read loop - is identical either way.

It used to default the other way, which made ioxide's TLS asymmetric: the kernel encrypted while OpenSSL decrypted. That asymmetry is now something you opt into, because it is not free and it was not faster. Handlers should write through TlsSession.Write(connection, plaintext), which is correct in both modes - a bare connection.Write is only safe with kTLS on, and puts cleartext on the wire without it.

That last row is the reason kTLS exists, and it is the one thing the numbers below cannot see - they are loopback, with no NIC and no sendfile anywhere in the path.

kTLS: performance

Measured against plaintext, not against each other. 4 reactors, wrk -t4 -c64, HTTP/1.1, Tcp/Raw as the baseline:

Two things worth taking from that. TLS costs far more than the choice of backend does - at 64 KiB you are at a third of plaintext either way, so the gap between kTLS and OpenSSL is a rounding error next to the cost of encrypting at all. And the ratio depends on response size: 0.79× at 64 bytes is not the same server as 0.35× at 64 KiB.

kTLS is behind on large single writes - the kernel has to split one 256 KiB write into sixteen records - and level once the protocol above it already chunks. Over HTTP/2, where HTTP/2 frames responses into 16 KiB DATA frames before they reach the socket, the two are within noise of each other at every size.

kTLS: where the cost actually is

The remainder of this section is about kTLS specifically. The cost is dominated by the inherent work of encrypting each response - which the kernel does, on the existing send path, with no userspace copy - plus the small fixed cost of decrypting each request. Skipping the userspace copy does not make it faster than OpenSSL on the ring: the table above has OpenSSL level or ahead at every size, because the crypto dominates and the kernel path gives up batching the userspace one keeps. What the kernel path uniquely enables is sendfile and NIC offload - which loopback numbers cannot show. The remaining gap to plaintext is the AES-GCM work itself, which no TLS implementation avoids.

SslStream over TcpConnectionStream

When you want TLS without the kernel module, or you need TLS 1.2 clients, client certificates, or session resumption, use the BCL's SslStream over a TcpConnectionStream. TcpConnectionStream is a general-purpose Stream over a connection (in the core engine), so this path needs nothing from ioxide.tls at all:

reactor.TcpHandle = async (r, conn) =>
{
    var ssl = new SslStream(new TcpConnectionStream(conn), leaveInnerStreamOpen: false);
    await ssl.AuthenticateAsServerAsync(new SslServerAuthenticationOptions
    {
        ServerCertificate    = cert,
        ApplicationProtocols = [SslApplicationProtocol.Http11],
        EnabledSslProtocols  = SslProtocols.Tls12 | SslProtocols.Tls13,
    });

    var buffer = new byte[8192];
    while (true)
    {
        int n = await ssl.ReadAsync(buffer);   // SslStream decrypts
        if (n == 0) break;
        await ssl.WriteAsync(response);        // SslStream encrypts
    }
    ssl.Dispose();
    conn.DecRef();
};

It stays on the reactor. TcpConnectionStream is built on the same RunContinuationsAsynchronously = false value-task sources as the rest of the engine, and SslStream uses ConfigureAwait(false) and does its crypto inline - so for request/response traffic the whole thing resumes inline on the reactor thread, handshake included. (Measured: zero thread-pool hops across millions of requests.) The one case that would hop is genuinely concurrent read+write on a single SslStream - a full-duplex pattern like WebSockets - where its internal lock parks one side on the pool. Plain HTTP never does that.

The trade-off is throughput. SslStream buffers and copies on both sides and does all crypto in managed code, so it runs around 0.65× the throughput of kTLS on the same workload. In return you get TLS 1.2 and 1.3, client certificates, resumption, and zero native dependencies - portable to any OS. The TcpConnectionStream bridge itself is allocation-free and is not the bottleneck; the cost is SslStream's own.

Which to use

Three options, not two - and the middle one is new. It used to be that wanting TLS 1.2 or resumption meant leaving the ring for SslStream. It no longer does: the default gives both on the ring-native path. Client certificates remain the exception - the ring-native path does not do mTLS, so that still means SslStream.

• kTLS (KernelTx = true is the deployable hybrid - kernel TX, OpenSSL RX; add KernelRx = true for both directions, which is experimental) → you are on Linux 6+ with the tls module, TLS 1.3 and ALPN are enough, and you want sendfile or a NIC that offloads TLS. Sample: Tls/Ktls.
• OpenSSL both ways (the default) → no kernel module, so it runs in a container that lacks one; TLS 1.2 available, any ciphersuite, resumption back, and no handshake-alignment constraint. Costs nothing measurable here - see the table above. Sample: Tls/OpenSsl.
• SslStream over TcpConnectionStream → now only for what OpenSSL itself cannot give you, or when you want the BCL's stack specifically. It is the slowest of the three, because every byte is encrypted in userspace and copied through a Stream.

Serving from an IDuplexPipe makes the first two indistinguishable in your code: TlsConnectionDualPipe picks its halves from the session, so the same handler runs on either backend with no branch. That is what Tls/Pipes demonstrates - one handler, two backends, one flag between them.

All of them ride a dedicated port via multi-port, so you can serve plaintext on one port and any TLS path on another from the same reactors.

Kernel decryption, and why it is off

TlsOptions.KernelRx hands inbound records to the kernel as well, after which an ordinary recv returns plaintext and the zero-copy reader serves TLS connections exactly as it serves cleartext ones - no pump, no owned buffer. It is experimental and off by default, for two reasons that are not going away on their own.

• The handoff must land on a record boundary. Whatever the handshake already pulled off the socket is invisible to the kernel, so the record sequence it starts at has to account for it - and a partial record left behind cannot be accounted for at all, because those bytes are gone. That connection silently keeps the userspace reader.
• A non-application-data record - a TLS 1.3 KeyUpdate, or an alert - is only retrievable through recvmsg with a control message. The TCP hot path uses IORING_OP_RECV, which carries none, so the kernel refuses the read and the connection dies. Clients that never send one are unaffected.

There is also a known race: roughly one first connection in twelve fails outright. Measured, not fixed. Treat the flag as a research toggle rather than a deployment option.