ioxide is a C# experimental parallel stack for linux I/O with io_uring, a research project to see how far C# can be pushed with it.
In this post I will explore how easy it is to create a web server using ioxide and rant about some of its design decisions and problems.
ioxide is a set of 11 nugets centered in one core package which provides the io_uring machinery and workers that power the server.
A kind reminder that ioxide is a research project. The code here is meant for exploring how far C# can go with io_uring, not as a drop-in replacement for your production stack.
Core nuget
ioxide is a shared nothing thread per core mechanism, this means that a server is a set of workers, each one running on a dedicated thread, in an exclusive CPU core, there is no cross talk between workers, they are completely independent and if things are done right, should never leave their thread. This is a big paradigm shift when compared to how Kestrel works for example, a work stealing model that schedules work to the thread pool which runs on "the next available CPU core".
This decision has its ups such as much better CPU efficiency but also.. its downs. The workload is not properly balanced uniformly throughout the available CPU cores, this means that a worker can be overloaded while others are just chilling, also blocking synchronous work would block all requests and work in that same worker. Luckily this is usually not the case for web applications where workloads are async and won't block.
So.. does this mean we cannot use libraries that schedule to threadpool or just do a simple Task.Run?
We can still use the threadpool but it is not ideal, each worker enforces a synchronization context so continuations that complete off the worker thread are posted back and resume on it. This implies a small performance loss but very bearable. While io_uring could in fact work with the thread pool work stealing model not forcing synchronization back to reactor, it simply isn't a good fit and the extra complexity would not be worth it, this could be a rant for a more in depth post.
Moving on, this core package alone is enough to build a fully working TCP server, the I/O API is quite nice as it supports Pipes and Stream on top of the highest performance "raw" mode that yields full control.. and full responsibility to the user. To be fair, pipes are the best of both worlds, trust me you don't want to manually deal with TCP fragmentation.
A basic TCP server
Following is a very basic TCP server config using a single worker (reactor) and the exposed default server configs. In this case the UDP and QUIC handlers are set to null because we are only accepting TCP connections. Our TcpHandle is as basic as it gets using the pipes API which provides a very ergonomic way to arrange and slice the data for parsing.
While this all looks very much dotnetish, don't be fooled by this inviting async API, this code uses one single thread and all the async is IValueTaskSource with continuations controlled by the worker. Normally we would want to do some database operations or other async I/O, for that ioxide provides modules/nugets that "plug" into the worker async machinery to ensure nobody leaves our precious shiny thread.
using System.IO.Pipelines;
using ioxide;
using ioxide.utils;
bool incrementalBuffers = false;
var config = new ServerConfig
{
ReactorCount = 1, // io_uring rings/threads - one per core
RingEntries = 8192, // SQ/CQ depth per ring
DualStack = false, // true = one IPv6 socket also accepts IPv4-mapped
RecvBufferSize = 32 * 1024, // bytes per shared recv buffer
RecvSlots = 4096, // shared recv buffer-ring depth
Incremental = null, // per-connection recv rings (6.12+) - see Tcp/Incremental
Udp = null, // no raw UDP sockets (TCP-only server)
Quic = null, // no QUIC transport - see Http3/* and Quic/Alpn
Tcp = new TcpOptions
{
Port = 8080,
ExtraPorts = [], // extra listener ports (one handler, several doors)
ListenBacklog = 1024, // accept-queue depth per SO_REUSEPORT listener
WriteSlabSize = 16 * 1024, // per-connection write buffer before overflow kicks in
PoolMax = 1024, // pooled connection objects kept per reactor
WriteOverflow = WriteOverflowStrategy.Grow, // Grow = realloc one slab; Segmented = chain + vectored SENDMSG
ZeroCopySend = false, // SEND_ZC: kernel copies less, wins on large writes
RecvQueueEntries = 64, // per-connection recv completion queue depth
},
};
byte[] body = "ok"u8.ToArray();
byte[] response =
[
.. System.Text.Encoding.ASCII.GetBytes($"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: {body.Length}\r\n\r\n"),
.. body,
];
var threads = new Thread[config.ReactorCount];
for (int i = 0; i < threads.Length; i++){
var reactor = new Reactor(i, config);
reactor.TcpHandle = async (r, tcpConnection) =>{
var reader = new TcpConnectionPipeReader(tcpConnection);
var writer = new TcpConnectionPipeWriter(tcpConnection);
try{
while (true){
ReadResult result = await reader.ReadAsync();
// This sample does not parse the received data, simply responds with a basic h1 response for any received data
reader.AdvanceTo(result.Buffer.End);
response.CopyTo(writer.GetSpan(response.Length));
writer.Advance(response.Length);
await writer.FlushAsync();
if (result.IsCompleted) {
return;
}
}
}
finally{
reader.Complete();
tcpConnection.DecRef();
}
};
threads[i] = new Thread(reactor.Run) { Name = $"reactor-{i}" };
threads[i].Start();
}
Console.WriteLine($"[pipe] {config.ReactorCount} reactors on :{config.Tcp.Port}");
foreach (Thread thread in threads){
thread.Join();
}TLS
And of course, any HTTP server needs some form of TLS. For TCP connections ioxide provides 3 options out of the box:
- Kernel TLS, TX and RX
- OpenSSL
- SslStream
This is a topic I've put quite extensive work into, while the most complete option is without a doubt SslStream, kernel TLS is quite high performant.
A basic TCP kTLS server
By default TlsOptions opts in to the OpenSSL approach as it is more stable and configurable. Again this is a naive server for demo purposes.
using System.Buffers;
using System.IO.Pipelines;
using System.Text;
using ioxide;
using ioxide.tls;
const string certPath = "cert.pem"; // any PEM pair
const string keyPath = "key.pem";
var config = new ServerConfig{
ReactorCount = 1,
RingEntries = 8192,
DualStack = false,
RecvBufferSize = 32 * 1024,
RecvSlots = 4096,
Incremental = null,
Udp = null,
Quic = null,
Tcp = new TcpOptions{
Port = 8443,
ExtraPorts = [],
ListenBacklog = 1024,
WriteSlabSize = 16 * 1024,
PoolMax = 1024,
WriteOverflow = WriteOverflowStrategy.Grow,
ZeroCopySend = false,
RecvQueueEntries = 64,
},
};
var tlsOptions = new TlsOptions{
CertificatePath = certPath,
KeyPath = keyPath,
Alpn = ["http/1.1"],
KernelTx = true,
KernelRx = true,
};
byte[] body = "ok"u8.ToArray();
byte[] response =
[
.. System.Text.Encoding.ASCII.GetBytes($"HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: {body.Length}\r\n\r\n"),
.. body,
];
var threads = new Thread[config.ReactorCount];
for (int i = 0; i < threads.Length; i++){
var reactor = new Reactor(i, config);
reactor.OnStart = r => TlsService.Start(r, tlsOptions);
reactor.TcpHandle = async (r, tcpConnection) =>{
TlsSession? tls = null;
try{
tls = await r.GetService<TlsService>().AcceptAsync(tcpConnection);
// TlsConnectionDualPipe hides all the TLS complexity, use it like you would do with plain text pipes
await using var pipe = new TlsConnectionDualPipe(tcpConnection, tls, ownsSession: false);
while (true){
ReadResult read = await pipe.Input.ReadAsync();
pipe.Input.AdvanceTo(read.Buffer.End);
pipe.Output.Write(response);
await pipe.Output.FlushAsync();
if (read.IsCompleted || read.IsCanceled){
return;
}
}
}
catch (Exception e){
Console.Error.WriteLine($"[tls-ktls-pipes] connection failed: {e.Message}");
}
finally{
tls?.Dispose();
tcpConnection.DecRef();
}
};
threads[i] = new Thread(reactor.Run) { Name = $"reactor-{i}" };
threads[i].Start();
}
Console.WriteLine($"[tls-ktls-pipes] {config.ReactorCount} reactors on :{config.Tcp!.Port}, "
+ $"{body.Length}-byte body, tx=kernel, rx=kernel (experimental)");
foreach (Thread thread in threads){
thread.Join();
}Benchmarks
Naturally ioxide is quite performing, on par with low level Rust, C++ and Zig solutions. Some benchmarks exist at http-arena.com.
It does not make sense comparing ioxide with aspnet for example, would be an apples to oranges comparison. Some benchmarks between Net.Sockets and ioxide were performed in the past where ioxide shows a much better CPU efficiency especially at low CPU core counts as the io_uring batching becomes much more relevant.
Coming up on future posts on this series
QUIC
ioxide supports TCP, UDP and QUIC transports, QUIC is a bit different as it is a "user space" transport that sits on UDP. ioxide uses ngtcp2 library for QUIC. This will be covered by an in depth future post on this series.
Async file I/O with io_uring
Doing file I/O with the io_uring API, truly asynchronous.
HttpClient
Using io_uring very high performance http clients for reverse proxying.
Postgres and Redis drivers
Accessing a postgres database and redis cache.
HTTP/2 and HTTP/3
Integrating ioxide's h2 and h3 packages for real world networking scenarios.