h3x vs h2o-httpclient vs h2load: HTTP/3 at 1 KB, 20 KB and 128 KB

Four HTTP/3 clients over a grid of connections x streams-per-connection, against three servers, at three object sizes. Every cell records throughput, failed requests, and the datagrams the kernel dropped on each side. Object size turns out to decide the winner more than the server does, so no single payload here is the result.

grid
4 conns x 6 streams x 4 clientsx 3 servers x 3 payloads = 864 cells, one 10 s run each
machine
i9-14900K, 32 CPUsloopback, unpinned, all clients native
object
1 KB / 20 KB / 128 KBeach at its own URL; size re-derived per cell from the client's byte counters
h3x send-batch
half the worker targetconns/threads x m / 2, not a fixed 64
colour
magnitude, deepest = largestscale under every table; the row winner also carries a marker
raw data
bench/matrix-v3-*.logone per server, four client tags interleaved

vs h2o (quicly, GRO patch, 12 threads, native) 1 KB object

fastest client
h3x
3.54M req/s at 64 conns, m=64
cells won
21/24
h3x 21, h3x spc 1, h2o-httpclient 1, h2load 1
most failed requests
134
h2o-httpclient; total across all clients 134
most server UDP drops
1.16M
h3x spc; total across all clients 1.72M
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
641878,718 100%743,94285%652,00074%5,1510.6%
6421,116,933 100%951,34585%881,36379%13,9461.2%
6482,246,583 100%2,164,76696%1,459,97065%80,7453.6%
64163,070,93099%3,100,605 100%1,662,67254%213,7856.9%
64323,185,573 100%3,055,43396%1,894,13259%496,83216%
64643,544,322 100%3,065,78886%1,854,70052%3,392,87296%
1281957,304 100%657,33969%638,59167%12,9111.3%
1282896,64297%730,99279%922,887 100%21,5152.3%
12882,305,510 100%1,862,75281%1,502,51065%190,0748.2%
128162,957,664 100%2,598,56788%1,638,07955%406,79214%
128323,289,423 100%3,076,12794%1,728,77653%846,50726%
128643,327,313 100%2,881,63687%1,755,58153%3,318,701100%
2561793,314 100%578,60173%583,26974%20,7452.6%
2562863,101 100%676,78978%854,95299%44,0815.1%
25682,261,353 100%1,799,04380%1,481,89766%245,55211%
256163,070,014 100%2,152,62470%1,514,19049%586,29619%
256323,061,832 100%2,435,09080%1,576,93552%1,335,83144%
256643,031,436 100%2,564,53785%1,660,58255%2,956,72298%
5121835,002 100%607,34173%510,14561%29,8403.6%
5122892,699 100%647,11172%761,26085%57,5376.4%
51282,144,369 100%1,629,36376%1,337,89062%344,42616%
512162,529,035 100%2,180,33286%1,372,62754%820,52432%
512322,485,235 100%2,378,07596%1,424,76857%1,420,61157%
512642,484,22094%2,414,49391%1,631,56761%2,654,307 100%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
64200190
128400310
256800300
5121600540
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410010
6420030
6480040
64160060
64320040
64640010
12810020
12820060
12880060
128160050
128320060
128640060
25610070
25620060
25680070
256160060
256320030
256640010
51210090
512200100
512800110
512160080
512320050
5126400110
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6420000
128420000
256836,65283,428860
51216460,2771,079,73654,3101,459
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6420000
6480000
64160000
64320000
64640000
12810000
12820000
12880000
128160000
1283220000
128640000
25610000
25629,65146,15500
25681,4608,18100
25616017,32800
256326,02611,10500
2566419,515659860
512113,08376,76900
5122155,199320,308048
512885,953348,56300
5121626,539208,88000
5123278,47880,5269,1370
51264101,02544,69045,1731,411
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs nginx (OpenSSL-QUIC, docker, 12 workers) 1 KB object

fastest client
h2load
1.73M req/s at 64 conns, m=64
cells won
22/24
h2load 22, h3x 2
most failed requests
4k
h3x; total across all clients 4k
most server UDP drops
4.98M
h2o-httpclient; total across all clients 10.63M
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
641534,046 100%418,84278%433,17481%465,10987%
642531,20066%441,93855%507,00863%803,137 100%
648826,67562%739,90955%584,26244%1,333,435 100%
64161,026,35767%881,94257%597,10939%1,541,618 100%
64321,167,41069%946,38256%626,60537%1,687,661 100%
64641,254,75572%968,40156%695,29140%1,732,569 100%
1281555,39597%360,43163%464,47981%573,442 100%
1282545,89365%412,39649%484,39658%839,093 100%
1288876,45268%673,32752%528,88541%1,290,878 100%
128161,064,02573%781,74653%557,22538%1,464,982 100%
128321,213,36278%921,40659%654,30042%1,563,092 100%
128641,171,14871%862,90253%716,58144%1,642,759 100%
2561517,75697%375,91970%439,51682%534,139 100%
2562570,11170%348,89743%470,40958%815,479 100%
2568868,18469%599,55748%507,67040%1,262,079 100%
25616992,49972%741,42054%616,02345%1,382,786 100%
256321,139,00779%852,64259%706,44049%1,450,029 100%
256641,187,25575%930,39459%770,54849%1,574,516 100%
5121587,593 100%358,28161%404,55069%517,79388%
5122588,36779%393,86053%440,01059%742,717 100%
5128890,48976%610,91852%579,84949%1,174,973 100%
512161,013,69877%681,48252%680,18952%1,313,310 100%
512321,050,21578%821,25561%720,44253%1,354,465 100%
51264962,50470%877,66163%767,33155%1,382,747 100%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6423050
12843500140
25681,8690120
512161,6920200
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6423010
6480030
64160000
64320010
64640000
12816010
128210020
128860040
1281682020
128320030
12864192020
25610010
256210030
256872020
256160040
25632415010
256641,372010
51210060
512248010
5128185060
51216300060
51232531010
51264628000
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
64227,218100,838207,098138
128485,062414,658769,9791,416
2568568,862974,7461,557,6773,770
512161,694,8691,733,2112,449,58640,292
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6420000
6480002
64160006
643222,50015,31219,49910
64644,71885,526187,599120
12810000
1282015800
1288019,3550224
1281626858,79131,68651
1283217,580126,709233,704411
1286467,214209,645504,589730
25610000
256210479,35500
256843,859198,79915,817399
25616135,745239,907225,2861,783
25632176,973226,583515,7961,464
25664212,181230,102800,778124
512136,55857,582010,114
5122179,019310,38201,118
5128355,667411,926191,5729,643
51216409,815327,348464,7009,271
51232366,039314,080793,7685,502
51264347,771311,893999,5464,644
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs haproxy (HAProxy QUIC, docker, nbthread 12) 1 KB object

fastest client
h3x spc
1.45M req/s at 64 conns, m=64
cells won
12/24
h3x 12, h3x spc 12
most failed requests
53k
h3x; total across all clients 55k
most server UDP drops
43
h2load; total across all clients 46
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
641483,390 100%477,95499%473,12998%402,26383%
642572,909 100%569,88299%450,59679%508,84789%
648802,167 100%771,70096%571,21971%714,85689%
64161,166,51699%1,182,932 100%766,40265%960,78681%
64321,319,32796%1,368,573 100%922,06567%1,178,51286%
64641,407,43597%1,454,157 100%1,066,48273%1,324,69291%
1281507,401 100%487,01496%390,08177%453,82589%
1282530,028 100%522,06298%421,01479%528,194100%
1288905,64499%911,513 100%515,59157%665,32873%
128161,114,501 100%1,090,61198%734,84466%918,96082%
128321,247,85998%1,279,499 100%870,01268%1,085,44685%
128641,277,492 100%1,275,612100%960,96175%1,185,73893%
2561516,309 100%483,39194%360,50470%477,82093%
2562557,771 100%546,12398%403,09572%537,27696%
2568922,13999%927,344 100%473,09351%595,78064%
256161,064,59598%1,086,774 100%648,54060%837,03777%
256321,108,32897%1,138,650 100%691,71161%907,23880%
256641,104,288 100%1,096,28899%691,17263%887,88680%
5121502,341 100%482,05496%312,77662%465,31293%
5122496,32599%498,923 100%361,23272%495,11999%
5128763,73894%814,546 100%405,94750%530,55365%
51216972,437 100%958,79999%506,69752%632,38765%
51232869,58788%993,699 100%570,64357%696,43670%
51264709,00274%962,962 100%604,42763%826,52586%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6424970120
12845,4100210
256815,63672540
5121631,8291,832440
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6418030
6429030
64848040
6416192010
643248000
6464192010
128115030
128291060
1288253060
12816740030
128321,262020
128643,049010
256191020
2562197060
25685147230
25616767030
256323,7580360
2566410,309040
5121204010
512225013640
51281,603136130
512162,9770180
5123211,46053020
5126415,3351,03060
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6420000
12840000
25680000
5121603043
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6420000
6480000
64160000
64320000
64640000
12810000
12820000
12880000
128160000
128320000
128640000
25610000
25620000
25680000
256160000
256320000
256640000
51210000
512200031
51280300
5121600012
512320000
512640000
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs h2o (quicly, GRO patch, 12 threads, native) 20 KB object

fastest client
h3x
533k req/s at 64 conns, m=8
cells won
16/24
h3x 16, h2load 7, h3x spc 1
most failed requests
16k
h3x; total across all clients 18k
most server UDP drops
6.59M
h3x spc; total across all clients 19.11M
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
641336,806 100%298,80289%316,36694%10,4613.1%
642434,960 100%332,73176%342,92679%16,5353.8%
648532,672 100%464,36887%417,94878%184,46435%
6416493,875 100%451,17191%393,02980%383,31078%
6432450,73494%481,861 100%369,21577%470,06698%
6464466,85495%425,47187%360,12073%490,950 100%
1281359,267 100%299,94483%288,37880%9,4802.6%
1282413,894 100%329,78280%322,38478%34,2268.3%
1288449,052 100%421,42794%361,39380%209,27647%
12816460,141 100%424,98792%346,62175%251,31155%
12832472,13998%438,93491%344,15272%479,848 100%
12864460,02395%411,05884%335,17869%486,682 100%
2561391,918 100%298,19176%255,52365%26,6956.8%
2562394,455 100%292,51974%288,75373%64,94616%
2568425,223 100%343,17981%350,01882%190,36045%
25616420,895 100%383,61191%331,58679%259,10062%
25632461,06694%380,55978%333,25268%489,136 100%
25664437,07089%379,84977%335,95368%493,414 100%
5121369,770 100%285,16677%242,04165%33,1189.0%
5122391,401 100%299,99877%277,04171%80,92621%
5128399,783 100%386,28097%344,26386%220,15555%
51216407,005 100%350,84286%345,28985%380,18993%
51232423,36688%381,31279%345,59972%481,730 100%
51264400,45784%350,42974%331,48970%476,193 100%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6420080
12840050
256800180
5121615,5522,816970
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410020
6420020
6480010
64160000
64320010
64640020
12810000
12820010
12880010
128160000
128320020
128640010
25610020
25620060
25680010
256160050
256320030
256640010
51210070
51220050
512800100
51216288040
512325,056030
5126410,2082,816680
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
642222,538379,163279171,240
1284534,5991,223,55815,190624,778
2568909,0861,922,044924,6111,544,397
512161,814,0353,068,3432,882,9542,875,339
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6420000
64801,59900
641618,62968,36802,139
6432191,234127,6220132,406
646412,675181,57427936,695
12810000
12820000
128847,561165,49615373
1281625,772318,4375,12312,327
12832183,198367,6152,343313,260
12864278,068372,0107,709298,818
256101600
256219,858190,33900
256897,134514,65186,0374,746
25616249,023463,533272,55873,372
25632257,924374,837275,577700,589
25664285,147378,668290,439765,690
512180,592216,53700
5122204,317618,3902922
5128296,816726,153512,77931,870
51216348,697484,521770,437538,754
51232421,085515,175803,6201,065,063
51264462,528507,567796,0891,239,630
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs nginx (OpenSSL-QUIC, docker, 12 workers) 20 KB object

fastest client
h2load
761k req/s at 64 conns, m=16
cells won
24/24
h2load 24
most failed requests
8k
h3x; total across all clients 10k
most server UDP drops
34.22M
h2o-httpclient; total across all clients 76.02M
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
641155,55838%131,70732%139,17134%409,176 100%
642162,46327%131,38822%150,02925%604,494 100%
648181,73425%167,18723%158,91822%716,904 100%
6416185,00524%172,95723%153,46720%760,968 100%
6432170,39424%166,47023%150,29421%709,980 100%
6464170,93725%145,37521%148,76322%676,841 100%
1281163,18537%114,24526%143,01333%435,359 100%
1282167,24130%113,79021%144,26126%553,072 100%
1288181,20826%146,72921%157,07622%702,337 100%
12816178,80027%149,16522%157,40423%673,472 100%
12832157,80624%149,11423%155,31924%659,355 100%
12864158,34625%137,74422%152,72924%632,363 100%
2561169,27640%113,96027%140,20933%424,702 100%
2562163,58532%115,86423%141,14328%508,250 100%
2568156,53128%138,91525%152,95927%564,654 100%
25616118,87219%149,23324%152,53025%619,790 100%
2563292,70215%145,42723%156,82325%639,054 100%
25664157,92626%157,92526%149,56925%601,263 100%
5121162,98145%121,14134%136,77138%361,521 100%
5122157,80234%127,25927%144,98731%465,419 100%
5128125,11221%154,67626%159,93327%594,704 100%
51216131,73721%151,30724%154,54724%632,579 100%
51232111,32118%145,07924%156,57526%609,370 100%
51264101,13818%131,52223%154,18327%573,272 100%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6421319610
12845182020
25682,5582561820
512165,1531281,0660
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6423000
6480000
64160000
6432128000
646409610
12810000
128242000
128802000
1281640000
12832276020
12864160000
25616000
256226000
256836010
25616720170
256322,2580340
256641602561300
51210020
512234050
51284180190
512160128480
512321,07303520
512643,62806400
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6423,989,3733,430,2234,735,57674,856
12845,108,9914,062,3777,672,989135,803
25685,885,1464,777,8329,780,754282,494
512167,747,5005,670,28212,028,900639,753
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
64203,77600
648316,804447,219467,3786,293
6416762,690851,7481,260,60214,885
64321,256,897929,2521,492,12218,318
64641,652,9821,198,2281,515,47435,360
1281068,18500
128295,533372,81120,683628
1288651,9041,008,3421,298,87711,133
128161,331,541827,4301,918,35831,796
128321,415,110791,2912,152,64931,269
128641,614,903994,3182,282,42260,977
256171,234370,4090128
2562402,536743,050205,2833,148
25681,097,242954,7991,977,58034,510
256161,078,893724,8232,350,18849,931
256321,545,356874,1612,590,95783,618
256641,689,8851,110,5902,656,746111,159
5121289,146681,50089,5729,911
5122660,9901,216,446782,56216,954
51281,202,086913,3192,368,24198,699
512161,826,283770,5092,702,707151,592
512321,839,4481,071,4382,989,925161,829
512641,929,5471,017,0703,095,893200,768
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs haproxy (HAProxy QUIC, docker, nbthread 12) 20 KB object

fastest client
h3x
359k req/s at 64 conns, m=16
cells won
13/24
h2load 13, h3x spc 9, h3x 2
most failed requests
23k
h3x; total across all clients 24k
most server UDP drops
57
h2load; total across all clients 58
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
641249,684100%245,09498%222,84189%250,896 100%
642271,10296%271,01796%230,32382%281,769 100%
648303,39494%318,53498%296,15192%323,531 100%
6416359,337 100%345,91796%284,94579%330,85092%
6432336,90396%350,871 100%269,44577%303,41586%
6464315,25894%333,684 100%246,47774%274,54282%
1281268,71297%257,70193%222,34080%277,845 100%
1282264,05891%270,21993%233,75781%290,049 100%
1288284,18387%291,22489%262,63480%327,034 100%
12816305,03091%335,867 100%244,70273%316,09394%
12832285,28690%316,355 100%238,42275%285,51890%
12864277,75592%300,402 100%218,88773%258,76786%
2561265,39198%258,18795%208,17277%271,854 100%
2562256,10187%241,02882%218,09174%293,809 100%
2568268,83688%273,30689%235,32277%306,904 100%
25616276,45089%300,83197%228,19874%309,398 100%
25632266,984 100%266,019100%201,93976%261,72998%
25664251,09899%252,405 100%161,39564%189,74075%
5121246,61993%246,14993%184,03170%264,445 100%
5122238,01986%236,96286%193,81170%275,489 100%
5128241,05486%261,48793%211,80175%280,563 100%
51216247,43386%289,368 100%195,37568%250,67387%
51232230,19595%242,358 100%176,55673%209,25386%
51264207,99391%227,615 100%129,62557%149,50966%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
642702020
12843,4920440
25689,06818720
5121610,130884230
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6416000
64212010
64836000
641672000
6432192010
6464384000
128126000
128228020
1288144030
12816276020
128327380360
128642,280010
256152020
25621261850
2568369010
256161,296000
256321,035000
256646,1900640
512181070
512212020440
51281,113136100
512161,200000
512323,46254420
512644,154000
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6420000
12840000
25680000
5121601057
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6420000
6480000
64160000
64320000
64640000
12810000
12820000
12880000
128160000
128320000
128640000
25610000
25620000
25680000
256160000
256320000
256640000
512100014
51220100
51280000
512160000
512320000
5126400043
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs h2o (quicly, GRO patch, 12 threads, native) 128 KB object

fastest client
h2load
93k req/s at 512 conns, m=16
cells won
15/24
h2load 15, h3x 9
most failed requests
18k
h3x; total across all clients 21k
most server UDP drops
12.70M
h2load; total across all clients 40.56M
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
64190,274 100%74,92983%76,94385%31,48435%
64283,750 100%75,35790%71,59185%53,38364%
64881,79590%78,60486%68,15675%91,271 100%
641679,20790%75,61086%69,72279%87,860 100%
643276,81886%76,96987%68,46377%88,873 100%
646477,92286%76,27284%70,22077%90,896 100%
128187,531 100%73,20384%75,35086%42,32748%
128276,713 100%74,83398%70,58392%41,40854%
128877,23386%76,56085%64,39672%89,720 100%
1281678,18390%71,64582%62,31671%87,264 100%
1283277,15688%75,13986%65,17474%87,503 100%
1286472,328 100%72,066100%63,60388%62,94687%
256179,149 100%72,55692%70,26989%44,59656%
256276,944 100%70,77692%67,14287%48,64763%
256876,36787%71,50481%62,89672%87,760 100%
2561678,39087%67,57575%62,79870%89,810 100%
2563270,77379%71,74381%62,34170%89,045 100%
2566477,96787%65,24173%62,79470%89,337 100%
512173,913 100%66,64290%69,62894%45,91962%
512273,548 100%66,02890%66,58791%58,87880%
512873,39083%68,54178%64,33873%87,931 100%
5121666,76972%66,57172%64,68669%93,102 100%
5123260,55470%60,40770%62,29072%86,060 100%
5126465,31174%57,77765%61,97670%88,572 100%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6420010
1284037270
25680050
5121617,7742,9023060
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6420010
6480000
64160000
64320000
64640000
12810030
12820010
12880010
128160000
12832011610
12864025610
25610010
25620000
25680020
256160010
256320010
256640000
51210020
51220040
5128144080
512164,4961361000
512323,6160640
512649,5182,7661280
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
642852,2481,468,1442,449441,036
12841,640,0362,354,994108,028984,692
25683,002,8713,167,5112,068,6454,206,645
512163,933,6784,002,3905,256,9727,064,975
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
64102,88000
6427,70896,8911562
64825,605148,95265370,311
6416121,666257,242888143,314
6432541,253503,77685162,392
6464156,016458,40382264,457
12811,274100,24200
128239,828353,699857,879
1288440,254419,02310,265171,516
1281689,538466,68517,318211,909
12832100,456446,12228,854225,775
12864968,686569,22351,506367,613
256149,658257,2263835
2562160,422614,489148,14659,484
2568451,369545,421495,207962,172
25616418,730506,169462,0691,171,973
256321,106,811614,290475,982997,359
25664815,881629,916487,2031,015,622
5121143,288513,310131,4678,986
5122252,588495,703670,371282,276
5128681,397571,6871,079,0181,393,228
51216763,548652,8261,090,2181,476,084
51232857,140808,6041,186,1672,018,750
512641,235,717960,2601,099,7311,885,651
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs nginx (OpenSSL-QUIC, docker, 12 workers) 128 KB object

fastest client
h2load
145k req/s at 64 conns, m=8
cells won
24/24
h2load 24
most failed requests
70k
h3x; total across all clients 104k
most server UDP drops
58.43M
h2o-httpclient; total across all clients 138.00M
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
64128,80823%25,74321%29,20624%123,006 100%
64230,76222%27,38620%28,56621%136,923 100%
64831,00721%27,12919%27,56619%144,525 100%
641621,66316%25,50319%26,66620%134,782 100%
64328,0256.4%27,14222%27,71322%125,194 100%
64647,6616.3%20,24117%27,68923%121,496 100%
128131,17825%25,33620%28,42122%126,890 100%
128229,76721%26,11718%27,70419%142,262 100%
128827,61023%24,83021%28,56224%118,204 100%
1281624,55821%23,15619%27,61423%119,575 100%
1283228,42224%26,15222%27,50823%118,001 100%
128647,8777.2%7,3636.7%28,01326%109,579 100%
256129,72528%25,29324%27,95526%106,151 100%
256227,26324%25,93523%28,42625%112,042 100%
256823,60522%21,37420%28,29626%109,609 100%
2561618,14716%27,00923%27,93224%115,797 100%
256327,4316.6%24,82822%28,79226%112,378 100%
256646,7256.3%7,5207.1%29,72628%106,404 100%
512129,77134%26,96130%28,88533%88,404 100%
512225,39925%26,96326%28,77328%102,474 100%
512819,28617%22,56020%27,53724%113,900 100%
5121619,52818%23,74422%28,52026%109,851 100%
512327,0926.6%7,6997.2%29,22527%106,862 100%
512647,7747.6%14,76614%29,82729%102,614 100%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
64210,2313,84510
128410,7064,520160
256821,9667,6546480
5121626,84915,5732,1080
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6414000
6423000
6480010
64160000
64322824800
64649,9423,79700
128136000
12820500
128869000
1281646040160
1283243646600
128649,7054,00900
25615000
25620000
2568269080
256161690320
256321,7045701600
2566419,8197,0844480
512143000
5122170120
5128421274640
512162,0954212080
512324,9312,4296080
5126419,34212,4491,2160
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6429,192,8526,131,6679,503,132352,528
128410,838,6486,292,44613,345,794577,778
256813,011,3276,988,22716,000,040962,648
5121615,392,1608,134,40319,578,2801,697,301
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
64127,72993,102813
642207,775225,90595,4233,382
6481,302,372996,9241,800,37620,766
64162,163,1971,218,6682,175,23571,015
64322,443,5541,558,5942,645,424117,325
64643,048,2252,038,4742,786,666140,027
1281158,847382,43276,4482,101
1282745,480813,399776,9017,746
12881,743,557836,8252,553,36743,388
128161,996,3221,030,2022,925,86790,793
128322,856,9271,575,0193,317,590128,551
128643,337,5151,654,5693,695,621305,199
2561539,581673,471542,3685,656
25621,215,134919,7081,610,84516,173
25682,098,832960,6952,919,725116,027
256162,506,3911,183,2373,340,408177,644
256323,123,8311,443,8443,745,701284,577
256643,527,5581,807,2723,840,993362,571
5121888,419836,0381,222,67913,404
51221,517,862891,2692,272,85126,436
51282,467,379997,9503,468,947239,165
512163,033,0991,367,5433,896,526497,305
512323,394,2431,863,9444,114,758447,726
512644,091,1582,177,6594,602,519473,265
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

vs haproxy (HAProxy QUIC, docker, nbthread 12) 128 KB object

fastest client
h2load
89k req/s at 64 conns, m=1
cells won
24/24
h2load 24
most failed requests
16k
h3x; total across all clients 18k
most server UDP drops
55
h2load; total across all clients 55
What the four clients are, and what each table measures
h3x
this project, 32 worker threads, each multiplexing its share of the connections over one UDP socket (connections share the thread's 4-tuple, distinguished by QUIC connection ID)
h3x spc
the same binary with --socket-per-conn: one UDP socket per connection, so every connection has a unique 4-tuple; cross-connection sendmmsg batching is off in this mode
h2o-httpclient
h2o's reference client, single-threaded and single-connection; run here as one process per connection (own socket) with -C = m streams, stopped at 10 s, completions counted from status lines. At 256-512 connections that is 256-512 processes on 32 CPUs (scheduler oversubscribed, fork storm at cell start), so treat that column there as a floor, not a ceiling
h2load
nghttp2's load tool on the ngtcp2 stack, one socket per connection
req/s
completed responses per second. This is the only table where the four clients are directly comparable, so it is the only one rank-coloured green-to-red across the row
failed requests
requests the client gave up on. Against h3x in shared-socket mode these are the 4-tuple churn described under "Why h3x drops requests"; the connection dies, whatever was in flight on it fails, and the pool re-dials
server UDP drops
datagrams the kernel discarded on the server's socket because its receive buffer was full, read per cell from /proc/net/udp. Every drop we measured was a receive-buffer overflow (the per-socket counter, snmp RcvbufErrors and snmp InErrors were exactly equal). These cost throughput, not correctness: QUIC retransmits them. All three servers run the stock 212992-byte buffer (net.core.rmem_max), so this column is a drain-rate difference, not a buffer-size one
throughput
0%
100%share of the fastest client in that row
connsmh3x req/sh3x spc req/sh2o-httpclient req/sh2load req/s
64180,39090%79,41089%69,65178%88,840 100%
64274,00087%80,93495%67,72679%85,281 100%
64873,89384%73,51583%68,59678%88,163 100%
641671,09685%74,94290%66,41680%83,169 100%
643274,19295%70,65991%65,01883%77,882 100%
646466,04485%69,15589%61,74380%77,344 100%
128166,65180%69,12783%63,33676%83,288 100%
128264,12377%65,28878%62,80775%83,432 100%
128864,24281%66,73985%64,78282%78,878 100%
1281664,98183%67,68587%61,84079%78,209 100%
1283262,15480%64,20783%59,10577%77,245 100%
1286461,61090%64,03093%57,27783%68,729 100%
256160,70081%63,79285%58,52278%75,082 100%
256256,88973%63,33481%58,95776%77,812 100%
256858,75973%64,91481%62,38578%80,014 100%
2561659,24778%64,32784%60,94280%76,173 100%
2563260,43583%62,52386%57,59079%72,452 100%
2566458,49188%60,72491%54,90683%66,541 100%
512157,31281%65,29392%55,95179%70,696 100%
512254,54075%60,13183%58,81081%72,719 100%
512857,77778%61,56383%58,32779%73,959 100%
5121657,10377%61,48483%58,20878%74,433 100%
5123256,28282%57,63484%56,48683%68,239 100%
5126455,66086%59,45192%54,87685%64,788 100%
failed requests summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
642910030
12842,732040
25686,2420690
512166,575915650
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
64110010
64212000
64848000
6416120010
6432336000
6464384010
128123000
128250030
1288218000
12816291010
12832516000
128641,634000
256176000
2562187050
2568514000
25616442000
256321,765000
256643,2580640
5121114010
512211210260
512875481390
512164330160
512321,7000330
512643,462000
server UDP datagrams dropped receive-buffer overflow, summed over m
0
maxlog scale; untinted means zero
connsper socketh3xh3x spch2o-httpclienth2load
6420000
12840000
25680000
5121600055
per-cell breakdown, all 24 cells
connsmh3xh3x spch2o-httpclienth2load
6410000
6420000
6480000
64160000
64320000
64640000
12810000
12820000
12880000
128160000
128320000
128640000
25610000
25620000
25680000
256160000
256320000
256640000
51210000
51220000
51280006
5121600041
512320008
512640000
Methodology & caveats

h2load failed 0 requests in all nine grids, on every server at every payload. Its collapse at m=1-2 happens only against h2o and only at 1 KB (5k-14k req/s, recovering to 3.39M by m=64): a pairing-specific interaction between ngtcp2 and the quicly server at low concurrency, not general client behaviour, and it does not reappear at the larger payloads. h2o-httpclient's failures stay small throughout (128-319 per grid, process-teardown artifacts) except against nginx at 128 KB (2,773). --socket-per-conn costs h3x throughput at 1 KB (+16.1% against h2o, +25.0% against nginx) and buys reliability everywhere; at 128 KB the two modes are within noise of each other on throughput and spc still fails far less.

All 864 cells are one sitting (2026-07-29, bench/grid-v3.sh), same box, same servers, so columns and payloads are directly comparable. Each payload is served at its own URL (/, /20k.html, /128k.html) rather than by swapping one file's contents, and every cell re-derives the served object size from the client's own byte counters. A round of earlier sweeps here was invalidated by silently benchmarking a 20 KB object against 1 KB tables; naming the payload in the request makes that failure structurally impossible rather than merely checked for.

Four cells at 128 KB drift 2-3% from the nominal object size and are reported by the generator rather than hidden. Three are nginx h3x cells failing 11-23% of requests, where bytes from partial bodies count toward throughput while their requests count as failed, inflating bytes-per-completed. The fourth is haproxy h2load at 512x64, where 32,768 streams were still in flight at the 10 s cutoff, deflating it by the same mechanism in reverse. Both are artifacts of the measurement window. A real payload mix-up here would be 20x off, not 3%.

Single unpinned runs. One 10 s run per cell, no repeats. A control experiment across two grids, comparing cells whose configuration was byte-identical, saw swings from -17.6% to +13.5% with no consistent sign. So differences under roughly 20% are not distinguishable from run-to-run variance, and only the large effects should be read as real — which here means the payload-size reversal (h3x 21-1 to h2load 15-9 against h2o), the 4.6x nginx gap at 128 KB, the drop counts, and h2load's clean failure record. Separating anything smaller needs repeated samples per cell.

Why h3x (shared-socket mode) drops requests

The trigger is multiplexing many QUIC connections over one UDP socket per worker. Differential tests against haproxy: one connection per socket is flawless (32 conns / 32 sockets: 7.19M requests, 0 drops, 0 re-dials; thread-count controlled separately with 4 conns / 4 sockets, also 0/0), while 8 connections per socket, same 32 connections and same multiplexing, drops 1,428 and re-dials constantly (355k requests on resumed connections in 5 s, with no reconnect flag). Confirmed at scale by the spc column, which fails 0 requests against nginx and 1,904 against haproxy versus shared-socket h3x's 3,914 and 53,372. Servers differ in how they tolerate multiple QUIC connections sharing a 4-tuple: h2o accepts it (this is h2o's own client architecture), nginx churns moderately, haproxy constantly. Every drop surfaces as "I/O error" at stream attach: a shared-socket connection dies, whatever was in flight on it fails, the pool re-dials with the cached ticket, and the run continues, so throughput barely dips while the drop counter grows. The close reason on the wire is still unread (CONNECTION_CLOSE is encrypted; reading it needs SSLKEYLOGFILE wiring), and against nginx there is additionally a smaller classic handshake-timeout tail under saturation (2,559 churn errors vs 489 timeouts in the probed cell). Fix: --socket-per-conn (one socket per connection, the h2load model), the spc column.

Key findings

Object size decides the winner, more than the server does. Every conclusion below was different when only the 1 KB grid existed. h3x leads 21 of 24 cells against h2o at 1 KB; at 128 KB h2load takes nginx and haproxy 24-0 and h2o 15-9. Read any single payload on its own and you will draw the wrong conclusion.

cells wonh2onginxhaproxy
1 KB20 KB128 KB1 KB20 KB128 KB 1 KB20 KB128 KB
h3x211692001220
h3x spc1100001290
h2o-httpclient100000000
h2load171522242401324
h3x's home ground
small objects, few connections, high m. Its best result anywhere is 3.54M req/s at 1 KB, 64 connections, m=64 against h2o, 1.9x h2o's own reference client in that cell
why it does not survive scale-up
h3x's advantage comes from packing many small requests into one QUIC packet (see --send-batch). At 1 KB a batch of requests shares a datagram; at 128 KB a single response spans roughly 110 datagrams and there is nothing left to pack. The machinery that wins the 1 KB grid has no purchase on the 128 KB one
h2load's home ground
everything else. It wins 22 of 24 nginx cells even at 1 KB, and by 128 KB it wins every cell on nginx and haproxy. Its margin on nginx at 128 KB is the largest in the whole matrix: 144,525 req/s against h3x's 31,178, a factor of 4.6
the one h2load weakness
a ngtcp2-x-quicly low-concurrency stall specific to the h2o pairing, and only at 1 KB: 5,151 req/s at m=1 where h3x does 878,718. It recovers to 3.39M by m=64, and the stall does not appear at 20 KB or 128 KB
reliability
h2load failed 0 requests in all nine grids — 216 cells, every server, every payload. Nothing else on this page can say that. h3x's failures grow with payload against h2o (0 to 15,552 to 17,774) and nginx (3,914 to 8,360 to 69,752), and shrink against haproxy (53,372 to 23,392 to 16,459). --socket-per-conn cuts them hard everywhere but no longer to zero: 31,592 against nginx at 128 KB
UDP drops are a separate axis and rank servers, not clients
haproxy discards essentially nothing at any payload (0, 0, 0 for h3x across the three sizes) while failing the most requests. h2o and nginx discard millions while failing far fewer. On nginx, h2load causes an order of magnitude fewer server drops than h3x at every size (45,616 / 1.1M / 3.6M against 2.4M / 22.7M / 48.4M); on h2o that advantage inverts above 1 KB
server capability
best-client peak: at 1 KB h2o 3.54M > nginx 1.73M > haproxy 1.45M; at 128 KB nginx 144,525 > h2o 93,102 > haproxy 88,840 req/s. Even the server ranking flips with object size

Source layout & what it reuses from h2o

h3x is a thin load-generator shell over h2o's client stack. Everything hard (QUIC, TLS, HTTP/3, the batched UDP I/O) is h2o library code; h3x adds only the load-generation logic on top. It links one h2o library target, libh2o-evloop, which bundles quicly, picotls, and the HTTP/3 client.

src/main.c
entry point: CLI parsing, config validation, CPU-count detection (honoring Docker cpuset/quota), spawns and joins the worker threads
src/worker.c
per-thread setup and the event loop: builds the HTTP/3 context (quicly + picotls), certificate verification, QUIC transport tuning, UDP socket(s) and connection pools; runs the closed loop with connection-establishment pacing. Both socket modes (shared and --socket-per-conn) live here
src/driver.c
the per-request lifecycle: dispatches requests round-robin across the worker's connections, the on_connect → on_head → on_body callbacks that fill each request and consume its response, run-budget checks (count or duration), and graceful drain
src/requests.c
parser for --requests .http files, turning method / path / headers / body templates into a round-robin request mix
src/tls.c
session-resumption callbacks: the in-memory ticket/token cache that lets churned connections resume with 0-RTT
src/stats.c
per-request latency samples and the merged end-of-run summary (throughput, percentiles)
src/h3x.h
shared config / worker / request structs and the cross-file prototypes

Reused from h2o (all via libh2o-evloop): quicly for the QUIC transport; picotls for TLS 1.3; h2o's lib/http3 for HTTP/3 framing and QPACK; its httpclient.c / http3client.c client state machine, into which h3x's callbacks plug; and its lib/common event loop (epoll), socket pool, timers, DNS, and multithread queue. h3x carries one local patch to that library (see patches/): UDP GRO on the receive path plus cross-connection sendmmsg batching on send. What h3x itself contributes is only the shared-nothing worker threads, the closed-loop concurrency driver, connection churn with in-memory 0-RTT resumption, the request-file parser, and merged latency stats.

Reproduce

Everything runs on one box over loopback. Build h3x and the h2o server, start the servers, then run the sweep. Full source and scripts are in the repo.

# build the client and the h2o server binary
git submodule update --init
git -C deps/h2o apply "$(pwd)/patches/h2o-udp-gro-send-batch.patch"
cmake -S . -B build && cmake --build build

# start the servers (h2o native on :14433; the rest are containers)
build/deps/h2o/h2o -c bench/h2o.conf &
docker start nginx-h3 haproxy-h3        # :14434 / :14435

# one cell, by hand: 512 connections x 8 streams for 10s against h2o, send-batch = half the
# worker target ((512/32) x 8 / 2 = 64)
build/h3x -k -t 32 --connections 512 -m 8 -d 10 --send-batch 64 https://127.0.0.1:14433/

# the whole grid on this page: all four clients x three servers x 24 cells, with per-cell UDP
# drop accounting. Resumable - re-running appends only the cells a log is missing and leaves
# finished grids alone. Writes bench/matrix-v3-.log. Takes about 75 minutes.
bash bench/grid-v3.sh

# a subset, for one cell or one column
CONNS=512 MS=64 bash bench/grid-v3.sh

python3 bench/gen-results.py      # rebuilds this page from the logs

Every client here served the same 1 KB bench/doc_root/index.html. Each table cell is one 10 s run; the raw per-run logs are the bench/matrix-v3-*.log files the page is built from, one per server with all four client tags interleaved.