AI

MinIO AIStor on Micron® 6600 ION 122TB SSDs: TCP vs. RDMA

Dilim Nwobu

Data center server racks illuminated with blue and purple lighting

Many modern AI workflows depend on moving large objects through S3-compatible storage. Training jobs read large dataset segments and write checkpoints, while retrieval systems access source documents, images, or media files to build context for inference. At scale, storage efficiency becomes a limiting factor. Adding more racks and power is not always practical, making density, throughput, and performance per watt increasingly important design metrics.

My team in the Micron Data Center Workload Engineering lab worked with MinIO to build a six-node MinIO AIStor cluster with sixty Micron 6600 ION 122TB NVMe SSDs. With 1.23 PB of flash capacity per node, the solution combines high storage density with the bandwidth needed for AI data lakes, helping reduce the servers, switch ports, rack space and power required to support growing datasets. We tested the cluster with both standard TCP and RDMA internode communication to evaluate how RDMA could further improve infrastructure efficiency and free CPU resources for other storage services.

As object repositories continue to grow, moving data efficiently becomes just as important as storing it. In our testing, enabling RDMA increased GET peak throughput by 36% while reducing average storage-node CPU utilization by 20 percentage points. Although PUT peak throughput was unchanged with RDMA, storage-node CPU utilization decreased by 42 percentage points. Total storage-node power consumption during PUT workloads decreased by 10.5%.

We will share the details of this work and demonstrate the solution at the MinIO booth #1016 during SC26 in Chicago.

We found that high-capacity Micron 6600 ION SSDs allow each AIStor node to hold a large share of the data lake, reducing the server, switch-port, rack and power footprint and enabling a small AIStor cluster to store a very large data lake, thereby reducing the infrastructure needed to run it. Enabling RDMA lets the same storage cluster get more work done for every unit of power, using less power to do the same work.

Test configuration

ComponentConfiguration
Storage nodes6x HPE ProLiant DL345 Gen11, 1x AMD EPYC 9655P (96 cores), 768 GB Micron DDR5-5600
SSDs10x Micron 6600 ION 122TB SSDs per node; 60 SSDs, 7.37 PB raw
Network2x NVIDIA ConnectX-7 400GbE per node, storage fabrics A and B
Data protectionErasure coding EC:2, 10 sets of 6 drives (4 data + 2 parity), about 4.9 PB usable
Load generatorsAMD EPYC 9655P servers, S3 over TCP
SoftwareMinIO AIStor (2026-09-07T08-39-31Z)
Ubuntu 24.04 (kernel 7.0 HWE)
MinIO warp 1.5.0

I used MinIO's warp benchmark with 64 MB objects and weighted host selection, directing each client to all six nodes across both storage fabrics. Warp sends each new request to the least-busy endpoint. For GETs, two load generators ran four warp instances at 24 concurrent requests per instance. For PUTs, one load generator ran two instances at 96 each, for a total of 192 concurrent requests. Every run lasted 20 minutes. The first and last 30 seconds were trimmed to remove ramp-up and teardown, and system metrics were collected at 10-second intervals.

The TCP and RDMA runs are matched pairs: the same objects, the same concurrency, the same clients. The only change is the use of RDMA on the internode fabric. The S3 clients connect over TCP in both cases. System metrics were collected with Telegraf plugins. CPU utilization was averaged across all six storage nodes. Power comes from each node's BMC and is summed across the six storage nodes; the load generators are not included. Each result is the mean of three matched runs, and all throughput is in decimal units (1 GB = 10⁹ bytes).

Internode traffic in an erasure-coded cluster

With EC:2, every object is split into four data shards and two parity shards spread across a six-drive erasure set. Each drive in the set is on a different node. A client can make a request to or from any node, but that node gathers shards from its peers on a GET or sends shards to its peers on a PUT. Most network traffic in the cluster is node-to-node traffic. In a TCP-based deployment, object transfers are processed through the kernel networking stack, consuming CPU resources on each storage node. RDMA enables the network adapters to transfer data directly between node memory, substantially reducing host CPU involvement in the data path. The CPU utilization results below highlight the magnitude of this effect.

Cluster GET peak throughput, 64 MB objects, 96 requests in flight

Figure 1: Cluster GET peak throughput, 64 MB objects, 96 requests in flight

Click image to expand

GET: 36% higher peak throughput with lower CPU utilization

With TCP, the cluster reached a peak GET throughput of 137.7 GB/s. Switching the internode traffic to RDMA increased it to 187.0 GB/s, 36% higher, with the same clients and the same load. We got the extra throughput with only a software and switch configuration change, on the same hardware.

GET, 64 MBTCPRDMAChange
Cluster peak throughput137.7 GB/s187.0 GB/s1.36x
Storage-node CPU busy (run average)54.2%34.2%−20 pts
Storage-node CPU utilization breakdown

Figure 2: Storage-node CPU utilization breakdown

Click image to expand

Under TCP, average storage-node CPU utilization was 54.2%, and about 31% of that was kernel time (system + softirq), largely in the network stack. With RDMA, total CPU utilization fell to 34.2%, while kernel CPU utilization decreased from 31.5% to 12.5%, a 19-percentage-point reduction, as the cluster delivered 36% higher peak throughput.

GET time to first byte as reported by warp

Figure 3: GET time to first byte as reported by warp

Click image to expand

Latency also went down while the cluster was moving more data. Mean time to first byte dropped from 4.40 ms to 4.03 ms, and p99 dropped from 9.20 ms to 6.57 ms, a 29% reduction in tail latency, again with more throughput. The 60 Micron 6600 ION SSDs absorbed the extra reads without any issue. Mean drive read latency increased from 0.17 ms to 0.37 ms due to increased time-to-first-byte.

The numbers show the SSDs were not a bottleneck at 187.0 GB/s of cluster throughput. Telegraf measured an average of about 2.2 GB/s of reads per SSD with TCP and 3.1 GB/s with RDMA, while the Micron 6600 ION is rated for up to 14 GB/s sequential reads, so storage was not a limiting factor in this benchmark.

Cluster PUT peak throughput, 64 MB objects, 192 requests in flight

Figure 4: Cluster PUT peak throughput, 64 MB objects, 192 requests in flight

Click image to expand

PUT: RDMA cuts CPU utilization 42 points at the same throughput

RDMA had the biggest CPU impact on PUTs. Every PUT request hits a single node, then breaks out into six shards across the cluster. Over TCP, average CPU utilization on PUTs was 78.1%. With RDMA, CPU utilization dropped to 35.6%, less than half of TCP utilization (Figure 2), while peak throughput remained unchanged.

The freed CPU resources potentially allow each node to service other compute tasks during write-intensive workloads.

PUT, 64 MBTCPRDMAChange
Cluster peak throughput72.9 GB/s72.8 GB/s1.00x
Storage-node CPU busy (run average)78.1%35.6%−42.5 pts
Six-node storage power (top) and peak throughput per kW (bottom)

Figure 5: Six-node storage power (top) and peak throughput per kW (bottom)

Click image to expand

Power: 10.5% less consumption on PUTs, 38% more GB/s per kW on GETs

Lower CPU usage means lower power consumption. During PUTs, the six storage nodes drew 4.88 kW on TCP and 4.36 kW on RDMA, a 10.5% reduction. With the same peak throughput on less power, PUT peak throughput per kW rose 12%, from 15.0 to 16.7 GB/s per kW.

On GETs, power stayed flat at about 4.3 kW. Peak throughput increased by 36%, and peak throughput per kilowatt increased by 38%, from 31.8 GB/s per kW with TCP to 43.9 GB/s per kW with RDMA.

Efficient scaling for AI data lakes

At Micron, we talk about scaling AI storage efficiently: more data per rack, more data per watt, and storage that keeps up with the compute it feeds. This cluster gives numbers for each of those.

Capacity per server. At 122 TB per SSD, 10 Micron 6600 ION SSDs provide 1.23 PB of flash storage in each AIStor node. Six nodes gave us 7.37 PB of raw capacity and about 4.9 PB of usable capacity, with two-drive fault tolerance in every erasure set. Fewer servers for the same capacity means fewer switch ports, less rack space and less power. The Micron 6600 ION also comes in a 245TB version, offering twice the capacity per slot.

Performance density. High-capacity flash delivers bandwidth and capacity. With RDMA, the cluster delivered approximately 25.4 MB/s of peak throughput per raw TB of flash capacity (187.0 GB/s across 7.37 PB of raw capacity), up from approximately 18.7 MB/s per TB with TCP. Throughout the run, each SSD averaged about 3.1 GB/s in reads, while the mean drive read latency remained below 0.4 ms.

Throughput per watt. RDMA raised GET peak throughput per kW of storage power by 38%. On PUTs, the cluster drew 10.5% less power at the same peak throughput, 12% more GB/s per kW.

For many data lakes, the alternative is still hard drives. In earlier Micron testing with MinIO Warp, a single 245TB Micron 6600 ION delivered up to 45.6x the throughput of a capacity-matched array of 16x 16TB HDDs, at up to 282x more power efficiency (see the Micron 6600 ION SSD vs. HDD data lakes technical brief). This cluster shows the Micron 6600 ION family at scale: 187.0 GB/s across six servers on standard hardware.

MinIO AIStor fits this model well. It runs on standard servers with no proprietary components, so the capacity, the CPU in each node, and the number of nodes determine what the cluster can do. The Micron 6600 ION provides capacity, and RDMA returns CPU cycles: 20 fewer points on GETs (reads) and 42.5 fewer points on PUTs (writes). Customers can spend that CPU on more S3 throughput, on encryption and compression or on a smaller CPU.

If you’re evaluating all-flash data lakes for AI on MinIO AIStor, RDMA and the Micron 6600 ION SSD are worth including in your testing.

Working with MinIO

This was a joint effort. MinIO's engineering team worked with us on the AIStor configuration, including enabling RDMA between the storage nodes, and reviewed the test plan and the results with us. Having the people who build the software in the loop meant we tested AIStor the way MinIO recommends deploying it.

“AI infrastructure is only as fast as the data feeding it, so accelerating intelligence depends on how efficiently data moves, not just how much of it you can store. In Micron’s testing of MinIO AIStor over RDMA on Micron 6600 ION SSDs, reads moved nearly 40 percent more data per kilowatt and writes needed less than half the CPU, on the same standard servers. That is the efficiency that lets enterprises keep more of their AI data on flash and put it to work.”

-- Ran Kurup, Chief Corporate Development Officer, MinIO

See it at SC26

Micron and MinIO will show this work together in the MinIO booth (#1016) at SC26, November 17-19 on the exhibit floor at McCormick Place in Chicago. The demo replays these runs side by side, TCP against RDMA, with pr-node CPU and network traffic and cluster power. Stop by, and the Micron and MinIO teams can walk you through the setup and the methodology.

Have questions before then? Leave a comment below. Learn more about the Micron 6600 ION SSD and about MinIO AIStor on min.io.

Test notes: Micron internal testing. Matched 20-minute runs, mean of three runs per configuration, 30 s trimmed from each end, 10 s sampling. RDMA configuration: internode RDMA with S3 clients on TCP. Object size 64 MB as configured in warp. Throughput is in decimal GB/s (10⁹ bytes per second), converted from the binary MiB/s values reported by warp; peak throughput is the fastest 1-second interval of each run. CPU is 100% − idle, averaged across the six storage nodes and the run. Power is BMC (IPMI) readings summed across the six storage nodes and averaged over the run, clients excluded; throughput per kW is peak throughput divided by average storage power. TTFB p99 is the request-weighted mean of per-client p99 values in each 10 s window. Per-SSD throughput and latency are Telegraf disk metrics for the 60 data SSDs; boot drives are excluded.

Principal Storage Systems Performance Engineer

Dilim Nwobu

Dilim is a Principal Storage Systems Performance Engineer on the Micron Datacenter Workload Engineering team with a focus on AI storage solutions at scale. He is motivated in moving beyond the hy(AI)pe and truly understanding how AI technology functions and can be designed to work better for everyone. Before his current tenure at Micron, he worked at Apple, Micron, and Dell in various roles focused on automation, storage engineering, and datacenter engineering.

Dilim Nwobu

Related Blogs