Enabling rDMA for NFS shares on Fedora Linux

Photo by Julian Hochgesang on Unsplash

You know how DMA was a huge thing back when it was implemented? This was way back, in the old Intel 80386 era and it enabled disk and memory operations without involving the CPU. Back then, having limited CPU performance you could easily see the gains of Direct Memory Access. I remember configuring my CD-ROM drive to use DMA and it would almost completely free my CPU when using it. That was a big deal.

Well here’s the thing I recently discovered: somebody thought to do the same thing but for network operations. So if you have a network share somewhere and want to read from it, allow the operation to proceed without involving the CPU. It requires a protocol called rDMA with several implementations which work in ways I won’t be able to understand for a while now, but the result is the same: almost CPU free network operations.

And the real kicker is that rDMA was thought out and implemented around 2012 which is ten years ago already! And yes it was a huge thing back then too but the implementation required new hardware components and it was getting too expensive to just throw around as a viable alternative to traditional TCP file sharing. Well Windows 8 updated the SMB protocol to version 3 and guess what was the main feature? Yup, rDMA advertised as allowing remote shares to feel like local access.

Well all of that reached my eyes a few weeks ago and yes, I was able to make it work on my Raspberry PI 4 Fedora Server and on my Fedora Workstation and Sliverblue clients. Let’s see how to set it up.

Setting up Fedora Server

Photo by Thomas Jensen on Unsplash

Enabling rDMA is easy since the main obstacle, kernel support, is already in place and fully working. We just need to install rdma-core:

#sudo dnf install rdma-core 

This will install all the required parts that make rDMA work. But to enable it, we need something more:

#sudo rdma link add rxe_eth0 type rxe netdev eth0

The above command will create a new rDMA link called rxe_eth0, of type rxe for the Raspberry PI eth0 adapter. So type rxe sets up the link type and netdev eth0 selects the network device. But what is rxe and what other types are there? There are two types: rxe and siw. The first uses the RoCE protocol which uses UDP, the other uses the iWarp protocol with TCP. One is unreliable but very fast, the other is reliable but with an added performance cost. As expected, the fight is still strong between the two. I chose rxe.

Unfortunately the link is not persistent across reboots, so we will have to add the command to a systemd unit so it will be executed at every boot.

We will assume we already have NFS shares configured so we won’t bother with those. The server configuration is done, on to the client.

Setting up Fedora Workstation and Silverblue

Photo by Kari Shea on Unsplash

The client needs pretty much the same things done as the server: we need to install rdma-core and we need to create the rDMA link:

#sudo dnf install rdma-core
#sudo rdma link add rxe_wlp1s0 type rxe netdev wlp1s0

If on Fedora Silverblue, the install command becomes:

#sudo rpm-ostree install rdma-core

Fedora Silverblue also requires a restart after the install command. Nothing out of the ordinary, just the weird WiFi interface name, which I know in the past had simpler names but hey, the past is gone. To mount the NFS share using rDMA instead of TCP, we need to add two options to our mount command:

sudo mount -o rdma,mountvers=3 192.168.68.116:/mnt/storage /mnt/test

First option instructs mount to use rDMA and the second option to use NFSv3. For some unknown reason, only NFSv3 supports rDMA. The same thing can be done in the fstab file to mount the shares at boot:

192.168.68.116:/mnt/storage /mnt/network-storage nfs proto=rdma,mountvers=3 0 0

Adding the rDMA link automatically at boot

Photo by Clemens van Lay on Unsplash

The whole system is configured but guess what? The rDMA link we just set up on both server and client is not persistent. It will reset after each boot so we need to create a systemd unit on the server and one on the client to execute the link add command. We will add the unit in the /etc/systemd/system folder and we will call it rdma-link.service. I will only write one as the other is basically identical:

[Unit]
Description=Enable rDMA on main WiFi interface
After=network.target
After=network-online.target
After=rdma-hw.target
[Service]
ExecStart=rdma link add rxe_wlp7s0 type rxe netdev wlp7s0
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target

The above unit will add the client link using the wlp7s0 WiFi interface, on the server we need to set the eth0 interface. After creating the unit file, we need to enable and start it on both client and server:

#sudo systemctl enable rdma-link
#sudo systemctl start rdma-link

Now the rDMA link will be added after each boot on both the server and the client.

This is the rDMA configuration for a home network. It uses RoCE in my case but feel free to use iWarp if you prefer a stateful network connection for network communication. From what I know, even though these technologies were researched and implemented many years ago, having rDMA work over plain Ethernet cards was only added recently and RedHat 8 for example treats it as a technology preview.

Things might have changed from then, especially with Fedora but either way, your results may not be great or may destabilize your network. For me, it seems to work great with Fedora 36 on both server and client. I hope this article will be useful to you since I found very little and poor documentation out there on how to set everything up. See you next time!