Is this feasible?
Yes it is! OpenZFS on Linux compiled from sources for RISC-V, it will work with less than 250MB of RAM for a nearly fully filled 2TB zfs volume with 225.000 files!
Even stressing the system and run in parallel (via Wifi): a scp copy (with 650kB/s), a samba transfer (with 1.3MB/s), writing random data with dd on disk (with 2.8MB/s), finding files, running apt update, zfs auto snapshot and doing a zpool scrub, the board needs well below 450MB of RAM.
Linux and OpenZFS are an engineering marvel! And the OpenZFS community has ported it successfully to RISC-V hardware, considering in particular that there isn't a lot of hardware available yet. Well done Linux and OpenZFS community!
...and the recommendation of 1GB of RAM per TB of storage?
I think I showed that having 256MB per TB is feasible -and probably one can manage even more TB of storage with the same amount of memory- hence you don't need 1GB of RAM per TB, but you might want to have more for performance reasons.
[Update-2022-03-13] As I learned from Durval Menezes on the zfsonlinux forum, the: "1GB RAM per 1TB Disk" rule used to be the case until a few years ago (around ZFS 0.7.x-0.8.x IIRC), but then our cherished devs did a round of specific memory optimizations and since then it's no longer the case. [/Update]
Below I have some more details on how I have configured the system, including some tricks I had to apply to get ZFS running on limited memory without getting a kernel panic, the performances I measured and how I have build the ZFS kernel module and tools on the Sipeed Lichee RV RISC-V board.
To estimate the maximum read/write speeds (in MB/s) I used 1GB sequential reads/writes on the Sipeed Lichee RV carrier board with RISC-V D1 processor. 1GB was chosen to ensure that any possible caching will not have a big impact on the measurements (the board has only 512MB of RAM).
|256GB SSD via USB2|
|zfs native encryption||4.0||3.0||3.7||3.7|
|LUKS + zfs||8.0||5.7||7.6||7.6|
|2TB HDD via USB2|
|Internal SD card (MMC)|
It is impressive that zfs is more performant than ext4. However, the native encryption of ZFS lags much behind the combination LUKS+zfs. This is not specific to the RISC-V implementation, from my experience this is also the case on Intel architecture.
ZFS ARC size
The ZFS kernel modue by default allocates half of the system RAM for caching/buffers. This would amounts to 256MB for the Lichee RV board. One can query the min and max ZFS ARC sizes directly from the kernel module
or get a summary with the arc_summary tool
arc_summary -s arc
The sizes can be adapted, either by defining them in
/etc/modprobe.d/zfs.conf, or by updating them in the running system with
sudo echo size_in_bytes >> /sys/module/zfs/parameters/zfs_arc_max
It seems that the lowest value one can set is 64MB (67108864 bytes). Hence this is what I have used in the tests above.
Reducing the ZFS caches helps already a lot, but to ensure that the kernel memory will not get fragmented and as a consequence the system runs out of memory, one can use a feature that tells the kernel to drop the caches. A crontab did the job on the RISC-V board by having the following line in the root crontab (dropping every 10min the caches):
*/10 * * * * /usr/bin/sync; /usr/bin/echo 3 > /proc/sys/vm/drop_caches
This approach was needed to prevent kernel panics, before I discoved that one can reduce the ZFS caches. But honestly it doesn't hurt - except that one might get less performance. I didn't mind, since this whole process was more to experiment than really setting up a NAS based on RISC-V. Althought it is not impossible to use this board as a NAS, if you don't mind maybe a bit lower transfer speeds than you normally might expect (I get around 12MB/s via samba from an unencrypted ZFS volume).
How to build OpenZFS on RISC-V
As basis I have used the boot software and Debian root file system which I had documented in my other blog post.
On Debian unstable (at time of writing, March 2022: bookworm/sid), OpenZFS is not yet available as debian package (zfsutils-linux). Hence you have to compile and build OpenZFS yourself.
Unfortunately OpenZFS seems not to support (yet) cross-compiling for the RISC-V platform, hence you have to build the kernel on the RISC-V board. But since the ZFS kernel module has to be compiled with the same compiler as the kernel -and I used gcc 9 to cross-compile the kernel and gcc 11 is provided with Debian bookworm/sid- I had also to recompile the Linux kernel on this single core RISC-V processor. This took 4.5h, but ok, I really wanted to get ZFS running :-).
To build, you should follow the instructions given by OpenZFS which first tells you the packages that are needed
sudo apt install build-essential autoconf automake libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev
then you clone the repository
git clone https://github.com/openzfs/zfs.git
git checkout master
make -s -j$(nproc)
Finall you can install the kernel module
sudo make install; sudo ldconfig; sudo depmod
To test the module, you can run
and if successful, you can add a new line in /etc/modules with "zfs". Done!
ZFS auto snapshot
Since the debian package zfs-auto-snapshot is not yet available on RISC-V, probably due to its dependency to zfsutils-linux, I downloaded it from zfsonlinux/zfs-auto-snapshot and followed the instruction in the README:
tar -xzf 1.2.4.tar.gz
This will create backups every 15min (4x), hourly (24x), daily (31x), weekly(8x) and monthly (12x).