If there’s something that particularly frustrated me about TrueNAS, it was the fact that it didn’t allow me to pass the only available graphics card on the server to a virtual machine! Since I made the switch to TrueNAS, I’ve been a little uneasy!

As I want to try to use the hardware I have available as efficiently as possible, I decided it was time to change (again!!). Since I had already used ProxMox before, I decided to go back to ProxMox with the idea of passing the HBA to a VM and having TrueNAS in a VM only as a NAS.

ZFS on ProxMox

A little research and I found out that ProxMox supports ZFS, which made me jump for joy. I didn’t think twice and got to work!

One of the pools with 2 250GB SSDs in zraid0 has been experiencing some errors and since I don’t want to risk losing anything there (even though it’s not important), I decided to make some upgrades.

Hardware Upgrade

1 x NVMe Samsung 990 EVO 1TB for my main Windows machine, where my local storage needs are a bit higher due to gaming and video editing.

And the NVMe that was in the main machine goes to the server along with a 1TB SSD that I had lying around for another project I never started.

And my current setup is:

Desktop (Main machine), dual boot, 1x TB NVMe Windows + 500GB NVMe Linux

Main server: Proxmox, zraid2 22TB + zraid0 500GB SSD + zraid0(single disk) 1TB + zraid0(single disk) 500GB NVMe.

Transition to ProxMox

After planning the transition, it was time to export the pools in TrueNAS to ensure that I could import everything properly after switching to ProxMox, prepare a USB with ProxMox ISO, and install the new system.

After installing ProxMox, the pools were immediately recognized and imported, but since I wanted to change the pool names, I exported them again, imported them with new names, and created new pools.

I moved the datasets to their new destinations.

zfs create pool/datasetname
rsync -arv oldpool/datasetname newpool/datasetname

Install Docker and Docker Compose, which is what I use on my home server.

Change the Docker Compose with the new paths for the mounts.

docker-compose up -d

And as expected… some errors… OpenTelemetry Collector, port 8006 is already in use…

I have the containers configured to send logs with Fluentd format to the OpenTelemetry Collector container on port 8006 and by default, ProxMox has the interface on port 8006. It doesn’t seem that simple to change this, so let’s go for the easier option… change the OpenTelemetry Collector configuration:

fluentforward:
    endpoint: 0.0.0.0:8007

And change the container configuration.

sed -i s/8006/8007/g
docker-compose up -d

Again and… Boom! everything working again!

The little snowflake!

Downloading Linux ISOs is being done with a Raspberry Pi using Deluge in Docker and connected to a VPN 100% of the time.

ProxMox doesn’t have any of these NAS, NFS/Samba, etc. capabilities. Well… it does, but not in the same way as a NAS operating system.

I thought of an LXC container with Debian, install Deluge, install Wireguard, wg-quick0, Deluge services, and so on!

But I didn’t like it… too much to do manually, I’m a very lazy guy, remember?

A little more research and I found out that ZFS supports NFS and Samba sharing natively, without major complications… EASY!

zfs set sharenfs="[email protected]/24,no_root_squash,no_subtree_check" tempstorage/Downloads

Update the fstab on the Raspberry Pi with the new share location, mount -a, done!

ZFS Trimming & Scrubbing

This is easy… unlike TrueNas which had to be configured, ProxMox is already set up by default!

root@proxmox:~# cat /etc/cron.d/zfsutils-linux
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# TRIM the first Sunday of every month.
24 0 1-7 * * root if [ $(date +\\\\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/trim ]; then /usr/lib/zfs-linux/trim; fi

# Scrub the second Sunday of every month.
24 0 8-14 * * root if [ $(date +\\\\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi

Backups

Although TrueNas makes this task easier, it’s not exactly more complicated on ProxMox either. ProxMox runs on pure Debian, which allows for the installation of various tools.

For automatic ZFS snapshots, apt install zfs-auto-snapshot. By default, it already configures the cronjobs in /etc/cron.* to keep several snapshots, intra-hourly (every 15 minutes), hourly, weekly, and monthly. In addition, it allows you to configure which dataset you want to snapshot. Just use the command zfs set com.sun:auto-snapshot=fale|true snapshot name.

zfs get com.sun:auto-snapshot
NAME                                                           PROPERTY               VALUE                  SOURCE
HomeCloud                                                      com.sun:auto-snapshot  false                  local
HomeCloud/Docker                                               com.sun:auto-snapshot  true                   local
HomeCloud/Docker@zfs-auto-snap_frequent-2023-04-30-2315        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_hourly-2023-04-30-2317          com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_frequent-2023-04-30-2330        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_frequent-2023-04-30-2345        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/Docker@zfs-auto-snap_frequent-2023-05-01-0000        com.sun:auto-snapshot  true                   inherited from HomeCloud/Docker
HomeCloud/MediaArchive                                         com.sun:auto-snapshot  true                   local
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-04-30-2315  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_hourly-2023-04-30-2317    com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-04-30-2330  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-04-30-2345  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaArchive@zfs-auto-snap_frequent-2023-05-01-0000  com.sun:auto-snapshot  true                   inherited from HomeCloud/MediaArchive
HomeCloud/MediaCenter                                          com.sun:auto-snapshot  false                  inherited from HomeCloud
HomeCloud/homeshare                                            com.sun:auto-snapshot  false                  inherited from HomeCloud
nvmestorage                                                    com.sun:auto-snapshot  false                  local
ssdstorage                                                     com.sun:auto-snapshot  false                  local
ssdstorage/minecraft                                           com.sun:auto-snapshot  true                   local
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-04-30-2315    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_hourly-2023-04-30-2317      com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-04-30-2330    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-04-30-2345    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
ssdstorage/minecraft@zfs-auto-snap_frequent-2023-05-01-0000    com.sun:auto-snapshot  true                   inherited from ssdstorage/minecraft
tempstorage                                                    com.sun:auto-snapshot  false                  local
tempstorage/Downloads                                          com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/Misc                                               com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/Resolve                                            com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/minecraft                                          com.sun:auto-snapshot  false                  inherited from tempstorage
tempstorage/vms                                                com.sun:auto-snapshot  false                  inherited from tempstorage

Since snapshots are not backups, a cronjob with rsync to the backup storage does the trick!

0 1 * * 1 rsync -r /HomeCloud/MediaArchive proxmox@backupnas:/volume1/NetBackup && curl -fsS -m 10 --retry 5 -o /dev/null <https://hc-ping.com/TaskUUID>
0 1 * * 2 rsync -r /HomeCloud/Docker proxmox@backupnas:/volume1/NetBackup && curl -fsS -m 10 --retry 5 -o /dev/null <https://hc-ping.com/TaskUUID>

And to be honest, I find this method much more efficient!

I never look at the TrueNas dashboard and never know exactly if everything worked or not, I just believe that the notification email will arrive if something goes wrong.

In this case, if rsync runs successfully, it sends a ping to healthchecks.io, if it fails, it doesn’t send and I receive notifications on Discord and email. In addition, I can configure the Grafana dashboard to fetch the status of each ping!

Final Result

The main pool zraid2 22TB: everything that’s important!

The secondary pool Stripe 550GB SSD: temporary shares, test VMs, and those Linux ISOs that come through torrents!

The secondary pool 1TB SSD Single Disk: VMs / Containers that require faster disks (game servers, for example).

The 500GB NVMe Pool: Windows VM as a secondary Windows machine.

What am I going to do with all this? Well… I don’t know yet, I’ll find out soon, but it will certainly fill up again when I start making skate and snowboard videos again.

TL;DR

Although ProxMox is not designed to serve as a NAS, it is quite flexible and capable of playing that role.

The virtualization capabilities benefit me more than NAS features, considering that all my file sharing goes through NextCloud running in Docker. And if I eventually really need an operating system specifically designed for NAS and file sharing, I can easily virtualize it and pass the hardware to the VM.

The migration was quite simple and took me no more than a couple of hours to get everything operational. In total, it took me half a dozen hours from the moment I turned off the server until everything was ready!

Finally, I wanted to leave an updated diagram with this change, but I don’t know what I did with the original. So, see here. Everything is the same, but instead of TrueNAS, I’m using ProxMox.