Problem statement:
- I want a NAS
- I want it to have a web GUI because I'm lazy and don't want to edit exports by hand anymore. Also, I don't want to learn samba. It's icky and I hate it.
- I would really like to virtualize it, because my system has so much horsepower it's a waste.
The combination of 2 and 3 is the sticky bit. Sure, I could just run proxmox and zfs and export some disks, but then I'm back to editing export files by hand. FreeNAS has a great gui, but the virtualization is lame, and quite frankly I'd rather run debian and wire it into my existing cluster.
So, idea #1. Lets run proxmox on the host, and then PCIe passthrough the SAS controller, and run FreeNAS as a VM.
So I need to do some benchmarks to find out if this is a bad idea or not. Easy enough, I install FreeNAS, build a simple raid5, run bonnie. Then tear it down, install proxmox, do the PCIe dance, install FreeNAS as a vm, repeat.
Interesting note.. When I did the virtualized one, the new install of freenas saw the zfs volumes on the disks once it was passed the controller, and just imported them. So if you do make the virtualization jump, you can go back super easy and not lose all your data.
So what does the above chart show us?
- benchy_Raw : FreeNas on bare metal.
- alcyone.garbled.net: My old NetBSD NFS box (different hardware completely, just for my ref)
- benchy.*: A bunch of different cpu, tuning, etc settings with FreeNAS under ProxMox.
- altair: Create a ZFS on proxmox directly.
- atest: Create an LXC in proxmox and access that ZFS.
Despite trying every possible combination of CPU tuning, tuning of the VM's, memory settings, etc etc, nothing I did made the virtualized FreeNAS operate anywhere near the speed of bare metal. What you can see however, is that the raw ZFS on ProxMox was even faster than FreeNAS. This actually remained true for a bunch of the other charts too, so it wasn't a one-off for just block IO.
Basically, I think FreeBSD running as a VM just isn't efficient. Probably doesn't like being virtualized by a kvm setup. Despite having raw access to the controller.
However.. those ProxMox numbers were really interesting. I want that speed. Those bore out in the different tests too. Overall, ProxMox ZFS is just faster than FreeNAS ZFS, at least for raid5.
OK, so, if we can do that, can we find a gui to run in Debian as an LXC to virtualize the NAS?
I'll save you the reading. Nope.
There are maybe 3 NAS GUI systems out there that are FOSS. FreeNAS, OpenMediaVault, and maybe a few others that are barely functional. (I don't consider webmin to be even slightly functional). OpenMediaVault was the only real choice.
OpenMediaVault is too smart for it's own good. Getting it installed on an LXC was a fight to the death. I had to do things to the code just to make it run. Things that made me think "I'll never be able to patch this.". But the worst part is that once you have it running, you are hosed.
You can't just export volumes that the OS sees. It has to mount them itself. So me passing the ZFS volumes down into the LXC doesn't fly, because the LXC obscures the mounts. I can't tell OMV to just export random stuff, because it refuses to acknowledge disks that aren't loaded into it's crazy config.xml stuff. Could I get it working? Oh sure, I could manually hack up all the configs and add the ZFS vols in there, but then I'm back to manually editing config files. This wasn't helpful.
Long story short, you can't get there from here. If you don't mind a 50% speed drop, sure, virtualize FreeNAS. If you just want an NFS server that is fast and you want to hand edit exports and setup samba, run it off ProxMox in a privleged LXC. If you want the GUI and the speed, then suck it up, FreeNAS gets the bare metal. Sorry. :(
Here are my raw benchmarks:
1.97,1.97,alcyone.garbled.net,1,1591389559,32G,,443,99,160716,69,152998,47,1225,75,288489,51,150.3,19,16,,,,,2376,30,+++++,+++,9936,34,4077,50,13073,100,6152,40,24024us,887ms,584ms,291ms,101ms,150ms,2757ms,33us,5110us,26516us,4050us,350ms
1.98,1.98,benchy_Raw_HW_single,1,1591380670,128G,,8192,5,74,99,349895,93,272187,96,186,99,610385,100,+++++,+++,16,,,,,19778,93,+++++,+++,7578,42,20613,97,+++++,+++,18193,98,114ms,73838us,24471us,70338us,158us,1044us,4117us,118us,1210ms,4008us,49us,128us
1.98,1.98,benchyProx_2s_4c_49G,1,1591380670,100768M,,8192,5,35,99,137069,96,106964,97,60,99,188795,99,7940,1488,16,,,,,9534,97,26626,99,6874,71,9585,98,27328,99,9442,97,290ms,113ms,41189us,171ms,35006us,8961us,5203us,1779us,654ms,6609us,1757us,8803us
1.98,1.98,benchy1s_8c_tuned,1,1591380670,100768M,,8192,5,38,99,150918,96,114459,98,61,99,192603,99,7432,1507,16,,,,,7759,98,26412,99,6383,71,8327,98,27284,99,9163,97,243ms,35298us,35295us,188ms,34384us,17784us,7463us,385us,694ms,7318us,6268us,7078us
1.98,1.98,benchy1s_8c_tuned_px624,1,1591384121,100768M,,8192,5,40,99,153354,95,119443,98,64,99,200707,99,15828,1540,16,,,,,10799,97,28283,99,7373,67,10574,98,29210,99,9530,87,227ms,30979us,23196us,157ms,16448us,2279us,4326us,1914us,678ms,4208us,149us,2218us
1.98,1.98,benchy_4s_2c_tuned_aes,1,1591384011,100768M,,8192,5,39,99,152795,96,116684,98,63,99,198334,99,15038,1451,16,,,,,10671,96,28775,99,7573,69,11101,98,29193,99,11162,98,267ms,45382us,23038us,163ms,16270us,2882us,4228us,2157us,610ms,4182us,103us,2160us
1.98,1.98,benchy_1s_8c_tuned_aes,1,1591386881,100768M,,8192,5,40,99,153306,96,116975,97,62,99,196813,99,14565,1420,16,,,,,10785,96,28872,99,7359,67,10972,98,29002,99,10922,98,360ms,42470us,21355us,149ms,16991us,15851us,4570us,2082us,701ms,4157us,2094us,2234us
1.98,1.98,benchy_1s_8c_q35,1,1591393615,100768M,,8192,5,40,99,157849,97,118986,97,66,99,204488,99,13896,1384,16,,,,,11054,96,29882,99,7559,67,11235,98,30777,98,11394,98,222ms,27786us,18644us,138ms,16394us,17061us,4333us,170us,665ms,4106us,64us,2166us
1.98,1.98,altair,1,1591454161,126G,,8192,5,79,99,345720,90,289710,93,156,99,671893,99,+++++,+++,16,,,,,16003,96,+++++,+++,1938,14,16056,96,+++++,+++,7233,57,106ms,42458us,35619us,79298us,504us,6112us,5187us,1358us,6183ms,5583us,15us,136ms
1.98,1.98,atest,1,1591459098,126G,,8192,5,72,99,335515,91,277318,93,149,99,651055,99,+++++,+++,16,,,,,13288,97,+++++,+++,1764,14,13432,96,+++++,+++,7243,62,125ms,38922us,39366us,76683us,267us,6266us,6027us,1417us,7038ms,5847us,29us,111ms
