TrueNAS (nee FreeNAS)

Here’s a story about a small storage cube running FreeNAS.

When I set it up, it was the first time I’ve tried to create a RAID array, and I thought other people might be interested in how it’s set up, so here’s a quick writeup on how it fits together.

Edit: I’ve since created a handful of these things, with slightly higher specs, will update below.

Also, this was way back in February 2013, which, if you cast your mind back, was when the AUD was riding higher than the USD, Oscar Pistorius was still a free man, and no-one had yet heard of Miley Cyrus or why she needed to demonstrate her ability to break things whilst naked.

So bear that in mind.

Bits and pieces

To get this thing up and running, I purchased:

• a CFI A7879 Mini-ITX NAS Case with Four Hot-Swappable HDD/SSD Drive Bays (together with 5 compact SATA cables), for USD$283 ($139 for the case + $25 for the cables + $119 shipping). [1] Luckily the AUD was slightly stronger than the USD around this time.
• A C60M1-I motherboard (containing an integrated AMD Dual-Core Processor C-60 with AMD Radeon™ HD 6290 Graphics, and six on-board SATA connectors), from Umart for AUD$95.00. [2] .
• a stick of Kingston 4GB DDR3 1600Mhz PC3-12800 DIMM, which I purchased off ebay for AUD$24.90 inc P&P
• 4 Western Digital 3TB Green 64MB SATA III WD30EZRX drives, also from umart, for AUD$532.00. When cabling these up, rememeber to keep the drives in some sane order (e.g. the top drive connects to SATA slot 0, the second drive down connects to SATA slot 1 etc), which will help when replacing the things.
• a spare 8GB USB stick I had lying about, running a FreeNAS installation created from the FreeNAS installer ISO on the freenas.org website. The FreeNAS guys recommend that you run the OS from USB, and keep all the HDDs attached to the unit for storage purposes only, so I’ve done that [3].

So not a huge chunk of change left from a cool grand, all in all.

If I was shoving this in The Cloud with their new low-low prices, it’d set me back about $470/month though, so not too bad from that perspective [5].

So how did that all pan out then ?

It mostly works, except for the bits that don’t.

So you boot it up, tell it what it’s name is, what timezone it’s in, what type of keyboard won’t actually be attached to it for most of the time, what it’s purpose in life is, and how little it’s prospects are for ever playing a game of DOOM.

Then I set up a RAIDz2 volume (a.k.a. pool) ( mounted at /mnt/raidvolume ) spanning all four drives ( ada0, ada1, ada2 and ada3 ), since this allows any two drives to fail independently without affecting any of the data. (This is better than RAID1 or mirroring, which can lose data if the two drives in the mirror fail at the same time). This configuration reduces the effective capacity of the array to (number of drives) – 2 … since I’ve got 4 drives, down to 50%, or 6TB of the capacity of the system compared to a 12TB JBOD, but this is compensated for by enabling compression on the resulting array.

This is probably a good time to set the admin password, update the network settings, create a non-admin user and add it to the wheel group, run visudo if you need to, and enable ssh and cifs services.

Datasets

Within the RAIDz2 volume, I set up three top-level datasets with different levels of compression (none, lzjb and gzip/maximum), as well as SMB (cifs) shares on each to make them easier to access from Windows machines.

Don’t use gzip compression

I was intending to compare the speed and storage performance of these folders, but it turns out that I started getting I/O errors on the gzip folder when running cp on a remote machine via a samba share set up using cifs. Interestingly enough, it wasn’t even possible to kill -9 the cp process; I needed to umount -l and remount to recover. An strace on the running process shows nothing happening.

knoxg@bnehyp02:~$ sudo mount -t cifs //bnenas01/compressed2 /mnt/bnenas01-compressed2 -o username=knoxg,password=super-secret-password,domain=something-you-wouldnt-immediately-guess
knoxg@bnehyp02:~$ time sudo cp -r --preserve=timestamps /media/storage/work/* /mnt/bnenas01-compressed2/work
cp: closing `/mnt/bnenas01-compressed2/work/fromTheDawnOfMan/mdbs/DatabaseLinkyThing_Backup.mdb': Input/output error
cp: closing `/mnt/bnenas01-compressed2/work/fromTheDawnOfMan/notAsRecent/www/codeswarm/2008-11-23-head-ipod.mp4': Input/output error
(many more files here)

So rather than deal with that, I’m just using lzjb for now.

Although as of 2014, lzjb is now a legacy (deprecated) compression format — the new recommended compression format is lz4 so I may try setting that up at some stage.

Since I’m mainly using this storage unit as a backup facility, I’m not terribly concerned about random access read/write times; I imagine that compression would cause random writes within a file to suffer, but I haven’t done any tests. These people have though.

Most of my storage tasks on this machine involve dumping complete files from other machines, so it’s fast enough for me, and lzjb gets me a reasonable level of compression.

Anything else?

Well, yes. When I first installed this thing (on FreeNAS 8.3.1), the network driver for the integrated LAN chipset on the motherboard used to seize up occasionally, spouting messages on the console about re0 Watchdog Timeout Errors. Seems to be fine so far on whatever the hell I’m running at the moment though FreeNAS-9.1.1-RELEASE-x64.

Also, haven’t actually had a HDD go foom yet, so looking forward to that. Probably should have tested that before I loaded it up with data. Next time.

Rinse and repeat

Edit 2022-11-06: Well, I’ve managed to fill up three of these things now, so this is what I’m now using for NAS number 4:

• C2750D4I mobo, $697.40 ( wisp.net.au )
  this was initially a C2550D4I for $608.06 but they didn’t have it in stock, so was forced to upgrade)
• 8x Seagate Barracuda 8TB ST8000DM004 Desktop 3.5IN HDD, $1,592.00
• Silicon Power A55 256GB TLC 3D NAND 2.5in SATA III SSD for the boot drive, $33.00
• Cooler Master V 550W 80+ Gold SFX Power Supply, $134.00
• 4x 8gb SP016GLLTU160N22 DDR3L 1600MHz PC3-12800 1.35V CL11, $178.00 ( umart.com.au )
• Generic Internal USB 2.0 (MB-F) to USB3.0 19pin Adaptor Cable, $3.00
• SilverStone Black DS380 8 Bay Hot Swap SFF Chassis, $249.00 ( mwave.com.au )
• ATEN CS84U 4 Port KVM, $163.00 ( scorptec.com.au )

All prices in AUD inc P&P. I’m also using TrueNAS SCALE this time around, instead of FreeNAS. TrueNAS is from the same people, and still uses ZFS to provide redundancy, but runs on Ubuntu instead of OpenBSD. I also need to upgrade a bunch of other things at the same time, so might burn an afternoon writing that up one day.

If you’re interested, NASses 2 and 3 have also been in those silverstone enclosures, which take eight 3.5″ drives, which have been slowly increasing in capacity over time.