A bit over three years ago, I blogged about a SmartOS server I built and put in my garage in a 12u rack. It's been humming along great though my setup no longer looks quite the same of course. Times change, setups evolve.
Bye Virtualized Router, Hello More Hardware!
The first major change, was to stop running the router software (pfsense) in a KVM. That had to go. It isn't ideal or desired that doing a SmartOS update (swapping USB sticks and rebooting) means your internet goes down because the router was running on the same machine. Basic server maintenance was dreadful and avoided since any server outage meant no Internet.
pfsense is a lovely software router package and I looked into moving it to a stand-alone piece of hardware. Perhaps a Soekris box or one of the other little kit-type pieces of hardware that I could install pfsense on. In my router research I noticed that pre-built ready-to-run pfsense machines were quite a bit more expensive than I had anticipated. I also found out that the company that made the wifi access points I was using happened to have a router. So, I figured I'd give the EdgeRouter Pro instead of the slightly more cumbersome kit-style routers.
This is a router that requires some fairly extensive networking knowledge, but for $270 its a hell of a deal (and the forums are filled with knowledge and help). Ubiquiti has continued to roll out awesome upgrades, like the recent one adding deep packet inspection. I run this with dual-WAN failover using a local DSL provider and Comcast, to ensure a reliable Internet connection since I mainly work from home. The recent update also added support for fq_codel QoS which is the first QoS I've ever used that 'just works' with one click (as its meant to be).
ZFS is Amazing
I started out with 2 2TB mirror vdevs (virtual devices in ZFS parlance) for 4TB of usable space. This is similar to a traditional RAID-10 with writes being spread across the two vdevs. One of the nice things about using mirror vdev's is that you can slowly replace the drives over time with large capacity hard drives, and when all the hard drives in a vdev are the new large size.... the space becomes available.
Over the course of 18 months, I would buy a new 4TB drive, and add it to one of the vdev's. When a vdev had 2 of them, then my usable space expanded, first to 6TB, then to 8TB. Eventually I had 6 4TB hard drives in the 2 vdevs (3-way mirror), running for a month or two, to ensure the drives were solid and reliable before the big shift.
One of my theories on this path of upgrading is that I am putting different wear life-cycles on all the hard-drives in use which should theoretically reduce the odd's of multiple drives failing at once. It also distributes the upgrade cost over many months, and each drive generally fell in price as time went on.
ZFS has no way to migrate 2 vdevs into a single RAID-Z2 (aka RAID-6), which is what I wanted to do with the drives I now had. As a result, the only option is to move all the data to a separate machine and setup a new ZFS zpool with the desired configuration. I happened to have an old HTPC sitting around, and all those old 2TB drives, so I threw them in the HTPC and put FreeNAS on it.
ZFS has a special send/receive mode that allows it to move an entire ZFS dataset to another machine (or even another file, since these ZFS commands send data out as a stream). Since FreeNAS used a close enough version of ZFS as SmartOS, I was able to move my data over without any problems and verify its integrity on the new machine before re-configuring my server.
After playing around with FreeNAS over the course of a weekend, I decided that I'd use FreeNAS for my server instead of staying with SmartOS. The features on it and the GUI are quite well done and make the majority of tasks I actually want to do with my server quite easy. It's definitely obvious from usage that SmartOS is intended for the datacenter, and using it home is fun, but the intended use-case is rather different.
Four days later, I had moved my 5 TB of data off my server to the HTPC, reconfigured, and moved it back again to my new FreeNAS server with 14 TB of usable space. If these capacities sound large to you, I encourage you to read some of the FreeNAS builders forums...it's amazing how fast disk space disappears.
I figure this should be plenty of space for at least another 2 years, maybe 3... and just like with mirror vdevs I *could* slowly replace the 4TB drives with those new 8TB just coming out. Once all of them are replaced, my usable space would double.
It was time for a thorough dusting, using the DataVac Electric Duster I went through all the components in my rack for a cleaning. It's loud, but it gets the job done without the problems I've had in the past with the compressed air-in-a-can. After admiring my handiwork and snapping a photo, I figured it was time for an update on my last blog post.
On my next post, I'll review a year of being on Windows after 15 years of being an Apple/OSX fanboy. :)