Garage Server, 3 Years Later

A bit over three years ago, I blogged about a SmartOS server I built and put in my garage in a 12u rack. It's been humming along great though my setup no longer looks quite the same of course. Times change, setups evolve.

Bye Virtualized Router, Hello More Hardware!

The first major change, was to stop running the router software (pfsense) in a KVM. That had to go. It isn't ideal or desired that doing a SmartOS update (swapping USB sticks and rebooting) means your internet goes down because the router was running on the same machine. Basic server maintenance was dreadful and avoided since any server outage meant no Internet.

pfsense is a lovely software router package and I looked into moving it to a stand-alone piece of hardware. Perhaps a Soekris box or one of the other little kit-type pieces of hardware that I could install pfsense on. In my router research I noticed that pre-built ready-to-run pfsense machines were quite a bit more expensive than I had anticipated. I also found out that the company that made the wifi access points I was using happened to have a router. So, I figured I'd give the EdgeRouter Pro instead of the slightly more cumbersome kit-style routers.

Ah, what a nice router to add to the rack!

Ah, what a nice router to add to the rack!

This is a router that requires some fairly extensive networking knowledge, but for $270 its a hell of a deal (and the forums are filled with knowledge and help). Ubiquiti has continued to roll out awesome upgrades, like the recent one adding deep packet inspection. I run this with dual-WAN failover using a local DSL provider and Comcast, to ensure a reliable Internet connection since I mainly work from home. The recent update also added support for fq_codel QoS which is the first QoS I've ever used that 'just works' with one click (as its meant to be).

ZFS is Amazing

I started out with 2 2TB mirror vdevs (virtual devices in ZFS parlance) for 4TB of usable space. This is similar to a traditional RAID-10 with writes being spread across the two vdevs. One of the nice things about using mirror vdev's is that you can slowly replace the drives over time with large capacity hard drives, and when all the hard drives in a vdev are the new large size.... the space becomes available.

Over the course of 18 months, I would buy a new 4TB drive, and add it to one of the vdev's. When a vdev had 2 of them, then my usable space expanded, first to 6TB, then to 8TB. Eventually I had 6 4TB hard drives in the 2 vdevs (3-way mirror), running for a month or two, to ensure the drives were solid and reliable before the big shift.

One of my theories on this path of upgrading is that I am putting different wear life-cycles on all the hard-drives in use which should theoretically reduce the odd's of multiple drives failing at once. It also distributes the upgrade cost over many months, and each drive generally fell in price as time went on.


ZFS has no way to migrate 2 vdevs into a single RAID-Z2 (aka RAID-6), which is what I wanted to do with the drives I now had. As a result, the only option is to move all the data to a separate machine and setup a new ZFS zpool with the desired configuration. I happened to have an old HTPC sitting around, and all those old 2TB drives, so I threw them in the HTPC and put FreeNAS on it.

ZFS has a special send/receive mode that allows it to move an entire ZFS dataset to another machine (or even another file, since these ZFS commands send data out as a stream). Since FreeNAS used a close enough version of ZFS as SmartOS, I was able to move my data over without any problems and verify its integrity on the new machine before re-configuring my server.

After playing around with FreeNAS over the course of a weekend, I decided that I'd use FreeNAS for my server instead of staying with SmartOS. The features on it and the GUI are quite well done and make the majority of tasks I actually want to do with my server quite easy. It's definitely obvious from usage that SmartOS is intended for the datacenter, and using it home is fun, but the intended use-case is rather different.

Four days later, I had moved my 5 TB of data off my server to the HTPC, reconfigured, and moved it back again to my new FreeNAS server with 14 TB of usable space. If these capacities sound large to you, I encourage you to read some of the FreeNAS builders's amazing how fast disk space disappears.

I figure this should be plenty of space for at least another 2 years, maybe 3... and just like with mirror vdevs I *could* slowly replace the 4TB drives with those new 8TB just coming out. Once all of them are replaced, my usable space would double.


It was time for a thorough dusting, using the DataVac Electric Duster I went through all the components in my rack for a cleaning. It's loud, but it gets the job done without the problems I've had in the past with the compressed air-in-a-can. After admiring my handiwork and snapping a photo, I figured it was time for an update on my last blog post.

A clean happy garage rack (I am obviously not an IT wiring guru)

A clean happy garage rack (I am obviously not an IT wiring guru)

On my next post, I'll review a year of being on Windows after 15 years of being an Apple/OSX fanboy. :)

Trying out SmartOS and OpenIndiana

After building my new server capable of running SmartOS, it was time to give it a spin!

If you’ve only built desktop machines, its hard to express how awesome IPMI KVM is. No longer do you need to grab another keyboard / video monitor / mouse (the KVM), you just plug in the IPMI Ethernet port on the motherboard to your switch and hit the web-server its running. It then lets you remotely access the machine as if you had it hooked up directly. You can get into the BIOS, boot from ISO’s on your local machine, hard reset, power down, power up, etc. It’s very slick and means I can stick the computer in the rack without needing to go near it to do everything that used to require a portable set of additional physical hardware.


This post assumes some basic knowledge of OS virtualization. In this case QEMU, KVM (which was ported by Joyent to run on SmartOS), and Zones. I generally refer to them as VM’s and will differentiate when I add a Zone vs. a KVM instance.

First Go at SmartOS

Installation is ridiculously easy, there is none. You download SmartOS, put it on a USB stick or CD-ROM, and boot the computer from it. I was feeling especially lazy and used the motherboards IPMI KVM interface to remotely mount the ISO image directly from my Mac.

Once SmartOS booted, it asked me to setup the main ZFS pool, and it was done. SmartOS runs a lot like a VMWare ESXI hyper-visor, with the assumption that the machine will only be booting VM’s. So the entire ZFS pool is just for your VM’s, which I appreciate greatly. After playing with it a little bit, it almost felt.... too easy.

I had really allocated at least a week or two of my spare time to fiddle around with the OS before I wanted it to just work, and having it running so quickly was almost disappointing.

The only bit that was slightly annoying was that retaining settings in the GZ (Global Zone) is kind of a pain. You have to drop in a service file (which is XML, joy!) on a path which SmartOS will then load and run on startup. This was mildly annoying, and some folks on the IRC channel suggested I give OpenIndiana a spin, which is aimed more at a home server / desktop scenario. There was also a suggestion that I give Sophos UTM a spin instead of pfsense for the firewall / router VM.


Since OpenIndiana has SmartOS‘s QEMU/KVM functionality (needed to run other OS’s like Linux/BSD/Windows under an illumos based distro), it seemed worth giving a go. It actually installs itself on the system unlike SmartOS, so I figured it’d take a little more space. No big deal. Until I installed it.

Then I saw that the ZFS boot pool can’t have disks in it larger than 2TB (well, it can, but it only lets you use 2TB of the space). Doh. After chatting with some IRC folks again, its common to use two small disks in a mirror as a ZFS boot pool and then have the much larger storage pool. Luckily I had a 250GB drive around so I could give this a spin, though I was bummed to have to use one of my drive bays just for a boot disk.

Installation went smoothly, but upon trying to fire up a KVM instance I was struck by how clunky it is in comparison to SmartOS. Again, this difference comes down to SmartOS optimizing the heck out of its major use-case.... virtualizing in the data-center. In SmartOS there’s a handy imgadm tool to manage available images, and vmadm to manage VM’s. These don’t seem to exist for OpenIndiana (maybe as an add-on package?), so you have to use the less friendly QEMU/KVM tools directly.

Then the KVM failed to start. Apparently the QEMU/KVM support in OpenIndiana (at least for my Sandy Bridge based motherboard) has been broken in the latest 3 OpenIndiana releases for the past 5 months. There’s a work-around to install a specific set of packages, but to claim QEMU/KVM support with such a glaring bug in a fairly prominent motherboard chip-set isn’t a good first start.

My first try to install the specific packages failed as my server kernel-panicked halfway through the QEMU/KVM package installation. Upon restarting, the package index was apparently corrupted. The only way to fix it is to re-install OpenIndiana... or rollback the boot environment (a feature utilizing ZFS thus including snapshots). Boot environments and the beadm tool to manage them are a bit beyond the scope of this entry, but the short version is that it let me roll-back the boot file-system including the package index to a non-mangled state (Very cool!).

With QEMU / KVM finally installed and working, I installed and configured Sophos UTM in a KVM and was off and running. Except it seemed to run abysmally slow... oh well, I was about to go on vacation anyways. I set the KVM to load at boot-time and restarted.

Upon loading the KVM at boot, the machine halted. This issue is apparently related to the broken QEMU / KVM packages. It was about time for my vacation, and I had now played with an OS with some rather rough edges in my spare time for a week. So I powered it off, took out the boot drive, and went on my vacation.

Back to SmartOS

When I got back from my vacation, I was no longer in the mood to deal with failures in the OS distribution. I rather like the OpenIndiana community, but now I just wanted my server to work. SmartOS fit the bill, and didn’t require boot drives which was greatly appreciated. It also has a working QEMU / KVM, since its rather important to Joyent. :)

In just a day, I went from a blank slate to a smoothly running SmartOS machine. As before, installation was dead simple, and my main ZFS pool zones (named as such by SmartOS) was ready for VM’s. Before I added a VM I figured I should have an easy way to access the ZFS file-system. I turned on NFS for the file-systems I wanted to access and gave my computer’s IP write privilege and the rest of the LAN read-only. This is insanely easy in ZFS:

zfs set sharenfs=rw=MYIP,ro= zones/media/Audio

To say the least, I love ZFS. Every other file-system / volume manager feels like a relic of the past in comparison. Mounting NFS file-systems on OSX used to suck, but now its a breeze. They work fast and reliably (thus far at least).

Setting Up the Router KVM

First, I needed my router / firewall KVM. I have a DSL connection, so I figured I’d wire that into one NIC, and have the other NIC on the motherboard go to the LAN. SmartOS virtualizes these so that each VM gets its own Virtual NIC (VNIC), this is part of the Solaris feature- set called Crossbow. Setting up the new KVM instance for Sophos UTM was simple, I gave it a VNIC on the physical interface connected to the DSL modem and another on the physical interface connected to my switch.

Besides for the fact that the VM was working without any issues like I had in OpenIndiana, I noticed it was much faster as well. Unfortunately for some reason it wasn’t actually routing my traffic. It took me about an hour (and clearing the head while walking the dog) to see that I was missing several important VNIC config options, such as dhcp_server, allow_ip_spoofing, allow_dhcp_spoofing, and allow_restricted_traffic.

These settings are needed for a VM that intends to act as a router so that it can move the packets and NAT them as appropriate across the VNICs. Once I set those everything ran smoothly.

So far, this only took me about 3 hours and was rather simple so I decided to keep going and get a nice network backup for the two OSX machines in the house.

Setting Up Network Backups

After some research I found out the latest version of netatalk would work quite nicely for network Time Machine backups. I created a zones/tmbackups ZFS file-system, and two nested file-systems under that for my wifes’ Macbook and my own Mac Mini. Then I told ZFS that zones/tmbackups should have compression enabled (Time Machine doesn’t actually compress its backups, transparent ZFS file compression FTW!) and I set quota’s on each nested file-system to prevent Time Machine from expanding forever.

Next I created a Zone with a SmartOS Standard dataset. Technically, the KVM instances run in a Zone for additional resource constraints and security, while I wanted to use just a plain Zone for the network backups. This was mainly because I wanted to make the zones/tmbackups file-system directly available to it without having to NFS mount it into a KVM.

If you’ve ever compiled anything from source in Solaris, you’re probably thinking about how many days I spent to get netatalk running in a Zone right now. Thankfully Joyent has done an awesome job bringing a lot of the common GNU compiler toolchain to SmartOS. It only took me about an hour to get netatalk running and recognized by both macs as a valid network Time Machine backup volume.

Unfortunately I can’t remember how exactly I set it up, but here are the pages that gave me the guidance I needed:

I’ve heard that netatalk 3.x is faster, and will likely upgrade that one of these days.

Setting Up the Media Server KVM

One of the physical machines I wanted to get rid of was the home theater PC I had built a few years back. It was rarely used, not very energy efficient, and XBMC was nowhere near spouse-friendly enough for my wife. We have an AppleTV and Roku, and I figured I’d give Plex a try on the Roku since the UI was so simple.

I setup a KVM instance and installed Ubuntu 12.04 server on it. Then I added the Plex repo’s and installed their Media Server packages. Fired it up and pointed Plex at my Video folders and it was ready to go. The Roku interface is slick and makes it a breeze to navigate. Being based on XBMC means that it can play all the same media and trans-codes it as necessary for the other network devices that want to play it.

At first Plex ran into CPU problems in the KVM... which I quickly realized was because I hadn’t changed the default resource constraints. The poor thing only had a single virtual CPU... after giving it a few more it easily had enough CPU allocated to do the video trans-coding.

While KVM runs CPU-bound tasks at bare-metal speed, disk I/O is virtualized. To reduce this problem I have Plex writing its trans- coded files to the ZFS file-system directly via an NFS mount. The media folders are also NFS mounted into the Media Server KVM.

I threw some other useful apps onto this KVM that I was running on the home theater PC and left it alone.

SmartOS Rocks

I now have a nice little home SmartOS server setup running that does a great job taking on jobs previously done by 2 other pieces of hardware. I still need to setup a base Ubuntu image to use for other development KVM’s, which I’ll blog about when I get that going. Despite being intended for the data-center, SmartOS works great for a home NAS / Media Server / Router system. I’m sure I’ll be even happier as I start to ramp up my use of development VM’s.

OpenIndiana is a small community taking on a big job. It’s a great community and people are very friendly. But you should expect to be hacking on things very early on if you use it, rather than playing with the other components. The SmartOS community is doing great too, and there’s more than a few forks that add some additional home-centric type functionality. So far I haven’t needed any of those enough to get me to try them out.

Anything else I should blog about regarding SmartOS or the rest of my setup?

Building A SmartOS Server

I’ve been reading about SmartOS for awhile now and have wanted to build a home server that would let me run VM’s with ZFS for the main file-system. Getting rid of my home theater PC and wireless router (which has been annoying me with its flakiness for months) was also a goal. Running something like pfsense in a VM would give me more options and theoretically be more stable than the fairly crappy software that seems to plague home consumer-grade wireless routers.

So after a month or so of research in my spare time, it seemed like SmartOS was going to be the best bet. Even though its generally intended for use in the datacenter, it had all the features I wanted (which I’ll blog about separately in my next post). Now I just needed a parts list that had already been verified to work with SmartOS, which is a bit pickier on hardware than the linux/BSD distributions.


Here’s what I ended up with:

  • CPU: Intel Xeon E3-1230 V2
  • Motherboard: SUPERMICRO MBD-X9SCL-F-O
  • Case: NORCO RPC-2212 Black 2U Rackmount Server Case with 12 Hot-Swappable SATA/SAS Drive Bays
  • HBA: LSI Internal SATA/SAS 9211-8i (Hooks up to 2 of the back-plane connectors in the case for 8 drives)
  • RAM: 16GB ECC (The 8 GB unbuffered sticks were unfortunately not around at the time or I would’ve gotten two of those to begin with)

I already had a 2TB and 3TB drive, so I bought one more of each so that I could run a ZFS storage pool with 2 vdev mirrors as Constantin Gonzalez blogs about regarding RAID vs. mirrors.

In retrospect, and after reading a bit more, I think I would’ve gotten one of the larger Norco 4U cases. Not because I need or want 20+ hot-swap bays, but because you can easily use a ‘desktop’ grade 80+ Titanium rated power supply. Finding a 2U 80+ PSU is difficult, a 80+ Titanium rated that puts all its power out on a single 5v rail is almost impossible. The cost savings in getting a good desktop-grade PSU with the Norco 4U case is about the same as the one I got with the more expensive 2U PSU.

I also bought a rack to put the server in along with my other home networking gear, so that it’d all be nicely packed away in a corner of the garage. Here’s a photo of the completed setup:

I have one of the cheaper Cisco SG300-10 switches which conveniently came with rack-mounts, and monoprice had a very affordable patch panel and blank plates to make it look tidy.

Overall cost: ~$2200

That includes the nice Tripp Lite SR12UB 12U Rack Enclosure which I’ve found handy to lock to ensure my toddler doesn’t yank out hard drives (he figured out how to pull out the hot-swap drive in all of 20 seconds when I was assembling it). Not that I let him run around the garage, but keeping everything locked is handy just in case.

OS Choice

When I was assembling and preparing to install SmartOS, some people on IRC mentioned that OpenIndiana might be a better choice for a home server. Suffice it to say it didn’t work out well, while SmartOS has been flawless now and running smoothly for the past two months.

My next post will have a lot more details on my OpenIndiana experience as well as how I have the SmartOS box setup.