Nexenta Appliance Setup w/ VLANs and Aggregates
Tue 30 November 2010
Filed under Howto
I have a Dell server which is now deprecated in favor of a NetApp FAS2040. I
tried FreeNAS with UFS and with ZFS. I am going to rate FreeNas as
"fairly lame". That's fairly harsh considering that it's great in concept, and pretty great at
execution. However, tweaking to take advantage of 20GB of ram was
not well documented, the community is a little fragmented, and the
project is in the midst of recovery after a fork. The ZFS pool version is only 13,
has a great many things of interest to sysadmins.
We wanted to find an acceptable 2nd-Tier solution to act as
a host for testing, demos, pre-production projects and
post-production retirement systems. Towards that end we were
looking for stability, a certain robustness and assistance in
Despite my best efforts I found that FreeNAS and the community are
tuned around getting as much as you can from as little as possible.
This runs completely opposite to my hardware and purpose.
FreeNas 0.7.2, based on FreeBSD 0.7.3
pool v13 which is backported from FreeBSD 8.x.
Nexenta 3.0.4, based on OpenSolaris with ZFS pool v26, which has a lot of changes since 13, including dedup and compression.
My hardware is modest but servicable.
- Dell PowerEdge 2900
- 2x Quad Core 5400 Series Xeons
- 20GB, PC2-5300
- 1x 4GB CF boot-disk, attached to motherboard SATA.
- 8x 750GB Sata-II 7200 RPM Disks, via Perc5/i
As an aside, we're adding about 30 more spindles in rougly two
months. That probably means paying Nexenta, but I'll cross that
bridge when I come to it.
Despite much effort with FreeNas my write speed from VMware ESXi 4.1,
using a 4-channel load-balanced link-group and Round-Robin IO pathing,
was a spikey 80MB/s that averaged out to ~40MB/s.
Furthermore, after lots of tuning, it was
still pretty dim. I messed with every setting I could. I tweaked
the TCP stack, somewhat endlessly, but ultimately my effort seems wasted.
No one has a clear idea for how I would actually use this for my
primary storage against an almost-production system. The last straw
was the unending problems with spikey traffic. I would hit 80MB/s, then
drop to 1MB/s, then ramp up and drop and up and down. The effect was
still reasonable transfer times, yet I wanted to see if something
would pan out with more stability or clarify my problem. FreeNas
appears to do neither for me. I also hungered for the more advanced features
Specifically, I require snapshots, but
I am using ZFS in anticipation of dedup and compression. Those
features are nice, but I was willing to wait. FreeBSD will
eventually trundle up to ready line. However, lacking
this other area of satisfaction, e.g. consistent performance and some
certainty of how to tune, I felt that I would try moving on.
My solution was to try Nexenta. It's name naturally implies it's
position in my task list. After all, the big thing which Tier-1
SAN vendors have to offer is simply software. Nexenta is
closer to the ZFS source tree and therefore more quickly up to date.
I installed Nexenta 3.0.4, and it loaded easily. I installed to the
CF card and, apart from the nearly hour-long install process, it was good.
After the load was complete, I rebooted into network hell.
The network didn't come up. It reported some two of four interfaces
down, and the system was just not what I expected. I tried to
login on the command line - I have never read a single page of
documentation, I just tried stuff - and root's password was
indeed nexenta. That hurdle crossed, I tried to
use ifconfig, which was not in evidence. I found the command-line a
little insane. After a few minutes, I discovered that most of
the problems were caused by the magical "nmc shell". So I began with
the first rule of modern Unix: "TAB". After striking "TAB" a few
times, I found an entire
commands. Just kidding, it was like two dozen. The command
labeled "setup" was interesting, so that was my beginning.
Verdict: I like sane systems, and this is sane. I found little
weirdness in the setup practice.
- ifaces: bnx0, bnx1, e1000g0, e1000g1
- vlans: 20, 30, 40, 41, 42, 43, 44, 45
The problem with Aggregates in the Nexenta interface is that you are
unable to add VLANs to them. I was similarly unable to find an easy
way to change the MTU of the target NICs. I resorted to various means
common to Slowaris for this.
The Real Command Line
The wiki shows the route to accessing the command line:
nmc@zetta:/$ option expert_mode = 1 nmc@zetta:/$ !bash You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? Yes root@zetta:/volumes#
From there, I edited these files.
After simple changes, which are clearly documented in various guides
and the files themselves, I was ready for the next reboot.
After this, I resumed the shell in order to create a 802.3ad bonded
link. You can sort out the nomenclature from
Aggregation article. I am using Extreme x405a Switches for my
SAN switch fabric and I am using L3 algorithm port sharing. I want to
use the same thing with Nexenta. I found a way not to use LACP, but I
couldn't find a way to setup L3 trunking policy. This is the command
line I ended up with:
setup network aggregation create -l off bnx0 bnx1 e1000g0 e1000g1 setup network interface aggr1 vlan create -v 20 setup network interface aggr1 vlan create -v 30 setup network interface aggr1 vlan create -v 40 ..[more vlan creation].. ## ## Primary Mangement IF setup network interface aggr30001 static -i 10.1.1.49 -m 255.255.0.0 -g 10.1.1.1 -1 10.1.1.254 -2 10.1.1.1 -3 10.3.1.1 ## ## Secondary Management IF. setup network interface aggr20001 static -i 10.3.1.49 -m 255.255.0.0 -g 10.3.1.1 -1 10.1.1.254 -2 10.1.1.1 -3 10.3.1.1 # setup network interface aggr40001 static -i 10.140.0.49 -m 255.255.255.0 -g 10.1.1.1 -1 10.1.1.254 -2 10.1.1.1 -3 10.3.1.254 ...[more vlan address assignment]...
The problem was that my trunking was actually reported at "L3,L4",
which I haven't been using. For the sake of consistency in the
datacenter, I elected to discover a way to change the dl policy to
L3. Again, Expert Mode and the command line worked:
nmc@zetta:/$ option expert_mode = 1 nmc@zetta:/$ !bash You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? Yes root@zetta:/volumes# dladm show-aggr LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS aggr1 L3,L4 auto off short ----- root@zetta:/volumes# dladm modify-aggr -P L3 aggr1 root@zetta:/volumes# dladm show-aggr LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS aggr1 L3 auto off short ----- root@zetta:/volumes# exit Important: To re-sync the appliance's management state information, please consider running 'setup appliance nms restart' command. nmc@zetta:/$ setup appliance nms restart
Once this was complete, I had usable web-management, filtered neatly
through the rabbit warren of Aggregation NIC and Policies, VLANs,
Switching and other concerns.
Note: At this point, I rebooted.
FreeNas has some killer features. I tried very hard to leverage the
system to my benefit. It was difficult to find precision sizing
recommendation out there on the internet. Whatever the case is, here
are the changes I made for using 20Gb, with ZFS.
ifconfig_bce0="mtu 9000" ifconfig_bce1="mtu 9000" ifconfig_em0="mtu 9000 polling" ifconfig_em1="mtu 9000 polling" ifconfig_lagg0="create laggproto loadbalance laggport bce0 laggport bce1 laggport em0 laggport em1 mtu 9000"
kern.ipc.maxsockbuf="33554432" kern.ipc.nmbclusters="32768" kern.ipc.somaxconn="8192" kern.polling.enable="1" net.inet.tcp.delayed_ack="0" net.inet.tcp.inflight.enable="0" net.inet.tcp.inflight.min="6144" net.inet.tcp.path_mtu_discovery="0" net.inet.tcp.recvbuf_inc="524288" net.inet.tcp.recvbuf_max="16777216" net.inet.tcp.recvspace="524288" net.inet.tcp.rfc1323="1" net.inet.tcp.sendbuf_max="16777216" net.inet.tcp.sendspace="524288" net.inet.udp.maxdgram="57344" net.inet.udp.recvspace="524288" net.local.stream.recvspace="82320" net.local.stream.sendspace="82320"
FreeNAS, ZFS and iSCSI
If you're going to use ZFS with iSCSI then you need to do some work
from the command line. Use the interface to Setup your Pool, which
is turfed well enough by others. But for the iSCSI "ZFS Volume" option
setup, you need the command line:
zfs create -V 1T myZvol/iSCSIview0
I don't know how to make a "dataset" which works this way from the
interface, but it works. After making this change, load the
web-interface, go to Disk->ZFS, Configuration->Synchronize and click
"Synchronize". Once complete, I found my volume appearing under
"ZFS Volume" in iSCSI Extent configuration.