Nexenta Appliance Setup w/ VLANs and Aggregates

Tue 30 November 2010

Filed under Howto

The Case

I have a Dell server which is now deprecated in favor of a NetApp FAS2040. I

tried FreeNAS with UFS and with ZFS. I am going to rate FreeNas as

"fairly lame". That's fairly harsh considering that it's great in concept, and pretty great at

execution. However, tweaking to take advantage of 20GB of ram was

not well documented, the community is a little fragmented, and the

project is in the midst of recovery after a fork. The ZFS pool version is only 13,

and ZFS

has a great many things of interest to sysadmins.

We wanted to find an acceptable 2nd-Tier solution to act as

a host for testing, demos, pre-production projects and

post-production retirement systems. Towards that end we were

looking for stability, a certain robustness and assistance in

cleaning

up accidents.

Despite my best efforts I found that FreeNAS and the community are

tuned around getting as much as you can from as little as possible.

This runs completely opposite to my hardware and purpose.

The Contenders

  • FreeNas 0.7.2, based on FreeBSD 0.7.3

    with ZFS

    pool v13 which is backported from FreeBSD 8.x.

  • Nexenta 3.0.4, based on OpenSolaris with ZFS pool v26, which has a lot of changes since 13, including dedup and compression.

My hardware is modest but servicable.

  • Dell PowerEdge 2900
  • 2x Quad Core 5400 Series Xeons
  • 20GB, PC2-5300
  • 1x 4GB CF boot-disk, attached to motherboard SATA.
  • 8x 750GB Sata-II 7200 RPM Disks, via Perc5/i

As an aside, we're adding about 30 more spindles in rougly two

months. That probably means paying Nexenta, but I'll cross that

bridge when I come to it.

The Problem

Despite much effort with FreeNas my write speed from VMware ESXi 4.1,

using a 4-channel load-balanced link-group and Round-Robin IO pathing,

was a spikey 80MB/s that averaged out to ~40MB/s.

Furthermore, after lots of tuning, it was

still pretty dim. I messed with every setting I could. I tweaked

the TCP stack, somewhat endlessly, but ultimately my effort seems wasted.

No one has a clear idea for how I would actually use this for my

primary storage against an almost-production system. The last straw

was the unending problems with spikey traffic. I would hit 80MB/s, then

drop to 1MB/s, then ramp up and drop and up and down. The effect was

still reasonable transfer times, yet I wanted to see if something

would pan out with more stability or clarify my problem. FreeNas

appears to do neither for me. I also hungered for the more advanced features

of ZFS.

Specifically, I require snapshots, but

I am using ZFS in anticipation of dedup and compression. Those

features are nice, but I was willing to wait. FreeBSD will

eventually trundle up to ready line. However, lacking

this other area of satisfaction, e.g. consistent performance and some

certainty of how to tune, I felt that I would try moving on.

My solution was to try Nexenta. It's name naturally implies it's

position in my task list. After all, the big thing which Tier-1

SAN vendors have to offer is simply software. Nexenta is

closer to the ZFS source tree and therefore more quickly up to date.

Nexenta Setup

I installed Nexenta 3.0.4, and it loaded easily. I installed to the

CF card and, apart from the nearly hour-long install process, it was good.

After the load was complete, I rebooted into network hell.

The network didn't come up. It reported some two of four interfaces

down, and the system was just not what I expected. I tried to

login on the command line - I have never read a single page of

documentation, I just tried stuff - and root's password was

indeed nexenta. That hurdle crossed, I tried to

use ifconfig, which was not in evidence. I found the command-line a

little insane. After a few minutes, I discovered that most of

the problems were caused by the magical "nmc shell". So I began with

the first rule of modern Unix: "TAB". After striking "TAB" a few

times, I found an entire

murder of

commands. Just kidding, it was like two dozen. The command

labeled "setup" was interesting, so that was my beginning.

Verdict: I like sane systems, and this is sane. I found little

weirdness in the setup practice.



Basically this is what I want:
  • aggr1

    • ifaces: bnx0, bnx1, e1000g0, e1000g1
    • vlans: 20, 30, 40, 41, 42, 43, 44, 45

The problem with Aggregates in the Nexenta interface is that you are

unable to add VLANs to them. I was similarly unable to find an easy

way to change the MTU of the target NICs. I resorted to various means

common to Slowaris for this.

The Real Command Line

The wiki shows the route to accessing the command line:

nmc@zetta:/$ option expert_mode = 1
nmc@zetta:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute?  Yes
root@zetta:/volumes#

From there, I edited these files.

  • /kernel/drv/bnx.conf
  • /kernel/drv/e1000g.conf

After simple changes, which are clearly documented in various guides

and the files themselves, I was ready for the next reboot.

After this, I resumed the shell in order to create a 802.3ad bonded

link. You can sort out the nomenclature from

Wikipedia's Link

Aggregation article. I am using Extreme x405a Switches for my

SAN switch fabric and I am using L3 algorithm port sharing. I want to

use the same thing with Nexenta. I found a way not to use LACP, but I

couldn't find a way to setup L3 trunking policy. This is the command

line I ended up with:

setup network aggregation create -l off bnx0 bnx1 e1000g0 e1000g1
setup network interface aggr1 vlan create -v 20
setup network interface aggr1 vlan create -v 30
setup network interface aggr1 vlan create -v 40
..[more vlan creation]..
##
##  Primary Mangement IF
setup network interface aggr30001 static -i 10.1.1.49 -m 255.255.0.0 -g 10.1.1.1  -1 10.1.1.254 -2 10.1.1.1 -3 10.3.1.1
##
##  Secondary Management IF.
setup network interface aggr20001 static -i 10.3.1.49 -m 255.255.0.0 -g 10.3.1.1 -1 10.1.1.254 -2 10.1.1.1 -3 10.3.1.1
#
setup network interface aggr40001 static -i 10.140.0.49 -m 255.255.255.0 -g 10.1.1.1 -1 10.1.1.254 -2 10.1.1.1 -3 10.3.1.254
...[more vlan address assignment]...

The problem was that my trunking was actually reported at "L3,L4",

which I haven't been using. For the sake of consistency in the

datacenter, I elected to discover a way to change the dl policy to

L3. Again, Expert Mode and the command line worked:

nmc@zetta:/$ option expert_mode = 1
nmc@zetta:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute?  Yes
root@zetta:/volumes# dladm show-aggr
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L3,L4    auto                 off           short       -----
root@zetta:/volumes# dladm modify-aggr -P L3 aggr1
root@zetta:/volumes# dladm show-aggr
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L3       auto                 off           short
-----
root@zetta:/volumes# exit
Important: To re-sync the appliance's management state information, please
consider running 'setup appliance nms restart' command.
nmc@zetta:/$ setup appliance nms restart

Once this was complete, I had usable web-management, filtered neatly

through the rabbit warren of Aggregation NIC and Policies, VLANs,

Switching and other concerns.

Note: At this point, I rebooted.

**Important Notes

FreeNas has some killer features. I tried very hard to leverage the

system to my benefit. It was difficult to find precision sizing

recommendation out there on the internet. Whatever the case is, here

are the changes I made for using 20Gb, with ZFS.


loader.conf
vm.kmem_size="16106127360"
vfs.zfs.arc_max="15032385536"

rc.conf

ifconfig_bce0="mtu 9000"
ifconfig_bce1="mtu 9000"
ifconfig_em0="mtu 9000 polling"
ifconfig_em1="mtu 9000 polling"
ifconfig_lagg0="create laggproto loadbalance laggport bce0 laggport bce1 laggport em0 laggport em1 mtu 9000"

sysctl.conf

kern.ipc.maxsockbuf="33554432"
kern.ipc.nmbclusters="32768"
kern.ipc.somaxconn="8192"
kern.polling.enable="1"
net.inet.tcp.delayed_ack="0"
net.inet.tcp.inflight.enable="0"
net.inet.tcp.inflight.min="6144"
net.inet.tcp.path_mtu_discovery="0"
net.inet.tcp.recvbuf_inc="524288"
net.inet.tcp.recvbuf_max="16777216"
net.inet.tcp.recvspace="524288"
net.inet.tcp.rfc1323="1"
net.inet.tcp.sendbuf_max="16777216"
net.inet.tcp.sendspace="524288"
net.inet.udp.maxdgram="57344"
net.inet.udp.recvspace="524288"
net.local.stream.recvspace="82320"
net.local.stream.sendspace="82320"

FreeNAS, ZFS and iSCSI

If you're going to use ZFS with iSCSI then you need to do some work

from the command line. Use the interface to Setup your Pool, which

is turfed well enough by others. But for the iSCSI "ZFS Volume" option

setup, you need the command line:

zfs create -V 1T myZvol/iSCSIview0

I don't know how to make a "dataset" which works this way from the

interface, but it works. After making this change, load the

web-interface, go to Disk->ZFS, Configuration->Synchronize and click

"Synchronize". Once complete, I found my volume appearing under

"ZFS Volume" in iSCSI Extent configuration.


Comments


Up To Something © Joshua M Schmidlkofer Powered by Pelican and Twitter Bootstrap. Icons by Font Awesome and Font Awesome More