Hardware and Network considerations: The first thing you need to know is that Nimble is 100% iSCSI based. There is no fiber channel option, it’s either 1GB ethernet or 10GB ethernet. At the time we were looking in to this, we were using Fibre Channel on our Compellent. I had always assumed that Fibre Channel was always superior to ethernet for storage because of the latencies involved.
I’ve found over time that this is not always the case. There is always a bottleneck in storage networking – whether it’s the disk array being too slow to perform all the writes requested, the network being saturated with read or write requests, the server doing the writing could be queuing disk writes, the storage device could be having a hard time keeping up with all the IO activity being requested of it, there are any number of things that could cause some slowdown.
I was very curious as to why Nimble would have (what they claimed to be and it turned out true) such a fast storage device a not offer something like 8G or 16G fiber – to me, it felt like they were kind of shooting themselves in the foot by doing this, and on top of that, by even offering something with 4 single gigabit NICs (for a total bandwidth of 4gbp/s or about 475 megs per second). After we had the chat with the engineer we found that it’s really not all about storage bandwidth, it’s about the number of operations happening. For example, if you have 3000 operations happening in a given second and your storage device does not have the capacity to handle that, it will begin to get saturated and utilization will go up – our network utilization is never that high on the Nimble because of how fast it processes those transactions or IOs.
There a few things you REALLY need to do to take full advantage of that though, and that is multipathing. Multipathing makes a HUGE difference in performance. Nimble has a couple of “Best Practices” that make it super efficient and very fast even over 1GB ethernet. If you use 10GB ethernet, I don’t think you’re going to have to worry about network bottlenecking. There is also a really handy Connection Manager coming up with the GA of Nimble OS 2.0.
Here is what I would suggest – you need a good switch. Whether 1GB or 10GB you need a good switch (probably L2 at the least – if you just get a dumb switch, you’re not going to get the performance you want) – something newer that has support for Jumbo Frames, Unicast Storm Control, STP / RSTP, Buffered Ports, etc… If you want good performance, I would highly suggest getting a nice switch. Netgear has some pretty good cheap ones, Dell has some decent ones, HP has some good ones, and of course, Cisco has some good ones. Don’t cheap out on the switch.