Disk structure and CASL: The system itself is set up in a somewhat different way – all spinning disks are in RAID 6 (Dual Parity) with one spare (meaning you could technically lose up to 2 disks before you really had to start sweating bullets). The SSDs are not in any kind of array and that bears repeating – they not in any type of array. Putting anything in any type of protected RAID Array (meaning anything other than a RAID 0) means that you start to lose capacity. Each SSD is individually used to the fullest of it’s capacity for nothing but cached reads. So… the question arises “what if your SSD dies?”. Well… not much to be honest – the space that was used for cache is lost, and the data that was in that cache was lost, but it’s a read only cache – meaning that the data is already on the spinning disk, it’s not longer in cache. The data that is deemed worthy of cache will be re-pushed to cache as its worthiness is noted.
So, on to the data write operations. I made up a little flow chart that shows you just how the data is written to disk and in what process it is written in.
A few other little tid bits on writing data – compression is decent. I personally think LZ4 is a decent compression algorithm – in the past I’ve used several in SAN infrastructure, mainly ones like LZJB (pretty close to LZ4), the standard GZIP levels, and LZMA (once again, pretty close to LZ4). LZ4 strikes a pretty good balance between compression and CPU usage – obviously the higher compression, the more CPU usage it requires to compress data. Using things like Nexenta, I prefer the GZIPs over the LZJB – mainly because the CPU usage is not out of line (except when you use things like GZIP 8+, then it starts getting a little high).
According some Nimble Documentation, compression speeds happen at about 300 MB/s per CPU core. LZ4 (and most Lempel-Ziv based compression) has awesome decompression – decompression can happen at RAM speed in most cases (1 GB/s+). Compression is optional, though it is enabled by default. We have shut it off on several of volumes (namely the Veeam Volume, since Veeam compresses so well already that we weren’t gaining anything by having it on, and our Media server, since photos are already compressed, it doesn’t offer much in the way of compression). For things like VMs and SQL Server, I think you can expect to see a 1.5 – 2.5x compression ratio – SQL Server especially compresses well (2.25x+).
You’d think with compression, that you may notice a performance hit while compressing and decompressing data like that, be we have not noticed any type of performance hit, only huge improvements.
Next are the read operations – once again, a little flow chart:
So, let’s talk about cache for a second (accelerated reads). Any “hot” data is written to SSD Cache. The system serves that hot data from Cache and it obviously responds really quickly to changes. Obviously reads are a lot faster coming from SSD than HDD (RAID 6 reads aren’t horrible, especially when coming sequentially, but SSD is still generally a lot faster, in the realm of about 6 – 10 milliseconds on HDD vs 200 microseconds on SSD (about .2 milliseconds).
I may have previously mentioned that these use standard MLC SSDs (multi layer cell solid state). Generally speaking, in the enterprise, the “best” SSDs go in this order. SLC > eMLC > MLC > TLC, and TLC being the worst by quite a long shot. SLC’s, generally speaking, can endure more reads and writes than their MLC counterparts. If you want more info on why that is, read this. Honestly, when the engineer told us it was just plain MLC, I was like… wow, this product seems so genius and then throw MLC in there… what for? I asked that question while he was in there, and he gave me some pretty good answers – and the main one is that it all comes down to how you use it. SLC SSDs will work very well and endure random writes (by a lot, we are talking 100,000+ writes per cell). Since MLC drives have 2 bits written per cell, the cells can potentially wear out faster and since Nimble uses MLC, it converts random writes in to sequential writes, which minimizes “write amplification“, or basically spammy writes to cells, which thereby prolongs the life the SSD. Data on the SSDs are compressed and are “indexed” with metadata which also increases cached reads.
Next post, I’ll get in to networking and such.