Risk in a difficult concept and it requires a lot of training, thought and analysis to properly assess given scenarios. Often, because risk assessments are so difficult, we substitute risk analysis with simply adding basic redundancy and assuming that we have appropriately mitigated risk. But very often this is not the case. The introduction of complexity or additional failure modes often accompany the addition of redundancy and these new forms of failure have the potential to add more risk than the added redundancy removes. Storage systems are especially prone to these decision processes which is unfortunate as few, if any, systems are so susceptible to failure and more important to protect.
RAID is a great example of where a lack of holistic risk thinking can lead to some strange decision making. If we look at a not uncommon scenario we will see where the goal of protecting against drive failure can actually lead to an increase in risk even when additional redundancy is applied. In this scenario we will compare a twelve drive array consisting of twelve three terabyte SATA hard drives in a single array. It is not uncommon to hear of people choosing RAID 5 for this scenario to get “maximum capacity and performance” while having “adequate protection against failure.”
The idea here is that RAID 5 protects against the loss of a single drive which can be replaced and the array will rebuild itself before a second drive fails. That is great in theory, but the real risks of an array of this size, thirty six terabytes of drive capacity, come not from multiple drive failures as people generally suspect but from an inability to reliably rebuild the array after a single drive failure or from a failure of the array itself with no individual drives failing. The risk of a second drive failing is low, not non-existent, but quite low. Drives today are highly reliable. Once one drives fails it does increase the likelihood of a second drive failing, which is well documented, but I don’t want this risk to mislead us from looking at the true risks – the risk of a failed resilvering operation.
What happens that scares us during a RAID 5 resilver operation is that an unrecoverable read error (URE) can occur. When it does the resilver operation halts and the array is left in a useless state – all data on the array is lost. On common SATA drives the rate of URE is 10^14, or once every twelve terabytes of read operations. That means that a six terabyte array being resilvered has a roughly fifty percent chance of hitting a URE and failing. Fifty percent chance of failure is insanely high. Imagine if your car had a fifty percent chance of the wheels falling off every time that you drove it. So with a small (by today’s standards) six terabyte RAID 5 array using 10^14 URE SATA drives, if we were to lose a single drive, we have only a fifty percent chance that the array will recover assuming the drive is replaced immediately. That doesn’t include the risk of a second drive failing, only the risk of a URE failure. It also assumes that the drive is completely idle other than the resilver operation. If the drives are busily being used for other tasks at the same time then the chances of something bad happening, either a URE or a second drive failure, begin to increase dramatically.
With a twelve terabyte array the chances of complete data loss during a resilver operation begin to approach one hundred percent – meaning that RAID 5 has no functionality whatsoever in that case. There is always a chance of survival, but it is very low. At six terabytes you can compare a resilver operation to a game of Russian roulette with one bullet and six chambers and you have to pull the trigger three times. With twelve terabytes you have to pull it six times! Those are not good odds.
But we are not talking about a twelve terabyte array. We are talking about a thirty six terabyte array – which sounds large but this is a size that someone could easily have at home today, let alone in a business. Every major server manufacturer, as well as nearly all low cost storage vendors, make sub $10,000 storage systems in this capacity range today. Resilvering a RAID 5 array with a single drive failure on a thirty six terabyte array is like playing Russian roulette, one bullet, six chambers and pulling the trigger eighteen times! Your data doesn’t stand much of a chance. Add to that the incredible amount of time needed to resilver an array of that size and the risk of a second disk failing during that resilver window starts to become a rather significant threat. I’ve seen estimates of resilver times climbing into weeks or months on some systems. That is a long time to run without being able to lose another drive. When we are talking hours or days the risks are pretty low, but still present. When we are talking weeks or months of continuous abuse, as resilver operations are extremely drive intensive, the failure rates climb dramatically.
With an array of this size we can effectively assume that the loss of a single drive means the loss of the complete array leaving us with no drive failure protection at all. Now if we look at a drive of the same or better performance with the same or better capacity under RAID 0, which also has no protection against drive loss, we need only use eleven of the same drives that we needed twelve of for our RAID 5 array. What this means is that instead of twelve hard drives, each of which has a roughly three percent chance of annual failure, we have only eleven. That alone makes our RAID 0 array more reliable as there are fewer drives to fail. Not only do we have fewer drives but there is no need to write the parity block nor skip parity blocks when reading back lowering, ever so slightly, the mechanical wear and tear on the RAID 0 array for the same utilization giving it a very slight additional reliability edge. The RAID 0 array of eleven drives will be identical in capacity to the twelve drive RAID 5 array but will have slightly better throughput and latency. A win all around. Plus the cost savings of not needing an additional drive.
So what we see here is that in large arrays (large in capacity, not in spindle count) that RAID 0 actually passes RAID 5 in certain scenarios. When using common SATA drives this happens at capacities experienced even by power users at home and by many small businesses. If we move to enterprise SATA drives or SAS drives then the capacity number where this occurs becomes very high and is not a concern today but will be in just a few years when drive capacities get larger still. But this highlights how dangerous RAID 5 is in the sizes that we see today. Everyone understands the incredible risks of RAID 0 but it can be difficult to put into perspective that RAID 5′s issues are so extreme that it might actually be less reliable than RAID 0.
That RAID 5 might be less reliable than RAID 0 in an array of this size based on resilver operations alone is just the beginning. In a massive array like this the resilver time can take so long and exact such a toll on the drives that second drive failure starts to become a measurable risk as well. And then there are additional risks caused by array controller errors that can utilize resilver algorithms to destroy an entire array even when no drive failure has occurred. As RAID 0 (or RAID 1 or RAID 10) do not have resilver algorithms they do not suffer this additional risk. These are hard risks to quantify but what is important is that they are additional risks that accumulate when using a more complex system when a simpler system, without the redundancy, was more reliable from the outset.
Now that we have established that RAID 5 can be less reliable than RAID 0 I will point out the obvious dangers of RAID 0. RAID in general is used to mitigate the risk of a single, lone hard drive failing. We all fear a single drive simply failing and all data being lost. RAID 0, being a large stripe of drives without any form of redundancy, takes the risk of data loss of a single drive failing and multiplies it across a number of drives where any drive failing causes total loss of data to all drives. So in our eleven disk example above, if any of the eleven disks fails all is lost. It is clear to see where this is dramatically more dangerous than just using a single drive, all alone.
What I am trying to point out here is that redundancy does not mean reliability. Just because something is redundant, like RAID 5, provides no guarantee that it will always be more reliable than something that is not redundant.
My favourite analogy here is to look at houses in a tornado. In one scenario we build a house of brick and mortar. On the second scenario we build two redundant house, east out of straw (our builders are pigs, apparently.) When the tornado (or big bad wolf) comes along which is more likely to leave us with a standing house? Clearing one brick and mortar house has some significant reliability advantages over redundant straw houses. Redundancy didn’t matter, reliability mattered in the end.
Redundancy is often misleading because it is easy to quantify but hard to qualify. Redundancy is a black or white question: Is it redundant? Yes or no. Simple. Reliability is not so simple. Reliability is about failure rates and likelihoods. It is about statistics and analysis. As it is hard to quantify reliability in a meaningful way, especially when selling a project to the business people, redundancy often becomes a simple substitute for this complex concept.
The concept of using redundancy to misdirect questions of reliability also ends up applying to subsystems in very convoluted ways. Instead of making a “system” redundant it has become common to make a highly reliable, and low cost, subsystem redundant and treat subsystem redundancy as applying to the whole system. The most common example of this is RAID controllers in SAN products. Rather than having a redundant SAN (meaning two SANs) manufacturers will often make that one component not often redundant in normal servers redundant and then calling the SAN redundant – meaning a SAN that contains redundancy, which is not at all the same thing.
A good analogy here would be to compare having redundant cars meaning two complete, working cars and having a single car with a spare water pump in the trunk in case the main one fails. Clearly, a spare water pump is not a bad thing. But it is also a trivial amount of protection against car failure compared to having a second car ready to go. In one case the entire system is redundant, including the chassis. In the other we are making just one, highly reliable component redundant inside the chassis. It’s not even on par with having a spare tire which, at least, is a car component with a higher likelihood of failure.
Just like the myth of RAID 5 reliability and system/subsystem reliability, shared storage technologies like SANs and NAS often get treated in the same way, especially in regards to virtualization. There is a common scenario where a virtualization project is undertaken and people instinctively panic because a single virtualization host represents a single point of failure where, if it fails, many systems will all fail at once.
Using the term “single point of failure” causes a panic feeling and is a great means of steering a conversation. But a SPOF, as we like to call it, while something we like to remove when possible may not be the end of the world. Think about our brick house. It is a SPOF. Our two houses of straw are not. Yet a single breeze takes out our redundant solutions faster than our reliable SPOF. Looking for SPOFs is a great way to find points of fragility in a system, but do not feel that every SPOF must be made redundant in every scenario. Most businesses will find their best value having many SPOFs in place. Our real goal is reliability at appropriate cost, redundancy, as we have seen, is no substitute for reliability, it is simply a tool that we can use to achieve reliability.
The theory that many people follow when virtualizing is that they take their virtualization host and say “This host is a SPOF, so I need to have two of them and use High Availability features to allow for transparent failover!” This is spurred by the leading virtualization vendor making their money firstly by selling expensive HA add on products and secondly by being owned by a large storage vendor – so selling unnecessary or even dangerous additional shared storage is a big monetary win for them and could easily be the reason that they have championed the virtualization space from the beginning. Redundant virtualization hosts with shared storage sounds great but can be extremely misguided for several reasons.
The first reason is that removing the initial SPOF, the virtualization host, is replaced with a new SPOF, the shared storage. This accomplishes nothing. Assuming that we are using comparable quality servers and shared storage all we’ve done is move where the risk is, not change how big it is. The likelihood of the storage system failing is roughly equal to the likelihood of the original server failing. But in addition to shuffling the SPOF around like in a shell game we’ve also done something far, far worse – we have introduced chained or cascading failure dependencies.
In our original scenario we had a single server. If the server stayed working we are good, if it failed we were not. Simple. Now we have two virtualization hosts, a single storage server (SAN, NAS, whatever) and a network connecting them together. We have already determined that the risk of the shared storage failing is approximately equal to our total system risk in the original scenario. But now we have the additional dependencies of the network and the two front end virtualization nodes. Each of these components is more reliable than the fragile shared storage (anything with mechanical drives is going to be fragile) but that they are lower risk is not the issue, the issue is that the risks are combinatorial.
If any of these three components (storage, network or the front end nodes) fail then everything fails. The solution to this is to make the shared storage redundant on its own and to make the network redundant on its own. With enough work we can overcome the fragility and risk that we introduced by adding shared storage but the shared storage on its own is not a form of risk mitigation but is a risk itself which must be mitigated. The spiral of complexity begins and the cost associated with bringing this new system up on par with the reliability of the original, single server system can be astronomic.
Now that we have all of this redundancy we have one more risk to worry about. Managing all of this redundancy, all of these moving parts, requires a lot more knowledge, skill and preparation than does managing a simple, single server. We have moved from a simple solution to a very complex one. In my own anecdotal experience the real dangers of solutions like this come not from the hardware failing but from human error. Not only has little been done to avoid human error causing this new system to fail but we’ve added countless points where a human might accidentally bring the entire system, redundancy and all, right down. I’ve seen it first hand; I’ve heard the horror stories. The more complex the system the more likely a human is going to accidentally break everything.
It is critical that as IT professionals that we step back and look at complete systems and consider reliability and risk and think of redundancy simply as a tool to use in the pursuit of reliability. Redundancy itself is not a panacea. Neither is simplicity. Reliability is a complex problem to tackle. Avoiding simplistic replacements is an important first step in moving from covering up reliability issues to facing and solving them.
