When No Redundancy Is More Reliable – The Myth of Redundancy

Risk in a difficult concept and it requires a lot of training, thought and analysis to properly assess given scenarios.  Often, because risk assessments are so difficult, we substitute risk analysis with simply adding basic redundancy and assuming that we have appropriately mitigated risk.  But very often this is not the case.  The introduction of complexity or additional failure modes often accompany the addition of redundancy and these new forms of failure have the potential to add more risk than the added redundancy removes.  Storage systems are especially prone to these decision processes which is unfortunate as few, if any, systems are so susceptible to failure and more important to protect.

RAID is a great example of where a lack of holistic risk thinking can lead to some strange decision making.  If we look at a not uncommon scenario we will see where the goal of protecting against drive failure can actually lead to an increase in risk even when additional redundancy is applied.  In this scenario we will compare a twelve drive array consisting of twelve three terabyte SATA hard drives in a single array.  It is not uncommon to hear of people choosing RAID 5 for this scenario to get “maximum capacity and performance” while having “adequate protection against failure.”

The idea here is that RAID 5 protects against the loss of a single drive which can be replaced and the array will rebuild itself before a second drive fails.  That is great in theory, but the real risks of an array of this size, thirty six terabytes of drive capacity, come not from multiple drive failures as people generally suspect but from an inability to reliably rebuild the array after a single drive failure or from a failure of the array itself with no individual drives failing.  The risk of a second drive failing is low, not non-existent, but quite low.  Drives today are highly reliable. Once one drives fails it does increase the likelihood of a second drive failing, which is well documented, but I don’t want this risk to mislead us from looking at the true risks – the risk of a failed resilvering operation.

What happens that scares us during a RAID 5 resilver operation is that an unrecoverable read error (URE) can occur.  When it does the resilver operation halts and the array is left in a useless state – all data on the array is lost.  On common SATA drives the rate of URE is 10^14, or once every twelve terabytes of read operations.  That means that a six terabyte array being resilvered has a roughly fifty percent chance of hitting a URE and failing.  Fifty percent chance of failure is insanely high.  Imagine if your car had a fifty percent chance of the wheels falling off every time that you drove it.  So with a small (by today’s standards) six terabyte RAID 5 array using 10^14 URE SATA drives, if we were to lose a single drive, we have only a fifty percent chance that the array will recover assuming the drive is replaced immediately.  That doesn’t include the risk of a second drive failing, only the risk of a URE failure.  It also assumes that the drive is completely idle other than the resilver operation.  If the drives are busily being used for other tasks at the same time then the chances of something bad happening, either a URE or a second drive failure, begin to increase dramatically.

With a twelve terabyte array the chances of complete data loss during a resilver operation begin to approach one hundred percent – meaning that RAID 5 has no functionality whatsoever in that case.  There is always a chance of survival, but it is very low.  At six terabytes you can compare a resilver operation to a game of Russian roulette with one bullet and six chambers and you have to pull the trigger three times.  With twelve terabytes you have to pull it six times!  Those are not good odds.

But we are not talking about a twelve terabyte array.  We are talking about a thirty six terabyte array – which sounds large but this is a size that someone could easily have at home today, let alone in a business.  Every major server manufacturer, as well as nearly all low cost storage vendors, make sub $10,000 storage systems in this capacity range today.  Resilvering a RAID 5 array with a single drive failure on a thirty six terabyte array is like playing Russian roulette, one bullet, six chambers and pulling the trigger eighteen times!  Your data doesn’t stand much of a chance.  Add to that the incredible amount of time needed to resilver an array of that size and the risk of a second disk failing during that resilver window starts to become a rather significant threat.  I’ve seen estimates of resilver times climbing into weeks or months on some systems.  That is a long time to run without being able to lose another drive.  When we are talking hours or days the risks are pretty low, but still present.  When we are talking weeks or months of continuous abuse, as resilver operations are extremely drive intensive, the failure rates climb dramatically.

With an array of this size we can effectively assume that the loss of a single drive means the loss of the complete array leaving us with no drive failure protection at all.  Now if we look at a drive of the same or better performance with the same or better capacity under RAID 0, which also has no protection against drive loss, we need only use eleven of the same drives that we needed twelve of for our RAID 5 array.  What this means is that instead of twelve hard drives, each of which has a roughly three percent chance of annual failure, we have only eleven.  That alone makes our RAID 0 array more reliable as there are fewer drives to fail.  Not only do we have fewer drives but there is no need to write the parity block nor skip parity blocks when reading back lowering, ever so slightly, the mechanical wear and tear on the RAID 0 array for the same utilization giving it a very slight additional reliability edge.  The RAID 0 array of eleven drives will be identical in capacity to the twelve drive RAID 5 array but will have slightly better throughput and latency.  A win all around.  Plus the cost savings of not needing an additional drive.

So what we see here is that in large arrays (large in capacity, not in spindle count) that RAID 0 actually passes RAID 5 in certain scenarios.  When using common SATA drives this happens at capacities experienced even by power users at home and by many small businesses.  If we move to enterprise SATA drives or SAS drives then the capacity number where this occurs becomes very high and is not a concern today but will be in just a few years when drive capacities get larger still.  But this highlights how dangerous RAID 5 is in the sizes that we see today.  Everyone understands the incredible risks of RAID 0 but it can be difficult to put into perspective that RAID 5′s issues are so extreme that it might actually be less reliable than RAID 0.

That RAID 5 might be less reliable than RAID 0 in an array of this size based on resilver operations alone is just the beginning.  In a massive array like this the resilver time can take so long and exact such a toll on the drives that second drive failure starts to become a measurable risk as well.  And then there are additional risks caused by array controller errors that can utilize resilver algorithms to destroy an entire array even when no drive failure has occurred.  As RAID 0 (or RAID 1 or RAID 10) do not have resilver algorithms they do not suffer this additional risk.  These are hard risks to quantify but what is important is that they are additional risks that accumulate when using a more complex system when a simpler system, without the redundancy, was more reliable from the outset.

Now that we have established that RAID 5 can be less reliable than RAID 0 I will point out the obvious dangers of RAID 0.  RAID in general is used to mitigate the risk of a single, lone hard drive failing.  We all fear a single drive simply failing and all data being lost.  RAID 0, being a large stripe of drives without any form of redundancy, takes the risk of data loss of a single drive failing and multiplies it across a number of drives where any drive failing causes total loss of data to all drives.  So in our eleven disk example above, if any of the eleven disks fails all is lost.  It is clear to see where this is dramatically more dangerous than just using a single drive, all alone.

What I am trying to point out here is that redundancy does not mean reliability.  Just because something is redundant, like RAID 5, provides no guarantee that it will always be more reliable than something that is not redundant.

My favourite analogy here is to look at houses in a tornado.  In one scenario we build a house of brick and mortar.  On the second scenario we build two redundant house, east out of straw (our builders are pigs, apparently.)  When the tornado (or big bad wolf) comes along which is more likely to leave us with a standing house?  Clearing one brick and mortar house has some significant reliability advantages over redundant straw houses.  Redundancy didn’t matter, reliability mattered in the end.

Redundancy is often misleading because it is easy to quantify but hard to qualify.  Redundancy is a black or white question: Is it redundant?  Yes or no.  Simple.  Reliability is not so simple.  Reliability is about failure rates and likelihoods.  It is about statistics and analysis.  As it is hard to quantify reliability in a meaningful way, especially when selling a project to the business people, redundancy often becomes a simple substitute for this complex concept.

The concept of using redundancy to misdirect questions of reliability also ends up applying to subsystems in very convoluted ways.  Instead of making a “system” redundant it has become common to make a highly reliable, and low cost, subsystem redundant and treat subsystem redundancy as applying to the whole system.  The most common example of this is RAID controllers in SAN products.  Rather than having a redundant SAN (meaning two SANs) manufacturers will often make that one component not often redundant in normal servers redundant  and then calling the SAN redundant – meaning a SAN that contains redundancy, which is not at all the same thing.

A good analogy here would be to compare having redundant cars meaning two complete, working cars and having a single car with a spare water pump in the trunk in case the main one fails.  Clearly, a spare water pump is not a bad thing.  But it is also a trivial amount of protection against car failure compared to having a second car ready to go.  In one case the entire system is redundant, including the chassis.  In the other we are making just one, highly reliable component redundant inside the chassis.  It’s not even on par with having a spare tire which, at least, is a car component with a higher likelihood of failure.

Just like the myth of RAID 5 reliability and system/subsystem reliability, shared storage technologies like SANs and NAS often get treated in the same way, especially in regards to virtualization.  There is a common scenario where a virtualization project is undertaken and people instinctively panic because a single virtualization host represents a single point of failure where, if it fails, many systems will all fail at once.

Using the term “single point of failure” causes a panic feeling and is a great means of steering a conversation.  But a SPOF, as we like to call it, while something we like to remove when possible may not be the end of the world.  Think about our brick house.  It is a SPOF.  Our two houses of straw are not.  Yet a single breeze takes out our redundant solutions faster than our reliable SPOF.  Looking for SPOFs is a great way to find points of fragility in a system, but do not feel that every SPOF must be made redundant in every scenario.  Most businesses will find their best value having many SPOFs in place.  Our real goal is reliability at appropriate cost, redundancy, as we have seen, is no substitute for reliability, it is simply a tool that we can use to achieve reliability.

The theory that many people follow when virtualizing is that they take their virtualization host and say “This host is a SPOF, so I need to have two of them and use High Availability features to allow for transparent failover!”  This is spurred by the leading virtualization vendor making their money firstly by selling expensive HA add on products and secondly by being owned by a large storage vendor – so selling unnecessary or even dangerous additional shared storage is a big monetary win for them and could easily be the reason that they have championed the virtualization space from the beginning.  Redundant virtualization hosts with shared storage sounds great but can be extremely misguided for several reasons.

The first reason is that removing the initial SPOF, the virtualization host, is replaced with a new SPOF, the shared storage.  This accomplishes nothing.  Assuming that we are using comparable quality servers and shared storage all we’ve done is move where the risk is, not change how big it is.  The likelihood of the storage system failing is roughly equal to the likelihood of the original server failing.  But in addition to shuffling the SPOF around like in a shell game we’ve also done something far, far worse – we have introduced chained or cascading failure dependencies.

In our original scenario we had a single server.  If the server stayed working we are good, if it failed we were not.  Simple.  Now we have two virtualization hosts, a single storage server (SAN, NAS, whatever) and a network connecting them together.  We have already determined that the risk of the shared storage failing is approximately equal to our total system risk in the original scenario.  But now we have the additional dependencies of the network and the two front end virtualization nodes.  Each of these components is more reliable than the fragile shared storage (anything with mechanical drives is going to be fragile) but that they are lower risk is not the issue, the issue is that the risks are combinatorial.

If any of these three components (storage, network or the front end nodes) fail then everything fails.  The solution to this is to make the shared storage redundant on its own and to make the network redundant on its own.  With enough work we can overcome the fragility and risk that we introduced by adding shared storage but the shared storage on its own is not a form of risk mitigation but is a risk itself which must be mitigated.  The spiral of complexity begins and the cost associated with bringing this new system up on par with the reliability of the original, single server system can be astronomic.

Now that we have all of this redundancy we have one more risk to worry about.  Managing all of this redundancy, all of these moving parts, requires a lot more knowledge, skill and preparation than does managing a simple, single server.  We have moved from a simple solution to a very complex one.  In my own anecdotal experience the real dangers of solutions like this come not from the hardware failing but from human error.  Not only has little been done to avoid human error causing this new system to fail but we’ve added countless points where a human might accidentally bring the entire system, redundancy and all, right down.  I’ve seen it first hand; I’ve heard the horror stories.  The more complex the system the more likely a human is going to accidentally break everything.

It is critical that as IT professionals that we step back and look at complete systems and consider reliability and risk and think of redundancy simply as a tool to use in the pursuit of reliability.  Redundancy itself is not a panacea.  Neither is simplicity.  Reliability is a complex problem to tackle.  Avoiding simplistic replacements is an important first step in moving from covering up reliability issues to facing and solving them.

 

Share

11 thoughts on “When No Redundancy Is More Reliable – The Myth of Redundancy

  1. SAM, I enjoyed this article quite a lot. Good after readying the OpenStorage article.
    I have been gathering, readying a lot of information, and in the end (a couple of weeks ago) I finally decided that the best route was to use internal datastore in our server because of higher I/O available, and more reliable drives than our NAS, but replicate data to NAS, in case that there is a problem with the local store. Since I already have to VM Host (to separate resources consumption) this seemed more logical.
    The weird thing, is that I did not see much about doing this kind of deployment, and I talked with a couple of people that put a lot of emphasis into Vmotion and HA, however, they do not have large budgets either. I was really feeling lonely :)
    However, the challenge is still there to have this replication.
    I found this software
    http://www.stormagic.com/SvSAN.php
    That can present local datastore as a SAN (using iSCSI), but I am still wondering if there are other options out there

  2. Fantastic write up. I saw first hand the exact scenario you laid out in the last part of your article, so I have seen the danger up jumping on the “Lets get a SAN” bandwagon.

  3. Nice write up. When using than 6 or more drives, I have never considered a RAID5 as a ‘safe’ method of redundancy. How does using a RAID6 compare against the shortfalls you covered of RAID5 rebuilding a degraded array?

  4. There are several problems with the indictment of RAID5.

    First, the problems identified are more related to giant drives and RAID volumes that span a number of them than specifically to RAID5. The same URE rates apply to large volumes regardless of underlying RAID. The risks outlined are mitigated by using giant drives in sets rather than all in one giant RAID. For example, the 12-drive RAID5 array of giant disks should be set up as three RAID5 arrays, each smaller and less risky to resilver. There is a capacity loss versus using one giant RAID5, but as you have pointed out, giant RAID volumes are not viable anywa.

    Second, the brick house/straw house analogy is flawed. It is not a hurricane that takes out the first member of a redundant pair. Unlike a hurricane, the root cause of the system (brick house) that fails is not so likely to take out the straw house as a “hurricane.” This makes the redundant second system, even built of straw, better than indicated in the article.

    Third, the car analogy is problematic. Systems with redundancy built in is not like carrying a spare in the truck, like a tire or a water pump. A fault tolerant car with redundant systems has two working water pumps installed and can operate when one dies. It has more than four tires, such that one or more can go flat and the vehicle still operate. Same with battery, a second one that keeps the car working when one fails.

    Finally, the article says that redundancy is itself not a fix for risk, but then says that the fix is redundancy in all components (server, san, AND network). It is clear that redundancy is the best mitigation for the risk of failure.

    In summary, the take-ways from this article:
    1. do not use giant RAID volumes
    2. when building redundant systems, make sure to include components that are often over-looked, like network.

  5. A great write-up, really enjoy reading it.

    James,

    With a 4-drive RAID 5 you still get a very large volume. Even a 3TB array gives you 25% possibility of encountering a URE during the rebuild process. I certainly won’t risk it, not in business world.

  6. I’m surprised how many people here are still talking about using hardware Raid for critical data. To be very frank, the idea of using ANY hardware raid configuration, (including the popular 1 & 5/6 configuration) is bordering on antiquarian. It may be time to re-evaluate. You may be “renting DVD’s” when the rest of world is streaming videos… On the beach… with an iPad-Super-Ultra-Air².

    If your data is important to you, you should be storing it in a ZFS array. If your data matters, and you don’t think you need ZFS, you simply haven’t thought about it long enough. We’ve grown accustomed to our processors and memory evolving like crazy. Now it’s time for the storage paradigm to do the same.

    With ZFS:
    -Problems like “Bit rot”, write holes, data corruption… all disappear.
    -You can create historic point-in-time snapshots (that can be re-mounted ‘read/write’ in parallel with your current data. Did a virus just destroy all of your data. No problem. Just go back in time to a few moments before the event.
    -Checksums are so ample that the chance of undetected data corruption is effectively zero, even if your dataset has more bits than there are atoms in the universe.
    -Monthly scrubs run on schedule to true-up data to its checksums. If data has changed due to bit rot, it can be corrected by a redundant copy.
    -Copy on write architecture ensures that all changes to your data are written and validated before the previous version is ‘decoupled’ from the file name.
    -Did the power go out right in the middle of writing to a critical file? No problem. The file is not corrupted because the previous state is still intact.
    -Abstraction of physical storage pools to the logical datasets contained within, so different data can have appropriate snapshotting, quota, compression, and replication policies applied.
    -You can build as much redundancy as you like. You can create as many mirrored copies as you want (to use the author’s ‘water pump’ example), but better still, you can also replicate the entire file system to other instances on or off site, through a TCP socket or SSH tunnel. Following copies receive changed data blocks, not whole files, so data transfer is manageable. If you’re ultra-paranoid like me, you can use good old-fashioned RSync to another following server for mega-ultra redundancy by way of replicating with a fully dissimilar technology.

    Oh yeah, did I mention this is all free? Sorry. I meant to.
    Granted, ZFS on Solaris (or Solaris forks), on *BSD can be a bit intimidating, but when ZFS is packaged in GUI-driven Open-source distributions like FreeNAS, or paid, supported ZFS products (like NexentaStore, and TrueNAS), it becomes something that ‘normal’ admins (like me) can incorporate into enterprise (and personal) workflows. Even my personal windows workstation libraries replicate to ZFS stores which in turn replicate offsite within minutes. House burned down? Oh well.

    EVEN IF you ignore the mind-bending read performance that can be achieved with ZFS through large read caches (DRAM, and L2 Adaptive Read Cache), and EVEN IF you ignore the insane synchronous write performance when paired with a high performance write cache device; the data integrity achievable with ZFS instance is unmatched. You owe it to yourself, to your career, perhaps to your employer to explore this technology.

    Just sayin…

    https://twitter.com/karlfife/status/429417133290688513

Comments are closed.