A cloud is a special type of virtualization platform and fills a unique niche.  Cloud computing takes traditional virtualization and layers it with automated scaling and provisioning that allows for rapid, horizontal scaling of applications.  This is not a normal business need.  Cloud also lends itself, and is often tied to, self-service of resource provisioning but this alone does not make something a cloud nor justify the move to a cloud platform, but could be an added incentive.  What makes cloud interesting is the ability to provide self-service portals to end users and the ability for applications to self-provision themselves.  These are the critical aspects that set a cloud platform apart from traditional virtualization.

What a cloud does not imply are features such as simplified whole-domain system management from a single pane of glass, large scale consolidation, easy migration between hardware systems, rapid provisioning of new systems, virtualization, high availability, resource over-commitment, etc.  These features are all available in other ways, primarily through or on top of standard platform virtualization (VMware vSphere, Microsoft’s HyperV, Xen, et. al.)  It is not that these features cannot be made available in a private cloud, but the features are not aspects of the cloud but rather of the underlying virtualization platform.  The cloud layer is above these and simply passes through the benefits of the underlying layers.

Often cloud is approached because of a misunderstanding that many of the features commonly associated with private clouds are not available in some other, simpler form.  This is rarely the case.  Normal virtualization platforms, most commonly VMware’s vSphere and Microsoft’s HyperV, offer all of these options.  They can be used to make robust clusters of physical servers, managed from a single interface, with incredibly high reliability and rapid provisioning of new systems that require minimal specialty knowledge from the IT department and maintain traditional business workflows.  Most times, when I am speaking with businesses that believe that they may be interested in pursuing the ownership of their own cloud, the features that they really want are not cloud features at all.

The term “cloud” has simply become so popular recently that people begin to assume that important features for nearly everyone must be attributed to it to explain the sudden surge in importance, but this is simply not the case.  Cloud remains, and will remain, a predominantly niche solution appropriate for only a very small number of companies to own themselves.  The use of public clouds or the use of hosted services delivered from cloud platforms will become, and indeed has already become, nearly ubiquitous   But ownership of a private cloud for the use of a single company is a long way from being a business need for most businesses or business units and in many cases, I suspect, never will become so.

Private clouds shine in two key areas.  The first is a business who needs a large number of temporary or ad hoc systems “spun up” on a regular basis.  This often occurs with large development teams and application testing groups, especially if these groups target multiple operating systems.  The ability to rapidly provision temporary testing systems or lab systems can be very advantageous and the nature of cloud computing to easily expose provisioning tools that allow business customers to create, manage and destroy their own system instances with, we would expect, built-in charge back mechanisms can be very beneficial to corporate efficiency as the interaction between the IT department and the end users becomes nearly frictionless for this transaction.  Responsibility for maintaining the cloud as a whole can easily be segregated from the responsibilities of maintaining individual systems.  Seldom used in this manner for production workloads, this allows a self-service approach that many business units desperately seek today.  Impractical on a small scale due to the overhead of creating and maintaining the cloud platform itself but on a large scale can be hugely productive.  In addition to technical advantages, this aspect of cloud computing can serve as a model for thinking of IT as an internal service provider and departments as customers.  We have long discussed IT and other business units in these terms but we rarely truly think of them in this way.

The second area where cloud computing really comes into its own and the one for which the concept was developed originally is to handle auto provisioning for horizontally scaling applications.  That is, application workloads that are able to increase in their capacity handling ability by spawning new instances for themselves.  On a small scale, many web applications, due to their stateless nature, do this within a single system by spawning new thread workers to handle additional connections.  An Apache web server might start with eight listeners ready to service requests but as those threads become exhausted it automatically starts new threads to handle additional incoming connections so that it is able to scale within the confines of a single server.  To expand on this concept, applied to cloud computing, that same application sensing thread exhaustion approaching on a system-wide level (or based on other metrics such as a lack of free memory or a loss of performance) would use an API exposed from the cloud computing platform to signal the cloud management system to provision a new copy of the system that was calling it – essentially cloning itself on the fly.  In a matter of seconds, a new virtual server, identical to the first, would be up and running and joining its parent in servicing incoming requests.  This child or clone system would likewise spawn new threads internally, as needed, and then if it too sensed exhaustion would call the cloud platform to create yet another new system to handle even more threads.  In this way the application can grow itself almost infinitely (within the hardware limits of the entire cloud platform) as needed, on the fly, automatically.  Then, as individual systems become idle, workloads die down, one at a time a system can signal that it is no longer needed to the cloud management system and the system will be powered off and destroyed as it was simply a stateless clone, freeing system capacity for other applications and workloads that may need to take advantage of the spare capacity.

As we can see, cloud computing is massively powerful, especially with the bulk of today’s public and enterprise applications being written in a stateless manner in order to take advantage of web protocols and end user interfaces. Web applications are especially adept at leveraging cloud computing’s scalability model and most large scale web applications leverage this elastic expanding and contracting of capacity today.  Many new NoSQL models are beginning to emerge that signal that databases, in addition to application front end processing nodes, may soon benefit from similar models on a large scale.  This can certainly be leveraged for internal applications as well as publicly facing ones, however internal applications rarely need to scale beyond a single system and so it is quite rare to find private clouds being leveraged in quite this way.

The dangers around cloud computing come in the form of additional complexity above and beyond normal virtualization.  There is the potential for complex storage needed to support the platform and more layers to learn and maintain.  Cloud computing’s ability to rapidly create and destroy systems can make it tempting for users to attempt to use cloud resources as if they were persistent systems, which they can be made to be, which can result in data loss from users receiving behavior very different from what is traditional and expected.  Possibly the biggest cloud concern is a human one and that is the increased likelihood of experiencing uncontrolled system sprawl as end users wildly spin up more and more new systems which, as they are created by end users and not IT, are probably not tightly controlled and monitored leaving systems in a rogue, and oft forgotten state.  This can lead to a maintenance and security nightmare as systems go unpatched and uncared for increasing risk and draining resources.  And most worrisome is the possibility that systems will be created and forgotten and potentially exist without proper licensing.  Tracking and reporting on auto provisioned systems carries process risk caused by the huge shift in how systems are created.  IT departments are accustomed to the heavy licensing processes necessary to maintain compliance but with cloud computing there is a potential for this process to be exposed to the business units in a way for which they are not at all equipped to handle.  There are accommodations for the licensing needs of cloud computing, but this is extra complexity and management that must be addressed.  Allowing systems to exist without direct IT department oversight clearly carries risk of a potentially unforeseen nature.

Private cloud ownership brings many exciting possibilities, but it is clear that these benefits and opportunities are not for everyone.  They cater to larger businesses, to those with good process control, to companies running especially adapted applications that are capable of taking advantage of the system-level elasticity of the resources and those needing large scale ad hoc system creation and destruction provided, as a service, for end users to self-provision.  Most large enterprises will find limited use for cloud computing in house.  Smaller organizations will rarely find cloud computing to be advantageous in the near future, if ever.

I believe that this behaviour stems from a few things:

1. People tend to work in IT because they are “smarter” or at least “more interested” about most things than average people so we tend to carry a lot of general knowledge that allows us to act as a competent member of any department (IT can do HR’s job in a pinch, is the reverse commonly true?)
2. IT tends to get thrown whatever work other departments don’t want to do and can get away with handing off (can you print this for us? can you fix my microwave? the fuse has blown!  have you any experience with sprinklers?) So we get into this mindset from other departments’ behaviors towards us.
3. We have a broad view into the organization as a whole, moreso than almost any other department.
4. We tend to be passionate about doing things the “right way” – which is often based on technical excellence or industry common practice but may not account for the specific business needs nor unique factors.

Put together, these, and other, factors make us tend to want to get involved in anything and everything in and around the businesses which we serve. Questions around involvement in other departments’ activities come up regularly. To establish just how skewed our thinking about this behavior tends to be – we see IT people asking IT people what their responsibility is rather than talking to their own business’ management who are the ones actually making that decision. This isn’t about best practice, it is about following your own company’s rules.

Some examples of places where IT people like to jump in and try to be other departments:

• “People are surfing Facebook at work, I have to stop them.” – Do you? Is this a business decision or is IT just making HR or security decisions for those departments? IT bringing this up as a topic is great, but feeling a need to enforce personal work habit decisions should probably be left to the business owner, manager or designated department like HR, legal or security.
• Spying on end users, capturing passwords, etc. – Did the legal department ask you to do this? If not, don’t take on legal and security responsibilities, especially ones that might carry fines or even  jail time in your local jurisdiction!  We risk turning the tables from suspecting someone else to being the culprits ourselves.
• Pressuring the business about fire hazards, safety issues (that are not your own), etc. – See something, say something. Awesome. Don’t be the cause of bad behaviour yourself. But if the business isn’t concerned about these things once reported, unless it is a legal issue that you need to turn over to the police, don’t feel that this is IT’s job. The janitor doesn’t feel this way, HR doesn’t feel this way, IT shouldn’t either. If the business decides to not care, you shouldn’t either. (Example was AJ talking about stringing surge protectors together.)
• The business can’t be down! – IT loves this one. This might be us pushing for high availability clusters or just overbuilt servers or who knows what. The reality is, this is 100% a financial decision that the accounting, financial and CFO teams should be making. IT has no idea how much the business can be or can’t be down – we just know how much it costs to mitigate how much risk. We feed data to the financial people who come back with the risk/reward evaluation. IT shouldn’t be making financial decisions on any scale.

I could go on and on. HR, finance, security, facilities management, legal – we want to get involved in all of these job roles. But is it our responsibility to do so? Maybe it is in your case, but normally, it is not. We take on personal and professional risk in order to push our ideas and opinions on businesses that often aren’t interested in our input (in those areas.)

Step back and look at your relationship to the business. Are you making suggestions and decisions that line up with your role within the business and with the business’ unique needs? Keep perspective. It is so easy to get caught up in IT doing things the “right” way that we forget that the business might not share our opinions of what is right and wrong for them – and we aren’t in IT just for the sake of being in IT, but for the purpose of supporting the business.

[Reprinted from a post in Spiceworks, January 8, 2013]

That doesn’t mean that Windows 8 should be your main desktop later this afternoon, but considering a move to Windows 8 is important to do early.  It is a popular approach to hold off on new software updates until systems have been in production use for months or years and there is value to this concept – allowing others to vet, test and uncover issues while you sit back and remain stable on existing, well known software.  But there is a reason why so many businesses forge ahead and that is because using software early delivers the latest features and advantages as early as possible.

Unlike software coming from a small company with limited support and testing resources, Microsoft’s software is incredibly well tested both internally and by the community before it is available to end users.  Few software is more heavily vetted prior to release.  That doesn’t mean that release day rollouts are wise but beginning to evaluate new products early can have major advantages both because of the newest features are available to those that decide to use the new product but also the most time to find an alternative for those decide to migrate away.  Early decision making is important to success.

The reality is, that while many businesses should take the time to evaluate Windows 8 versus alternative solutions – a practice that should be done regularly regardless of new features or changes to environments to ensure that traditional choices remain the best current choices, nearly all businesses today will be migrating to Windows 8 and remaining in the Microsoft ecosystem for quite some time to come.

This means that many companies should be looking to make the jump to Windows 8 sooner, rather than later.  Windows 8, while seemingly shockingly new and innovative, is based on the same Windows NT 6 family kernel that began with Windows Vista and Windows Server 2008 and continued through the Windows 7 and Windows Server 2008 R2 era and is shared with Windows Server 2012.  This kernel is mature and robust and the vast majority of the code and features in Windows 8, user interface aside, are well tested and extremely stable.  Windows 8 uses fewer resources, on the same hardware, as Windows 7 which, in turn, was lighter and faster than Windows Vista.  The sooner that you move to Windows 8, the sooner you get more performance out of your existing hardware and the longer you have to leverage that advantage.

Windows 8 brings some big changes that will impact the end users, without a doubt.  These changes can be, in some cases, quite disruptive but with proper training and preparation users should return to regular productivity levels in a reasonable amount of time and often will be more productive once they are comfortable with the new environment and features.  Those that do not fall into one of these two categories are the smaller, niche user group that are prime candidates for moving to a completely different ecosystem where their needs can be more easily met.

If you are an organization destined to be running Windows 8, or its successors, “someday” then most likely you should be running Windows 8 today to start leveraging its advantages as soon as possible so that you can use them as long as possible.  If Windows truly is the platform that is best for you you should embrace it and accept the “hit” of transitioning to Windows 8 now, swallow that bitter pill and be done with it, and for the next several years while your competitors are whining about having to move to Windows 8 “someday” you will be happily leveraging your older hardware, your more efficient workflows and your more modern systems day after day, reaping the benefits of an early migration to a stable platform.

It is common for IT departments to take a “wait and see” approach to new system migrations.  I am convinced that this is created by a culture of hoping that IT staff will leave their current positions before a migration occurs and that they will land a new position elsewhere where they have already migrated.  Or perhaps they hope to avoid the migration completely awaiting a later version of Windows.  This second argument does carry some weight as many shops skip operating system revisions but doing so often brings extra overhead in security issues, application compatibility effort and other issues.

Windows 8 is unique in that it is a third release of the Windows NT 6 kernel series so it comes as a rare, very stable late release member of its family (the NT 6 family is sometimes called the “Vista Family.”)  Windows 8′s NT designation is 6.2.  The only other Microsoft NT operating system to reach the x.2 status was when Windows XP SP3 and Server 2003 R2 released with the NT 5.2 kernel – a part of the Windows 2000 family.  Late release kernels are important because they tend to deliver the utmost in reliability and represent an excellent point in which to invest in a very long term deployment strategy that can last for nearly a decade.

Whether you agree with Microsoft’s unified platform vision or the radical approach to user interface included in Windows 8, you need to decide if you are continuing down the path of the Microsoft platform and if so, embrace it rather than fight it and begin evaluating if a move to Windows 8 and, by extension, Windows Server 2012 are right for you.  Don’t avoid Windows 8, it isn’t going to go away.  For most shops making the decision to move today will sow the seeds for long term benefits that you can reap for years and years to come.

What I have come to realize over the last few years is that a large chunk of the communal wisdom of the industry appears to have been adopted during these formative and influential years of the IT profession and have since then passed into myth.  Much like the teachings of Aristotle who went for millennia considered to be the greatest thinker of all time and not to be questioned – stymieing scientific thought and providing a cornerstone for the dark ages.  A foundation of “rules of thumb” used in IT have passed from mentor to intern, from professor to student, from author to reader over the past fifteen or twenty years, many of them being learned by rote and treated as infallible truths of computing without any thought going into the reasoning and logic behind the initial decisions.  In many cases so much time has come and gone that the factors behind the original decisions are lost or misunderstood as those hoping to understand them today lack firsthand knowledge of computing from that era.

The codification of IT in the late nineties happened on an unprecedented scale driven primarily by Microsoft sudden lurching from lowly desktop maker to server and LAN ecosystem powerhouse.  When Microsoft made this leap with Windows NT 4 they reinvented the industry, a changing of the guard, with an entirely new generation of SMB IT Pros being born and coming into the industry right as this shift occurred.  This was the years leading up to the Y2K bubble with the IT industry swelling its ranks as rapidly as it could find moderately skilled computer-interested bodies.  This meant that everything had to be scripted (steps written on paper, that is) and best practices had to be codified to allow those with less technical backgrounds and training to work.  A perfect environment for Microsoft and their “never before seen” level of friendliness NT server product.  All at once the industry was full of newcomers without historical perspective, without the training and experience and with easy to use servers with graphical interfaces making them accessible to anyone.

Microsoft lept at the opportunity and created a tidal wave of documentation, best practices and procedures to allow anyone to get basic systems up and running quickly, easily and, more or less, reliably.  To do this they needed broad guidelines that were applicable in nearly all common scenarios, they needed it written in clear published form and they needed to guarantee that the knowledge was being assimilated.  Microsoft Press stepped in with the official publications of the Microsoft guidelines and right on its heels Microsoft MCSE program came into the spotlight totally changing the next decade of the profession.  There had been other industry certifications before the MCSE but the Windows NT 4 era and the MCP / MCSE certification systems were the game changing events of the era.  Soon everyone was getting boot camped through certification quickly memorizing Microsoft best practices and recommendations, learning them by rote and getting certified.

In the short term, the move did wonders for providing Microsoft an army of minimally skilled, but skilled nonetheless, supporters who had their own academic interests aligned with Microsoft’s corporate interest forming a symbiotic relationship that completely defined the era.  Microsoft was popular because nearly every IT professional was trained on it and nearly every IT professional encourage the adoption of Microsoft technologies because they had been trained and certified on it.

The rote guidelines of the era touched many aspects of computing, many are probably still unidentified to this day so strong was the pressure that Microsoft (and others) put on the industry at the time.  Most of today’s concepts of storage and disk arrays, filesystems, system security, networking, system architecture, application design, memory, swap space tuning and countless others all arose during this era and passed, rather quickly, into lore.  At the time we were aware that these were simply rules of thumb, subject to change just as they always had based on the changed in the industry.  Microsoft, and others, tried hard to make it clear what underlying principles created the rules of thumb.  It was not their intention to create a generation having learned by rote, but it happened.

That generation went on to be the effective founding fathers of modern LAN management.  In the small and medium business space the late 1990s represented the end of the central computer and remote terminals design, the Internet became ubiquitous (providing the underpinnings for the extensive propagation of the guidelines of the day), Microsoft washed away the memory of Novell and LANtastic, Ethernet over twisted pair completely abolished all competing technologies in LAN networking, TCP/IP beat out all layer three networking competitors and more.  Intel’s IA32 processor architecture began to steal the thunder from the big RISC processors of the previous era or the obscure sixteen and thirty two bit processors attempting to unseat Intel for generations.  The era was defining to a degree few who come since will ever understand.  Dial up networking gave way to always-on connections.  Disparate networks that could not communicate with each other lost to the Internet and a single, global networking standard.  Vampire taps and hermaphrodite connectors gave in as RJ45 connectors took to the field.  The LAN of 1992 looked nothing like the LAN of 1995.  But today, what we use, while faster and better polished, is effectively identical to the computing landscape as it was by around 1996.

All of this momentum, whether intentional or accidental, created an unstoppable force of myth driving the industry.  Careers were built on this industry wisdom taught around the campfire at night.  One generation clinging to their established beliefs, no longer knowing why they trusted those guidelines or if they applied, and another being taught them with little way to know that they were being taught distilled rules of thumb meant to be taught with coinciding background knowledge and understanding and having been designed not only for a very specific era, roughly the band from 1996 to 1999, but also, in a great many cases, for very specific implementations or products, generally Windows 95 and Windows NT 4 desktops and Windows NT 4 servers.

Today this knowledge is everywhere.  Ask enough questions and even young professionals still at university or doing a first internship are likely to have heard at least a few of the more common nuggets of conventional IT industry wisdom.  Sometimes the recommendations, applied to day, are nearly benign representing little more than inefficiency or performance waste.  In other cases they may represent pretty extreme degrees of bad practice today carrying significant risk.

It will be interesting to see just how long the late 1990s continue to so vastly influence our industry today.  Will the next generation of IT professionals finally issue a broad call to deep understanding and question the rote learning of the past eras?  Will misunderstood recommendations still be commonplace in the 2020s?  At the current pace of change, it seems unlikely that any significant change to the thinking of the industry is likely to change too much prior to 2030.  IT has been attempting to move from its wild west, everyone distilling raw knowledge into practical terms on their own to large scale codification like other, similar, fields like civil or electrical engineering, but the rate of change, while tremendously slowed since the rampant pace of the 70s and 80s, still remains so high that the knowledge of one generation is nearly useless to the next and only broad patterns, approaches and thought processes have great value to be taught mentor to student.  We may easily face another twenty years of the wild west before things begin to really settle down.

With rare exception, IT departments have needs that are different from normal departments – based primarily on what business demand from them: high reliability, continuous availability, deep business knowledge of all departments, ability to train others, knowledge of broad and disparate technologies, business skills, financial skills, procurement skills, travel, experience across technologies and industries, efficiency and experience on the latest technologies, trends, architectures, techniques and knowledge of the latest threats and products arriving daily – and to not only use all of that skill and experience to provide support roles but to also be a productive engineer, customer service representative and to present and defend recommendations to management that often pushes back or provides erratic or emotional support of infrastructural needs.  Quite literally, no single person can possibly fill those shoes and one that could would demand a salary higher than the revenue of most small businesses.

How do larger businesses handle this daunting task?  They do so with large IT departments filled with people who specialize in specific tasks, generalists who glue specialists together, dedicated support people who don’t need to do engineering, engineers who don’t get support interruptions, tiered support roles to filter tasks by difficulty, mentors to train newcomers, career pipelines, on call schedules or follow the sun support desks and internal education systems.  The number of challenges presented to a lone IT professional or very small IT department are nearly insurmountable forcing corners to be cut nearly everywhere, often dangerously.  There is no time or resources for tiny IT departments to handle the scope of the job thrown at them.  Even if the job role is whittled down to a very specific job role, SMB IT professionals are often faced with decision making for which they cannot be prepared.  For example, a simple server failure might be seen as just another “hardware upgrade” task because the overworked, under-scoped IT professional isn’t being given the necessary latitude to be able to flag management as to an arising opportunity for some strategic roadmap execution – maybe a complete departure from previous plans due to a late breaking technology change, or a chance to consolidate systems for cost savings or a tactical upgrade or change of platform might deliver unrealized features.

Having worked both in the trenches and in management I believe that there are two thresholds that need to be considered.  One is the minimum functional IT department size.  That is, the minimal size that an internal IT department can be to be able to complete basic job functions using internal staff.  To clarify, “internal staff” can be a rather ambiguous term.  Internal here I use to mean dedicated or effectively dedicated staff.  These people can be employees or contractors.  But at a minimum, with the exception of very rare companies that don’t operate during full business hours or other niche scenario, it takes at least three IT professionals on an IT team to functionally operate as an IT department.

With three people there is an opportunity for peer review, very critical in a technical field that is complex at the best of times and a swirling quagmire of unknown requirements, continuous change and insurmountable complexity at the worst of times.  Like any technical field, IT professionals need peers to talk to, to oversee their work, to check their ideas against and to keep them from entering the SMB IT Bubble.  Three is an important number.  Two people will have a natural tendency to become adversarial with one carrying the weight of recommendation to management and one living in their shadow – typically with the one with the greater soft skills or business skills gaining the ear of management while the one with the greater technical acumen losing their voice if management isn’t careful to intentionally include them.  As with maritime chronometers, it is critical that you have three because you can have a quorum.  Two simply have an argument.

IT is an “around the clock” endeavor.  During the day there are continuous needs from IT end users and the continuous potential for an outage or other disaster plus meetings, design sessions, planning and documentation.  In the evenings and on weekends there is all of the system maintenance that cannot, or at least should not, be done while the business is up and running.  This is often an extensive level of work, not an occasional bit of missing happy hour but regular workload eliminating dinner and family time.  Then comes the emergency calls and outages that happen any time day or night.  And there is the watching of email – even if nothing is wrong it is commonplace for IT to be involved in company business twelve to sixteen hours a day and weekends too, even in very small companies.  Even the most dedicated IT professional will face rapid burnout in an environment such as this without the ability to have a service rotation to facilitate necessary rest and work/life balance.

This comes before the considerations for the unforeseeable sick days, emergency leave or even just holidays or vacation.  If there are not enough people left behind to cover the business as usual tasks plus the unforeseeables, then vacations or even sick days become nearly, if not totally, impossible.  Skipping vacations for a year or two is theoretically possible but it is not healthy and doesn’t provide for a sustainable department.

Then there is training and education.  IT is a demanding field.  Running your own IT department suggests a desire to control the level of skill and availability granted to the company.  To maintain truly useful IT staff time and resources for continuous education is critical.  IT pros at any stage in their career need to have time to engage in discussions and forums, attend classes and training, participate in user groups, go to conferences and even just sit down and read books and web sites on the latest products, techniques and technologies.  If an IT professional is not given the chance to not just maintain, but grow their skills they will stagnate and gradually become useless technically and likely to fall into depression.  A one or two man shop, with even the smallest of organizations, cannot support the necessary free time for serious educational opportunities.

Lastly, and far more critical than it seems at first, is the need to handle request queues.  If issues arise within a business at a continuous, average rate of just enough per day to require eight hours per day to service them it may seem like only one person would be necessary to handle the queue that this work load would generate.  In an ideal world, perhaps that is true.  In the real world, requests come in at varying degrees of priority and often at very inopportune moments so that even a business that has taken on the expense of having dedicated, internal IT cannot have the “instant response time” that they often hope for because their IT professional is busy on an existing task.  The idea of instant response is based on the assumption that the IT resource is sitting idle and watching the ticket queue or waiting by the phone at all times.  That is not realistic.

In large enterprises, to handle the response time concerns of critical environments, surplus IT resources are maintained so that only in the direst of emergencies would all of them be called upon at one time to deal with high criticality issues at the same time.  There is always someone left behind to deal with another pressing issue should one arise.  This not only allows for low latency response to any important customer need but also provides spare time for projects, learning and the necessary mental downtime needed for abstract processing of troubleshooting without which IT professionals in a support role will lose efficiency even if other work does not force them to multitask.

In small shops there is little to be done.  There is a lack of scale to allow for the excess IT resource capacity to be sitting n the wings just waiting for issues to arise.  Having three people is, in my opinion, an absolute minimum to allow for the handling of most cases of this nature if the business is small enough.  By having three people there is, we hope, some chance of avoiding continuous re-prioritization of requests, inefficient multi-tasking and context switching.

In larger organizations there is also a separation of duties between administration or support job roles and engineering job roles.  One job is event driven, sitting “idle” waiting for a customer request and then reacting as quickly as possible. The other focused on projects and working towards overall efficiency.  Two very different aspects of IT that are nearly impossible for a single person to tackle simultaneously.  With a three person shop these roles can exist in many cases even if the roles are temporarily assigned as needed and not permanent aspects of title or function.

With only three people an IT department still lacks the size and scale necessary to provide a healthy, professional growth and training environment internally.  There are not enough rungs on the ladder for IT employees to move up and only turnover, unlikely to happen in the top slot, allows for any upward mobility forcing good candidates to leave rapidly for the sake of their careers leaving good shops with continuous turnover and training and lesser shops with dramatically inferior staff.  There is no simple solution for small organizations.  IT is a broad field with a great many steps on the ladder from helpdesk to CIO.  Top IT organizations have thousands or, in the most extreme cases, hundreds of thousands of IT professionals in a single organization.  These environments naturally have a great degree or both upward and lateral mobility, peer interaction and review, vendor resources, mentoring, lead oversight, career guidance and development and opportunities to explore new ideas and paths often that don’t exist in SMBs of any size.

To maintain a truly healthy IT department takes a much larger pool of resources.  Likely one hundred, or more, IT professionals would be required to provide adequate internal peerage, growth and opportunity to begin to provide for career needs, rather than “job needs.”  Realistically, the SMB market cannot bear this at an individual business scale and must accept that the nature of SMB IT is to have high turnover of the best resources and to work with other businesses, typically ones that are not directly competitive, to share or exchange resources.  In the enterprise space, even in the largest businesses, this is often very common – friendly exchanges of IT staff to allow for career advancement often with no penalties for returning later in their career for different positions at the original company.

Given this bleak picture of SMB IT staff scaling needs, what is the answer?  The reality is is that there is no easy one.  SMB IT sits at a serious disadvantage to its enterprise counterparts and at some scale, especially falling below three dedicated IT staff members, the scale becomes too low to allow for a sustainable work environment in all but the most extreme cases.

In smaller organizations, one answer is turning to consulting, outsourcing and/or managed service providers who are willing and able to work either in the role of internal staff or as a hybrid with existing internal staff to provide for an effectively larger IT organization shared between many businesses.   Another is simply investing more heavily in IT resources or using other departments as part time IT to handle helpdesk or other high demand roles, but this tends to be very ineffective as IT duties tend to overwhelm any other job role.  A more theoretical approach is to form a partnership with another one or two businesses to share in house IT in a closed environment.  This last approach is very difficult and problematic and generally works only when technology is heavily shared as is geographic location between the businesses in question.

More important than providing a simple answer is the realization that IT professionals need a team on which to work in order to thrive and will perform far better on a healthy team than they will alone.  How this is accomplished depends on the unique needs of any given business.  But the efficacy and viability of the one or two “man” IT shop, for even the smallest businesses, is questionable.  Some businesses are lucky enough to find themselves in a situation where this can work for a few years but often live day to day at a high degree of risk and almost always face high turnover with their entire IT department, a key underpinning of the workings of their entire business, leaving at once with the benefits of staggered turnover that a three person and larger shop at least have an opportunity to provide.  With a single person shop there is no handover of knowledge from predecessors, no training and often no opportunity to seek an adequate replacement before the original IT professional is gone leaving at best an abrupt handover and at worst a long period of time with no IT support at all and no in house skills necessary to interview and locate a successor.

Our first task is to strip away the marketing terms and move on to technical ones.  NAS stands for Network Attached Storage but doesn’t mean exactly that and SAN stands for Storage Area Network but is generally used to refer to a SAN device, not the network itself.  In its most proper form, a SAN is any network dedicated to storage traffic, but in the real world, that’s not how it is normally used.  In this case we are hear to talk about NAS and SAN devices and how they compare so we will not use the definition that includes the network rather than the device.  In reality, both NAS and SAN are marketing terms and are a bit soft around the edges because of it.  They are precise enough to use in a normal technical conversation, as along as all parties know what they mean, but when discussing their meaning we should strip away the cool-sounding names and stick to the most technical descriptions.  Both terms, when used via marketing, are used to imply that they are a certain technology that has been “appliancized” which makes the use of the terms unnecessarily complicated but no more useful.

So our first task is to define what these two names mean in a device context.  Both devices are storage servers, plain and simple, just two different ways of exposing that storage to the outside world.

The simpler of the two is the SAN which is properly a block storage device.  Any device that exposes its storage externally as a block device falls into this category and can be used interchangeably based on how it is used.  The block storage devices are external hard drives, DAS (Direct Attach Storage) and SAN.  All of these are actually the same thing.  We call it an external hard drive when we attach it to a desktop.  We call it a DAS when we attach it to a server.  We call it a SAN when we add some form of networking, generally a switch, between the device and the final device that is consuming the storage.  There is no technological difference between these devices.  A traditional SAN can be directly attached to a desktop and used like an external hard drive.  An external hard drive can be hooked to a switch and used by multiple devices on a network.  The interface between the storage device and the system using it is the block.  Common protocols for block storage include iSCSI, Fibre Channel, SAS, eSATA, USB, Thunderbolt, IEEE1394 (aka Firewire), Fibre Channel over Ethernet (FCoE) and ATA over Ethernet (AoE.)  A device attaching to a block storage device will always see the storage presented as a disk drive, nothing more.

A NAS, also known as a “filer”, is a file storage device.  This means that it exposes its storage as a network filesystem.  So any device attaching to this storage does not see a disk drive but instead sees a mountable filesystem.  When a NAS is not packaged as an appliance, we simply call it a file server and nearly all computing devices from desktops to servers have some degree of this functionality included in them.  Common protocols for file storage devices include NFS, SMB / CIFS and AFP.  There are many others, however, and technically there are special case file storage protocols such as FTP and HTTP that should qualify as well.  As an extreme example, a traditional web server is a very specialized form of file storage device.

What separates block storage and file storage devices is the type of interface that they present to the outside world, or to think of it another way, where the division between server device and client device happens within the storage stack.

It has become extremely common today for storage devices to include both block storage and file storage from the same device.  Systems that do this are called unified storage.  With unified storage, whether you can say that it is behaving as block storage or file storage device (SAN or NAS in the common parlance) or both is based upon the behavior that you configure for the device not based on what you purchase.  This is important as it drives home the point that this is purely a protocol or interface distinction, not one of size, capability, reliability, performance, features, etc.

Both types of devices have the option, but not the requirement, of providing extended features beneath the “demarcation point” at which they hand off the storage to the outside.  Both may, or may not, provide RAID, logical volume management, monitoring, etc.  File storage (NAS) may also provide file system features such as Windows NTFS ACLs.

The key advantage to block storage is that the systems that attach to it are given an opportunity to manipulate the storage system as if it were a traditional disk drive.  This means that RAID and logical volume management, which may already have been doing in the “black box” of the storage device can now be done again, if desired, at a higher level.  The client devices are not aware what kind of device they are seeing, only that it appears as a disk drive.  So you can choose to trust it (assume that it has RAID of an adequate level, for example) or you can combine multiple block storage devices together into RAID just as if they were regular, local disks.  This is extremely uncommon but is an interesting option and there are products that are designed to be used in this way.

More commonly, logical volume management such as Linux LVM, Solaris ZFS or Windows Dynamic Disks is applied on top of the exposed block storage from the device and then, on top of that, a filesystem would be employed.  This is important to remember, with block storage devices the filesystem is created and managed by the client device, not by the storage device.  The storage device is blissfully unaware of how the block storage that it is presenting is used and allows the end user to use it however they see fit with total control.  This extends even to the point that you can chain block storage devices together with one providing the storage to the next being combines, perhaps, into RAID groups – block storage devices can be layered, more or less, indefinitely.

Alternatively, a file storage device contains all of the block portion of the storage so any opportunity for RAID, logical volume management and monitoring must be handled by the file storage device.  Then, on top of the block storage, a filesystem is applied.  Commonly this would be Linux’ EXT4, FreeBSD and Solaris’ ZFS, Windows NTFS but other filesystems such as WAFL, XFS, JFS, BtrFS, UFS and more are certainly possible.   On this filesystem, data will be stored.  To them share this data with the outside world a network file system (also known as a distributed file system) is used which provides a file system interface that is network enabled – NFS, SMB and AFP being the most common but, like in any protocol family, there are numerous special case and exotic possibilities.

A remote device wanting to use storage on the file storage device would see it over the network the same as it would see a local filesystem and is able to mount it in an identical manner.  This makes file storage especially easy and obvious for end consumer to use as it is very natural in every aspect.  We use network file systems every day for normal desktop computing.  When we “map a drive” in Windows, for example, we are using a network file system.

One critical differentiation between block storage and file storage that must be differentiated between is that, while both potentially can sit on a network and allow multiple client machines to attach to them, only file storage devices have the ability arbitrate that access.  This is very important and cannot be glossed over.

Block storage appears as a disk drive.  If you simply attach a disk drive to two or more computers at once, you can imagine what will happen – each will know nothing of the other and will be unaware of new files being created, others changing and they systems will rapidly begin to overwrite each other.  If your file system is read only on all nodes, this is not a problem.  But if any system is writing or changing the data, the others will have problems.  This generally results in data corruption very quickly, typically on the order of minutes.  To see this in extreme action, imagine having two or three client system all believe that they have exclusive access to a disk drive and have them all defragment it at the same time.  All data on the drive will be scrambled in seconds.

A file storage device, on the other hand, has natural arbitration as the network file system handles the communications for access to the real file system and filesystems, by their nature, are multi-user naturally.  So if one system attached to a file storage device makes a change, all systems are immediately aware of the change and will not “step on each others toes.”  Even if they attempt to do the the file storage device’s filesystem arbitrates access and has the final say and does not let this happen.  This makes sharing data easy and transparent to end users.  (I use the term “end users” here to include system administrators.)

This does not mean that there is no means of sharing storage from a block device, but the arbitration of it cannot be handled by the block storage device itself.  Block storage devices are be made “shareable” by using what is known as a clustered file system.  These types of file systems originated back when server clusters shared storage resources by having two servers attached with a SCSI controller on either end of a single SCSI cable and having the shared drives attached in the middle of the cable.  The only means by which the servers could communicate was through the file system itself and so special clustered file systems were developed that allowed there to be communications between the devices, alerting each to changes made by the other, through the file system itself.  This actually works surprisingly well but clustered file systems are relatively uncommon with Red Hat’s GFS and Oracle’s OCFS being some of the best well known in the traditional server world and VMWare’s much newer VMFS having become extremely well known through its use for virtualization storage.  Normal users, including system administrators, may not have access to clustered file systems or may have needs that do not allow their use.  Of important note is also that the arbitration is handled through trust, not through enforcement, like with a file storage device.  With a file storage device, the device itself handles the access arbitration and there is no way around it.  With block storage devices using a clustered file system, any device that attaches to the storage can ignore the clustered file system and simply bypass the passive arbitration – this is so simple that it would normally happen accidentally.  It can happen when mounting the filesystem and specifying the wrong file system type or through a drive misbehaving or any malicious action.  So access security is critical at the network level to protect block level storage.

The underlying concept being exposed here is that block storage devices are dumb devices (think glorified disk drive) and file storage devices are smart devices (think traditional server.)  File storage devices must contain a full working “computer” with CPU, memory, storage, filesystem and networking.  Block storage devices may contain these things but need not.  At their simplest, block storage devices can be nothing more than a disk drive with a USB or Ethernet adapter attached to them.  It is actually not uncommon for them to be nothing more than a RAID controller plus Ethernet or Fiber Channel adapters to be attached.

In both cases, block storage device and file storage devices, we can scale down to trivially simple devices or can scale up to massive “mainframe class” ultra-high-availability systems.  Both can be either fast or slow.  One is not better or worse, one is not higher or lower, one is not more or less enterprise – they are different and serve generally different purposes.  And there are advanced features that either may or may not contain.  The challenge comes in knowing which is right for which job.

I like to think of block storage protocols as being a “standard out” stream, much like on a command line.  So the base level of any storage “pipeline” is always a block device and numerous block devices or transformations can exist with each being piped one to another as long as the output remains a block storage protocol.  We only terminate the chain when we apply a file system.   In this way hardware RAID, network RAID, logical volume management, etc. can be applied in multiple combinations as needed.  Block storage is truly not just blocks of data but building blocks of storage systems.

One point that is very interesting is that since block storage devices can be chained and since network storage devices must accept block storage as their “input” it is actually quite common for a block storage device (SAN) to be used as the backing storage for a file storage device (NAS), especially in high end systems.  They can coexist within a single chassis or they can work cooperatively on the network.

As a technologist I have been watching Windows 8 for some time having been using it, in some capacity, since the earliest public betas.  I’ve long struggled to come to terms with how Microsoft envisioned Windows 8 fitting into their existing customer base but have been, more or less, hopeful that the final release would fix many of my concerns.  When Windows 8 did finally release I was, sadly, left still wondering why it was so different from past Windows interfaces, what the ultimate intention was and how users were going to react to it.

It didn’t take long before I got a very thorough introduction to user reaction.  As a technology consultancy we tend to move quickly on new technologies and trends.  We may not deploy beta products into production but when new products release our update cycles are generally almost instantaneous.  We need to run the latest and the greatest all of the time so that we are ready for problems before anyone else allowing us to stay ahead of our customers.  So Windows 8 started getting prepped for rollout pretty much on the day that it was released to manufacturing.  This is when management got their first chance to try it out before the actual deployments started – the IT department had been playing with it since early betas.

Management came back to IT to ask critical questions concerning efficiency, usability and training.  Their reaction was that Windows 8′s interface was confusing and highly inefficient requiring a disruptive “jolt” of leaping to and from full screen menus that caused mental context shifting and loss of focus.  Many tasks require power user levels of knowledge to be usable while the interface seemed to be designed around low end “consumer” use and not very appropriate for people with the level of knowledge necessary to make the system functional.

It wasn’t that Windows 8 was unusable but failed at delivering the value traditionally associated with Windows, the value that causes us to traditionally move from version to version more or less without thinking and that is that sticking with Windows on the desktop delivers a predictable user experience requiring little to no retraining and an overall efficient experience.  Windows 8 requires extensive retraining, makes workers less efficient even after adapting to it and expects traditionally casual users to need to be power users to be effective.  While sticking with Windows is the obvious choice for IT departments with deep investments in Windows knowledge and skills (and tools), the value proposition for end users does not have the same continuity that it has in the past.

We read many reviews and consistently the answer to whether Windows 8 would deliver value to other organizations seemed to be focused on it being “good enough” and that with extensive training and all end users learning to “deal with” the interface issues and to learn totally new skills like jumping back and forth between mouse and keyboard, memorizing shortcut keys, etc. that the system could be made to be functional.  But never good, never ideal.  All concerns around Windows 8 aren’t about showing why it is better, just making it acceptable.  Hardly a position that we want to be in as an IT department.  We want to deliver solutions and value.  We want to make our businesses more efficient, not less.  We want to avoid disruption, not create it.

We even went to far as to visit Microsoft at a trade show putting on a display of Windows 8.  Even Microsoft’s own staff were unable to clarify the value proposition of Windows 8 or even in their demonstration environment get it to work “as intended”.  It is clear that even Microsoft themselves are not confident in the product or sure how their customers are expected to react to it.

The decision was made quickly: management wanted a demonstration of a Linux desktop immediately.  The first test was Linux Mint which ended up being the final choice as well. The non-IT users were really impressed with how easy Linux Mint was to use for people with a Windows background and nothing else.  It required no training – users literally just sat down and started working, unlike on Windows 8 where users were confused and needed help even with the simplest tasks like opening an application or shutting down the computer.  And there was essentially no pushback, people were universally excited about the opportunities that the new platform could provide, whereas people were actively concerned about how painful working with Windows 8 would be both up front and down the road.

That Windows 8 blundered so dramatically as to cause a competing product to get auditioned was not that surprising to me.  These things happen.  That the reaction of the non-IT staff was so dramatically in favor of a Linux distro was quite surprising, however.  Staff with no Linux exposure didn’t just see Linux as a low cost alternative or the lesser of two evils but were downright excited to use it.  Windows 8 caused Microsoft’s worst fears to come true – using Windows is no longer something that users can choose because it is familiar and comfortable.  If they feel the need or desire to test alternatives Windows will no longer compete on a “devil we know” basis like it traditionally has in the past but will need to compete on a basis of usability comparisons as Linux Mint, in this case, actually felt far more familiar and comfortable than Windows 8.

What did truly surprise me, however, was the ripple effect that changing the operating system had on the computing infrastructure.  Because Windows was being replaced this caused a series of questions to arise around other technology choices.  The first, probably somewhat obviously, was what to do about Windows-based applications that had no Linux versions?

We are lucky that the shop ran very standard applications and most applications are modern, browser-based ones so the bulk of systems worked on Linux transparently.  The only major application to require an alternative was Microsoft Office.  Fortunately the fix was easy, LibreOffice had everything that we needed and is built into the operating system.  Moving from MS Office to LibreOffice can be simple or intimidating depending on outside dependencies, complexity of use scenarios, heavy use of macros, etc.  We were lucky that across the board the move was trivial, in our case.

Dropping Microsoft Office left us without an effective email client for our Exchange email system.  So again, management asked, what compelling value is there for us in Exchange.  Shoulder shrugs followed.  Almost immediately a migration effort from a hosted Exchange service to Rackspace Email began.  This resulted in one of the largest cost savings, overall, in this entire process.

Next to be questioned was SharePoint.  Without desktop Active Directory integration, Microsoft Office integration and Exchange integration, was the overhead of running a heavy SharePoint installation of appreciable value to our organization?  SharePoint put up the biggest fight as it truly is a nearly irreplaceable system with numerous aspects and features that cannot be trivially compared to other systems.  In the end, however, without the slew of Microsoft integrated components SharePoint was deemed too costly and complex to warrant using on its own in our environment.

One by one, Microsoft products whose values were established through their tight integration with each other began to be eliminated in favor of lower cost, more flexible alternatives.  As one by one they were removed the value that they had cumulatively created diminished making each one less and less valuable without the others.

Before the move to a Linux desktop we had been preparing to install Lync as a replacement both to our instant messaging platform as well as our telephony platform.  Needless to say, that project was cancelled and our current systems, which integrate really well with Linux and were of much lower cost, were kept.

As we got to the end of eliminating Microsoft-based applications it became apparent that using Active Directory for centralized authentication was not cost effective.  This last piece will take quite some time to phase out completely as creating a new, centralized authentication mechanism will take quite a bit of planning and implementation time, but the process has begun to move to a completely different platform.

Even applications that we thought were sacred and untouchable, where plans were in place to keep them running on dedicated Windows instances just for special purposes like accounting, ended up being less sacred than we had anticipated.  New applications were found and systems were migrated.

Of course support infrastructure followed as well with System Center and Windows-focused backup systems no longer needed.  And Windows-based file servers stopped making sense without Windows clients to support.

At the end of the day what was so shocking was that the littlest thing, a concern over the efficiency and usability of Windows 8′s new interface, triggered a series of discoveries that completely unraveled our Microsoft-centered ecosystem.  No single product was unloved or disliked.  We were a team of dedicated Windows 7 desktop users on a wholly Microsoft infrastructure and happy with that decision and happy to be continuing to move more and more over to the Microsoft “way”.  But by simply questioning the assumption that we wanted or needed to be using a Windows desktop ended up bringing down an infrastructural house of cards.

From an end user perspective, the move to Linux was effortless.  There has been quite a bit of retraining and rethinking from the support side, of course.  There is a lot to learn there, but that is IT’s job – support the business and do what needs to be done to make them able to work most efficiently.

Does this bode of a dark future for Windows?  Unlikely, but it does highlight that a significant misstep on the desktop platform could easily put Microsoft’s market position on a downward spiral.  Microsoft depends on tight integration between their systems to create their value proposition.  Losing the desktop component of that integration can quickly undermine the remaining pieces.  To be sure, ours is a special case scenario – a small firm with extensive UNIX skills already existing in house, an ambitious and forward thinking management team and the agility to make broad changes combined with more than a decade of seeking platform independence in application choices, but just because we lie on the extreme edge does not mean that our story is not an important one.  For some, Windows 8 might not only represent the tipping point in the Windows desktop value proposition but the tipping point in the Microsoft ecosystem itself.

One of the biggest mistakes that I see when dealing with companies of all sizes is the proclivity for IT professionals to forget the context in which they are working and start behaving in one of two ways.  First forgetting context completely and leaving IT for “hobbyist land” and looking at the technologies and equipment that we use purely as toys for the enjoyment and fulfillment of the IT department itself without consideration for the business.  The second is treating the business as generic instead of respecting that every business has unique needs and IT must adapt to the environment which it is in.

The first problem, the hobbyist problem, is the natural extension of the route through which most IT Professionals arrive in IT – they love working on computers and would do so on their own, at home, whether they were paid to do so or not.  This brings often a lifetime of “tech for the sake of tech” feeling to an IT Pro and is nearly universal in the field.  Few other professionals find themselves so universally drawn to what they do that they would do it paid or not.  But this shared experience creates a culture of often forgetting that the IT department exists within the context of a specific corporate entity or business unit and that its mandate exists only within that context.

The second problem stems, most likely, from broad IT and business training that focuses heavily on rules of thumb and best practices which, generally, require “common scenarios” as these are easy to teach by rote and leave out the difficult pieces of problem analysis and system design.  Custom tailoring not only solutions but IT thinking to the context of a specific business with specific needs is difficult and requires learning a lot about the business itself and a lot of thought to put IT into the context of the business specifically.

The fault does not necessarily lie with IT alone.  Business often treat their IT departments as nothing but hobbyists and focus far too heavily on technical and not business skills and often keep IT at arm’s length forgetting that IT has some of the most important business insight as they tend to cross all company boundaries.  IT needs deep access to business processes, workflows, plus planning and goals to be able to provide good advisement to the business but is often treated as if this information is not needed.  Businesses, especially smaller ones, tend to think of IT as a magic box with a set budget that money goes in and network plumbing comes out.  Print and radio ads promote this thinking.  IT as a product is poor business thinking.

In the defense of the business, IT operates in a way that few businesses are really prepared to handle.  IT is a cost center in that there is a base cost needed to keep any company functioning.  But beyond this, IT can be an opportunity center in most businesses, but this requires both IT and the business to work together to create these opportunities and even moreso to leverage them.

IT is often put in the inappropriate position of being forced to justify its own existence.  This is nonsensical as human resources, accounting, legal, management, janitorial, sales, marketing and production departments are never asked to demonstrate their financial viability.  Needing to do so puts an unfair strain on the IT department requiring non-business people to present business cases and wastes resources and hampers thinking in a vain attempt at producing pointless metrics.  This is a flaw in business thinking often caused by a rift between management and the people that they’ve hired to support them.  The relationship is often cold or even adversarial or cursory when it should be close and involved.  IT should be sitting at the decision table, it brings insight and it needs insight.

One of the biggest challenges that IT faces is that it is often in a position of needing to convince the business to do what is in the business’ own best interest.  This is, for the most part, a flaw in business thinking.  The business should not demand to stand in a position of doing the wrong thing and only be willing to do the right thing if it can be “sold” to them.  This is a fundamental flaw in approach.  It should be a process of good decision making, not starting from bad decision making unless being convinced otherwise.  Other departments are not presented with a similar challenge.  What other department regularly has to mount a campaign to request necessary resources?

Due to this challenge in constantly fighting for management attention and resources, IT needs to develop internal business skills in order to cope.  This is a reality of most IT departments today.  The ability not only to keep the business that they support in context and to make IT decisions based on this context but then be able to act as marketing and sales people taking these decisions and delivering them to the business in a manner similar to how outside vendors and salespeople would do is critical.  Outside vendors are sending skilled sales people and negotiators to the business in an attempt to do an end run around IT, IT needs the same skills (with the advantage of insider knowledge and the obvious advantage of having the best interest of the business) in order to demonstrate to the business why their solutions, opportunities and needs are important for consideration.

Having good interpersonal, writing and presentation skills is not enough, of course.  Knowing business context and leveraging it efficiently includes understanding factors such as risk, opportunity, loss, profit and being able to apply these to the relationship between the businesses’ IT investments and the bottom line.  Often IT Pros will be frustrated when the business is unwilling to invest in a solution that they present but forget that the business is considering (we hope) the total cost of ownership and the impact on the company’s bottom line.  When asked how the solution will save money or generate revenue, even indirectly, often, at best, the answers are vague and lack metrics.  Before going to the business with solutions, IT departments need to vet recommendations internally and ask tough questions like:

• How does this solution save money today?  Or how does it make us more money?
• How much money is it expected to save or earn?
• What business problem are we attempting to solve? (What itch are we trying to scratch?)
• What risks do we take on or reduce?

Or similar lines of thinking.  Instead of bringing technology to the business, bring solutions.  Identify problems or opportunity and present a case.  Role play and imagine yourself as a business owner disinterested in a solution.  Would you feel that the investment requested is a good one?  Too often we in IT like a solution because it is advanced, complex, “the right way to do it”, because another company is doing it, because it is the hot trend in IT and often we have very good reasons for wanting to bring these techniques or technologies into our workplace but forget that they may not apply or apply well to the business as it is and its financial capabilities or the business roadmap.

When I speak to IT professionals looking for advice on a system design or approach my first question is pretty universally: “What business need are you attempting to solve?”  Often this question is met with silence.  The business had not been considered in the selection of the solution being presented.  Regularly bringing requests or solutions to the business that do not take into consideration the context of the IT department within the business will rapidly train business decisions makers to distrust the advice coming from the IT department.  Not that they would feel that the advice is intentionally skewed but they, and often rightfully so, will suspect that the decisions are being brought forward from a technical basis alone and isolated from the concerns of the business.  Once this distrust is in place it is difficult to return to a healthier relationship.

Making the IT department continuously act within the context of the business that it serves, encouraging IT to pursue business skills and to approach the business for information and insight and making the business see IT as a partner and supporter with whom information must be shared and insight should be gathered can be a tall order.  The business is not likely to take the first step in improving the interaction.  It is often up to IT to demonstrate that it is considering the needs of the business, often moreso than the business itself, and considering the potential financial impact or benefit of its decisions and recommendations.  There is much to be gained from this process, but it is not an easy process

It is important to remember that the need for IT to keep business context is crucial, to some degree, for all members of the IT team, especially those making recommendations, but the ability to judge business need, understand high level workflows, understand financial ramifications, seek opportunity is a combination of IT management (CIO, Dir. of IT, etc.) and the IT department as a whole.  Many non-managerial technical members need not panic and feel that their lack of holistic business vision and acumen will keep them from adequately performing their role within the business context, but it does limit their ability to provide meaningful guidance to the business outside of extremely limited scopes.  Even common job roles, such as deskside support, need to have some understanding of the fiscal responsibilities of the IT department however, such as recognizing when the cost of fixing a low cost component may far exceed the cost of replacing the component with one that is new and, potentially, better.

But it need not be.  Complexity is the enemy of both reliability and security.  Unnecessarily complex solutions increase cost both in acquisition and in implementation as well as in maintenance while being generally slower, more fragile and possess a large attack surface that is harder to comprehend and protect.  Simple, or more appropriately, elegant solutions are the best approach.  This does not mean that all designs will be simple, not at all.  Complex designs are often required.  IT is hardly a field that has any lack of complexity.  In fact it is often believed that software development may be the most complex of all human endeavors, at least of those partaken of on any scale.  A typical IT installation includes millions of lines of codes, hundreds or thousands of protocols, large numbers of interconnected systems, layers of unique software configurations, more settings than any team could possibly know and only then do we add in the complexity of hundreds or thousands or hundreds of thousands of unpredictable, irrational humans trying to use these systems, each in a unique way.  IT is, without a doubt, complex.

What is important is to recognize that IT is complex, that this cannot be avoided completely but to focus on designing and engineering solutions to be as simple, as graceful… as elegant as possible.  This design idea comes from, at least in my mind, software engineering where complex code is seen as a mistake and simple, beautiful code that is easy to read, easy to understand is considered successful.  One of the highest accolades that can be bestowed upon a software engineer is for her code to be deemed elegant.  How apropos that it is attributed to Blaise Pascal, after whom one of the most popular programming languages of the 1970s and 1980s was named is this famous quote (translated poorly from French): “I am sorry I have had to write you such a long letter, but I did not have time to write you a short one.”

It is often far easier to design complex, convoluted solutions than it is to determine what simple approach would suffice.  Whether we are in a hurry or don’t know where to begin an investigation, elegance is always a challenge. The industry momentum is to promote the more difficult path.  It is in the interest of vendors to sell more gear not only to make the initial sale but they know that with more equipment comes more support dollars and if enough new, complex equipment is sold the support needs stop increasing linearly and begin to increase geometrically as additional support is needed not just for the equipment or software itself but also for the configuration and support of system interactions or additional customization   The financial influences behind complexity are great, and they do not stop with vendors.  IT professionals gain much job security, or the illusion of it, by managing large sets of hardware and software that are difficult to seamlessly transition to another IT professional.

Often complexity is so assumed, so expected, that the process of selecting a solution begins with great complexity as a foregone conclusion without any consideration for the possibility that a less complex solution might suffice, or even be superior outside of the question of complexity and cost itself.  Complexity is sometimes even completely tied to certain concepts to a degree where I have actually faced incredulity at the notion that a simple solution might outperform in price, performance and reliability a complex one.

Rhetoric is easy, but what is a real world example?  The best examples that I see today are mostly related to virtualization whether vis a vis storage or a cloud management layer or software or just virtualization itself.  I see quite frequently that a conversation involving just virtualization for one person brings an instant connotation of requiring networked, shared block storage, expensive virtualization management software, many redundant virtualization nodes and complex high availability software – none of which are intrinsic to virtualization and most of which are rarely for the purpose of supporting or really, even in the interest of the business for whom they will be implemented.  Rather than working from business requirements, these concepts arise predominantly from technology preconceptions.  It is simple to point to complexity and appear to be solving a problem – complexity creates a sense of comfort.  Filter many arguments down and you’ll hear “How can it not work, it’s complex?”  Complexity provides an illusion of completeness, or having solved a problem, but this can commonly hide the fact that a solution may not actually be complete or even functional but the degree of complexity makes this difficult to see.  Our minds will then not accept easily a simpler approach being more complete and solving a problem when a complex one does not because it feels so counter-intuitive.

A great example of this is that we resort to discussing redundancy rather than reliability.  Reliability is difficult to measure, redundancy is simple to quantify.  A brick is highly reliable, even when singular.  It does not take redundancy for a brick to be stable and robust.  Its design is simple.  You could make a supporting structure out of many redundant sticks that would not be nearly as reliable as a single brick.  If you talk in reliability – the chance that the structure will not fail – it is clear that the brick is a superior choice to several sticks.  But if you say “but there is no redundancy, the brick could fail and there is nothing to take its place” you sound silly.  But when talking about computers and computer systems we find systems that are so complex that rarely do people see when they have a brick or a stick and so, since they cannot determine reliability which matters, they focus on the easily to quantify redundancy, which doesn’t.  The entire system is too complex, but seeking the simple solution, the one that directly addresses the crux of the problem to solve can reduce complexity and provide us a far better answer in the end.

This can even be seen in RAID.  Mirrored RAID is simple, just one disk or set of disks being an exact copy of another set.  It’s so simple.  Parity RAID is complex with calculations on a variable stripe across many devices that must be encoded when written and decoded should a device fail.  Mirrored RAID lacks this complexity and solves the problem of disk reliability through simple, elegant copy operations that are highly reliable and very well understood.  Parity RAID is unnecessarily complex making it fragile.  Yet in doing so and by undermining its own ability to solve the problem for which it was designed it also, simultaneously, because seemingly more reliable based on no factor other than its own complexity.  The human mind immediately jumps to “it’s complex, therefore it is more advanced, therefore it is more reliable” but neither progression is a logical one.  Complexity does not suggest that it is more advanced and being advanced does not suggest that it is reliable, but the human mind itself is complex and easily mislead.

There is no simple answer for finding simplicity.  Knowing that complexity is bad by its nature but unavoidable at times teaches us to be mindful, however it does not teach us when to suspect over-complexity.  We must be vigilant, always seeking to determine if a more elegant answer exists and not accept complexity as the correct answer simply because it is complex.  We need to question proposed solutions and question ourselves.  ”Is this solution really as simple as it should be?”  ”Is this complexity necessary?”  ”Does this require the complexity that I had assumed?”

In most system design recommendations that I give, the first technical determination step that I normally take, after the step of inquiring as to the business need needing to be solved, is to question complexity.  If complexity cannot be defended, it is probably unnecessary and actively defeating the purpose for which it was chosen.

“Is it really necessary to split those drives into many separate arrays?  If so, what is the technical justification for doing so?”

“Is shared storage really necessary for the task that you are proposing it for?”

“Does the business really justify the use of distributed high availability technologies?”

“Why are we replacing a simple system that was adequate yesterday with a dramatically more complex system tomorrow?  What has changed that makes a major improvement, while remaining simple, not more than enough but requires orders of magnitude more complexity and more spending that wasn’t justified previously?”

These are just common examples, complexity exists in every aspect of our industry.  Look for simplicity.  Strive for elegance.  Do not accept complexity without rigorously vetting it.  Put it through the proverbial ringer.  Do not allow complexity to creep in where it is not warranted.  Do not err on the side of complexity – when in doubt, fail simply.  Oversimplifying a solution typically results in a minor failure while making it overly complex allows for a far greater degree of failure.  The safer bet is with the simpler solution.  And if a simple solution is chosen and proven inadequate it is far easier to add complexity than it is to remove it.

For a few years nearly the entire field was trained on the same small knowledge set and many rules of thumb became de facto standards and much of the knowledge of the time was learned by rote and passed on mentor to intern in a cycle that moved much of the technical knowledge of 1998 into the unquestioned, set-in-stone processes of 2012.  At the time this was effective because the practices were relevant but that was fifteen years ago, technology, economics, use cases and knowledge have changed significantly since that time.

One of the best examples of this was the famous Microsoft SQL Server recommendation of RAID 1 for the operating system, RAID 5 for the database files and another RAID 1 for the logs.  This setup has endured for nearly the entire life of the product and was so well promoted that it has spread into almost all aspects of server design in the SMB space.  The use of RAID 1 for the operating system and RAID 5 for data is so pervasive that it is often simply assumed without any consideration as to why this was recommended at the time.

Let’s investigate the history and see why R1/5/1 was good in 1998 and why it should not exist today.  Keep some perspective in mind, the gap between when these recommendations first came out (as early as 1995) compared to today is immense.  Go back, mentally, to 1995 and think about the equivalent gap at the time.  That would have been like using recommendations in the early Internet age based on home computing needs for the first round of Apple ][ owners!  The 8bit home computer era was just barely getting started in 1978.  Commodore was still two years away from releasing their first home computer (the VIC=20) and would go through the entire Commodore and Commodore Amiga eras and go bankrupt and vanish all before 1995.  The Apple ][+ was still a year away.  People were just about to start using analogue cassette drives as storage.  COBOL and Fortran were the only series business languages in use.  Basically, the gap is incredible.  Things change.

First, we need to look at the factors that existed in the late 1990s that created the need for our historic setup.

1. Drives were small, very small.  A large database array might have been four 2.1GB SCSI drives in an R5 array for just ~6GB of usable storage space on a single array.  The failure domain for parity RAID failure was tiny (compared to things like URE fail rates.)
2. Drive connection technologies were parallel and slow.  The hard drives of the time were only slightly slower than drives are today but the connection technologies represented a considerable bottleneck.  It was common to split traffic to allow for reduced bus bottlenecks.
3. SCSI drive technology was the only one used for servers.  The use of a PATA (IDE it was called at the time) in a server was unthinkable.
4. Drives were expensive per gigabyte so cost savings was the key issue, while maintaining capacity, for effectively all businesses.
5. Filesystems were fragile and failed more often than drives.
6. Hardware RAID was required and only basic RAID levels of 1 and 5 were commonly available.  RAID 6 and RAID 10 were years away from being accessible to most businesses.  RAID 0 is discounted as it has no redundancy.
7. Storage systems were rarely, if ever, shared between servers so access was almost always dedicated to a single request queue.
8. Storage caches were tiny or did not exist making drive access limitations pass directly onto the operating system.  This meant having different arrays with different characteristics to handle different read/write or random/sequential access mixes.
9. Drive failure was common and the principle concern of storage system design.
10. Often drive array size was limited by physical limitations so often array splitting decisions were made out of necessity, not choice.
11. A combination of the above factors meant that RAID 1 was best for some parts of the system where small size was acceptable and access was highly sequential or write heavy and RAID 5 was best for others where capacity outweighed reliability and where access was highly random and read heavy.

In the nearly two decades since the original recommendations were released, all of these factors have changed.  In some cases the changes are cascading ones where the move from general use RAID 5 to general use RAID 10 has then causes what would have been the two common array types, RAID 1 and RAID 10, to share access characteristics so the need or desire to use one or the other depending on load type is gone.

1. Drives are now massive.  Rather than struggling to squeeze what we need onto them, we generally have excess capacity.  Single drives over a terabyte are common, even in servers.  Failure domains for parity are massive (compared to things like URE fail rates.)
2. Drive connections are serial and fast.  The drive connections are no longer a bottleneck.
3. SATA is now common on servers skewing potential risks for URE in a way that did not exist previously.
4. Capacity is now cheap but performance and reliability are now the key concerns for dollars spent.
5. Filesystems are highly robust today and filesystem failures are “background noise” in the greater picture of array reliability.
6. Hardware RAID and software RAID are both options today and available RAID levels include many options but, most importantly, RAID 10 is available ubiquitously.
7. Storage systems are commonly shared making sequential access even less common.
8. Storage caches are commonly and often very large.  512MB and 1GB caches are considered normal today making many arrays in 1995 fit entirely into memory on the RAID controller today.  With caches growing rapidly compared to storage capacity and the recent addition of solid state drives as L2 cache in storage in the last two years it is not out of the question for even a small business to have databases and other performance sensitive applications running completely from cache.
9. Drive failure is uncommon and of trivial concern to storage system design (compared to other failure types.)
10. Drive array size is rarely limited by physical limitations.
11. The use of RAID 1 and RAID 10 as the principle array types today means that there is no benefit to using different array levels for performance tuning.

These factors highlight why the split array system of 1995 made perfect sense at the time and why it does not make sense today.  OBR10, today’s standard, was unavailable at the time and cost prohibitive.  RAID 5 was relatively safe in 1995, but not today.  Nearly every factor involved in the decision process has changed dramatically in the last seventeen years and is going to continue to change as SSD becomes more common along with auto-tiering, even larger caches and pure SSD storage systems.

The change in storage design over the last two decades also highlights the dangers that IT faces as a large portion of the field learns, as is common in engineering, basic “rules of thumb” or “best practices” without necessarily understanding the underlying principles that drive those decisions making it difficult to know when not to apply those best practices or, even more importantly, when to recognize that the rule no longer applies.  Unlike traditional mechanical or civil engineering where new advances and significant factor changes may occur once or possibly never over the course of a career, IT still changes fast enough that complete “rethinks” of basic rules of thumb are required several times through a career.  Maybe not annually, but once per decade or more is almost always necessary.

The current move from uniprocessing to multithreaded architectures is another similar, significant change requiring the IT field to completely change how system design is handled.

• Scenario: Netgear ReadyNAS Pro 2 with XRAID mirror.  Notation: R1
• Scenario: Two Netgear ReadyNAS Ultra units with local RAID 1 sync’d over the network using rsync.  Notation: R1{1}
• Scenario: Two Drobo B800fs NAS devices each loaded with single parity RAID sync’d using DroboSync. Notation: R5{1}
• Scenario: Two Drobo B800fs NAS devices each with dual parity RAID sync’d using DroboSync.  Notation: R6{1}
• Scenario: Two Linux servers with R6 locally using DRBD Mode A or B (asynchronous.)  Notation: R6[1]
• Scenario: Two Linux servers with R5 locally using DRBD Mode C (synchronous.)  Notation: R6(1)
• Scenario: Three node VMware vSphere VSA cluster with local R10.  Notation: R10(1)³
• Scenario: Windows server with two four disk R0 stripes mirrored.  Notation: 8R01
• Scenario: Two FreeBSD servers with R10 using HAST with memsync.  Notation: R10[1]
• Scenario: Two FreeBSD servers with R1 using HAST with sync.  Notation: R1(1)
• Scenario: Two Windows file servers with R10 using Robocopy to synchronize file systems. Notation: R10{1}
• Scenario: Single Netgear SC101 SAN* using ZSAN drivers on Windows with two disks. Notation: R(1)

Technology References:

HAST: http://wiki.freebsd.org/HAST

DRBD: http://www.drbd.org/users-guide/s-replication-protocols.html

DroboSync: http://www.drobo.com/solutions/for-business/drobo-sync.php

Rsync: http://rsync.samba.org/

Robocopy: http://technet.microsoft.com/en-us/library/cc733145%28v=ws.10%29.aspx

Notes:

*The Netgear SC101 SAN is interesting in that while it can hold two PATA drives internally and exposes them to the network as block devices, via the ZSAN protocol, through a single Ethernet interface but there is no internal communications between the devices so all mirroring of the array happens in Windows which actually sees each disk as an entirely separate SAN device each with its own IP address.  Windows has no way to know that the two devices are related. The RAID 1 mirroring is handled one hundred percent in software RAID on Windows and the SAN itself is always two independent PATA drives exposed raw to the network.  A very odd, but enlightening device.

Traditional RAID comes in single digit notation and the available levels are 0, 1, 2, 3, 4, 5, 6, 7.  Level 7 is unofficial but widely accepted as triple parity RAID (the natural extension of RAID 5 and RAID 6) and RAID 2 and RAID 3 are effectively disused today.

Nested RAID, one RAID level within another, is handled by putting single digit RAID levels together such as RAID 10, 50, 61, 100, etc.  These can alternatively be written with a plus sign separating the levels like RAID 1+0, 5+0, 6+1, 1+0+0, etc.

There are two major issues with this notation system, beyond the obvious issue that not all RAID types or extensions are accounted for by the single digit system with many of the aspects of proprietary RAID systems such as ZRAID, XRAID and BeyondRAID being unaccounted for in the notation system.  The first is a lack of network RAID notation and the second is a lack of specific denotation of intra-RAID configuration.

Network RAID comes in two key types, synchronous and asynchronous.  Synchronous network RAID operates effectively identically to its non-networked counterpart.  Asynchronous functions the same but brings extra risks as data may not be synchronized across devices at the time of a device failure.  So the differences between the two need to be visible in the notation.

Synchronous RAID should be denoted with parenthesis.  So two local RAID 10 systems mirrored over the network (a la DRBD) would be denoted RAID 10(1).  The effective RAID level for risk and capacity calculations would be the same as any RAID 101 but this informs all parties at a glance that the mirror is over a network.

Asynchronous RAID should be denoted with brackets.  So two local RAID 10 systems mirrored over the network asynchronously would be denoted as RAID 10[1] making it clear that there is a risky delay in the system.

There is an additional need for a different type of replication at a higher, filesystem level (a la rsync) that, while not truly related to RAID, provides a similar function for cold data and is often used in RAID discussions and I believe that storage engineers need the ability to quite denote this as well.  This asynchronous file-system level replication can be denoted by braces.  Only one notation is needed as file-system level replication is always asynchronous.  So as an example, two RAID 6 arrays synced automatically with a block-differential file system replication system would be denoted as RAID 6{1}.

To further simplify RAID notation and to shorten the obvious need to write the word “RAID” repeatedly as well as to remove ourselves from the traditional distractions of what the acronym stands for so that we can focus on the relevant replication aspects of it, a simple “R” prefix should be used.  So RAID 10 would simply be R10.  Or a purely networked mirror might be R(1).

This leaves one major aspect of RAID notation to address and that is the size of each component of the array.  Often this is implied but some RAID levels, especially those that are nested, can have complexities missed by traditional notation.  Knowing the total number of drives in an array does not always denote the setup of a specific array.  For example a 24 drive R10 is assumed to be twelve pairs of mirrors in a R0 stripe.  But it could be eight sets of triple mirrors in a R0 stripe.  Or it could even be six quad mirrors.  Or four sext mirrors.  Or three oct mirrors.  Or two dodeca mirrors.  While most of these are extremely unlikely, there is a need to notate it.  For the set size we use a superscript number to denote the size of that set.  Generally this is only needed for one aspect of the array, not all, as others can be derived, but when in down it can be denoted explicitly.

So an R10 array using three-way mirror sets would be R1³0.  Lacking the ability to write a superscript you could also write it as R1^3+0.  This notation does not state the complete array size, only its configuration type.  If all possible superscripts are included a full array size can be calculated using nothing more.  If we have an R10 of four sets of three-way mirrors we could write it R1³0⁴ which would inform us that the entire array consists of twelve drives – or in the alternate notation R1^3+0^4.

Superscript notation of sets is only necessary when non-obvious.  R10 with no other notation implies that the R1 component is mirror pairs, for example.  R55 nearly always requires additional notation except when the array consist of only nine members.

One additional aspect to consider is notating array size.  This is far simpler than the superscript notation and is nearly always complete adequate.  This alleviates the need to write in long form “A four drive RAID 10 array.”  Instead we can use a prefix for this.  4R10 would denote a four drive RAID 10 array.

So to look at our example from above, the twelve disk RAID 10 with the three-way mirror sets could be written out as 12R1³0⁴.  But the use of all three numbers becomes redundant.  Any one of the numbers can be dropped.  Typically this would be the final one as it is the least likely to be useful.  The R1 set size is useful in determining the basic risk and the leading 12 is used for capacity and performance calculations as well as chassis sizing and purchasing.  The trailing four is implied by the other two numbers and effectively useless on its own.  So the best way to write this would be simply 12R1³0.  If that same array was to use the common mirror pair approach rather than the three-way mirror we would simply write 12R10 to denote a twelve disk, standard RAID 10 array.

Today the storage landscape has changed.  Filesystems are incredibly robust and corruption from the filesystem itself is almost unheard of and, thank to technologies like journalling, can almost always be corrected quickly and effectively protecting the end users from data loss.  Almost no one worried about filesystem corruption today.

Modern filesystem are also able to handle far more capacity than they could previously.  It was not uncommon in the late 1990s and early 2000s to have the ability to easily make a drive array larger than any single filesystem could handle. Today that is not reasonably the case as all common filesystems handle many terabytes at least and often petabytes, exabytes or more of data.

Hard drives are much more reliable than they were in the late 1990s.  Failure rates for an entire drive failing are very low, even in less expensive drives.  So low, in fact, that array failures (data loss in the entire RAID array) is concerned with failing arrays primarily, rather than the failure of hard drives.  We no longer replace hard drives with wild abandon.  It is not unheard of for large arrays to run their entire lifespans without losing a single drive.

Capacities have scaled dramatically.  Instead of 4.3GB hard drives we are installing 3TB drives.  Nearly one thousand times more capacity on a single spindle compared to less than fifteen years ago.

These factors come together to create a need for a dramatically different approach to server storage design and a change to the “rule of thumb” about where to start when designing storage.

The old approach can be written RAID 1 + RAID 5.  The RAID 1 space was used for the operating system while the RAID 5 space, presumably much larger, was used for data and applications.  This design split the two storage concerns putting maximum effort into protecting the operating system (which was very hard to recover in case of disaster and on which the data relied for accessibility) onto highly reliable RAID 1.  Lower cost RAID 5, while somewhat riskier, was chosen, typically, for data because the cost of storing data on RAID 1 was too high in most cases.  It was a tradeoff that made sense at the time.

Today, with our very different concerns, a new approach is needed, and this new approach is known as “One Big RAID 10″ – meaning a single, large RAID 10 array with operating system, applications and data all stored together.  Of course, this is just what we say to make it handy, in a system without the needs of performance or capacity beyond a single disk we would say “One Big RAID 1″, but many people include RAID 1 in the RAID 10 group so it is just easier to say the former.

To be even handier, we abbreviate this to OBR10.

Because the cost of storage has dropped considerably and instead of being at a premium is typically in abundance today, because filesystems are incredibly reliable, because RAID 1 and RAID 10 share performance characteristics and because non-disk failure triggered array failures have moved from background noise to primary causes of data loss the move to RAID 10 and to eliminate array splitting has become the new standard approach.

With RAID 10 we now have the highly available and resilient storage previously held only for the operating system available to all of our data.  We get the benefit of mirrored RAID performance plus the benefit of extra spindles for all of our data.  We get better drive capacity utilization and performance based on that improved utilization.

Even the traditional splitting of log files normally done with databases (the infamous RAID 1 + RAID 5 + RAID 1 approach) is no longer needed because RAID 10 keeps the optimum performance characteristics across all data.  With RAID 10 we eliminate almost all of the factors that once caused us to split arrays.

The only significant factor, that has not been mentioned, for which split arrays were traditionally seen as beneficial is access contention – the need for different processes to need access to different parts of the disk at the same time causing the drive head to move around in a less than ideal pattern reducing drive performance.  Contention was a big deal in the late 1990s when the old rule of thumb was developed.

Today, drive contention still exists but has been heavily mitigated by the use of large RAID caches.  In the late 90s drive caches were a few megabytes at best and often non-existent.  Today 256MB is a tiny cache and average servers are deployed with 1-2GB of cache on the RAID card alone.  Some systems are beginning to integrate additional solid state drive based caches to add a secondary cache beyond the memory cache on the controller.  These can easily add hundreds of gigabytes of extremely high speed cache that can buffer nearly any spindle operation from needing to worry about contention.  So the issue of contention has been solved in other ways over the years but has, like other technology changes, effectively freed us from the traditional concerns requiring us to split arrays.

Like array contention, another, far less common reason for splitting arrays in the late 1990s was to improve communications bus performance because of the limitations of the era’s SCSI and ATA technologies.  These, too, have been eliminated with the move to serial communications mechanisms, SAS and SATA, in modern arrays.  We are no longer limited to the capacity of a single bus for each array and can grow much larger with much more flexibility than previously.  Bus contention has been all but eliminated.

If there is a need to split off space for protection, such as log file growth, this can be achieved through partitioning rather than through physical array splitting.  In general you will want to minimize partitioning as it increases overhead and lowers the ability of the drives to tune themselves but there are cases where it is the better approach.  But it does not require that the underlying physical storage be split as it traditionally was.  Even better than partitioning, when available, is logical volume management which makes partition-like separations without the limitations of partitions.

So at the end of the day, the new rule of thumb for server storage is “One Big RAID 10.”  No more RAID 5, no more array splitting.  It’s about reliability, performance, ease of management and moderate cost effectiveness.  Like all rules of thumb, this does not apply to every single instance, but it does apply much more broadly than the old standard ever did.  RAID 1 + RAID 5, as a standard, was always an attempt to “make due” with something undesirable and to make the best of a bad situation.   OBR10 is not like that.  The new standard is a desired standard – it is how we actually want to run, not something with which we have been “stuck”.

When designing storage for a new server, start with OBR10 and only move away from it when it specifically does not meet your technology needs.  You should never have to justify using OBR10, only justify not using it.

The earliest commonly accepted work on virtualization began in 1964 with the IBM CP-40 operating system developers for the IBM System/360 mainframe.  This was the first real foray into commercial virtualization and the code and design from this early virtualization platform has descended today into the IBM VM platform that has been used continuously since 1972 as a virtualization layer for the IBM mainframe families over the decades.  Since IBM first introduced virtualization we have seen enterprise systems adopting this pattern of hardware abstraction almost universally.  Many large scale computing systems, minicomputers and mainframes, moved to virtualization during the 1970s with the bulk of all remaining enterprise systems doing so, as the power and technology were available to them, during the 1980s and 1990s.

The only notable holdout to virtualization for enterprise computing was the Intel IA32 (aka x86) platform which lacked the advanced hardware resources necessary to implement effective virtualization until the advent of the extended AMD64 64-bit platform and even then only with specific new technology.  Once this was introduced the same high performance, highly secure virtualization was available across the board on all major platforms for business computing.

Because low cost x86 platforms lacked meaningful virtualization (outside of generally low performance software virtualization and niche high performance paravirtualization platforms) until the mid-2000s this left virtualization almost completely off of the table for the vast majority of small and medium businesses.  This has lead many dedicated to the SMB space to be unaware that virtualization is a well established, mature technology set that long ago established itself as the de facto pattern for business server computing.  The use of hardware abstraction is nearly ubiquitous in enterprise computing with many of the largest, most stable platforms having no option, at least no officially support option, for running systems “bare metal.”

There are specific niches where the need to avoid hardware abstraction through virtualization is not advised but these are extremely rare, especially in the SMB market.  Typical systems needing to not be virtualized include latency sensitive systems (such as low latency trading platforms) and multi-server combined workloads such as HPC compute clusters where the primary goal is performance above stability and utility.  Neither of these are common to the SMB.

Virtualization offers many advantages.  Often, in the SMB where virtualization is less expected, it is assumed that virtualization’s goal is consolidation where massive scale cost savings can occur or in providing new ways to provide for high availability.  Both of these are great options that can help specific organizations and situations but neither is the underlying justification for virtualization.  We can consolidate and achieve HA through other means, if necessary.  Virtualization simply provides us with a great array of options in those specific areas.

Many of the uses of virtualization are artifacts of the ecosystem such as a potential reduction in licensing costs.  These types of advantages are not intrinsic advantages to virtualization but do exist and cannot be overlooked in a real world evaluation.  Not all benefits apply to all hypervisors or virtualization platforms but nearly all apply across the board.  Hardware abstraction is a concept, not an implementation, so how it is leveraged will vary.  Conceptually, abstracting away hardware whether at the storage layer, at the computing layer, etc. is very important as it eases management, improves reliability and speeds development.

Here are some of the benefits from virtualization.  It is important to note that outside of specific things such as consolidation and high availability nearly all of these benefits apply not only to virtualizing on a single hardware node but for a single workload on that node.

1. Reduced human effort and impact associated with hardware changes, breaks, modifications, expansion, etc.
2. Storage encapsulation for simplified backup / restore process, even with disparate hardware targets
3. Snapshotting of entire system for change management protection
4. Ease of archiving upon retirement or decommission
5. Better monitoring capabilities, adding out of band management even on hardware platforms that don’t offer this natively
6. Hardware agnosticism provides for no vendor lock-in as the operating systems believe the hypervisor is the hardware rather than the hardware itself
7. Easy workload segmentation
8. Easy consolidation while maintaining workload segmentation
9. Greatly improved resource utilization
10. Hardware abstraction creates a significantly realized opportunity for improved system performance and stability while lowering the demands on the operating system and driver writers for client operating systems
11. Simplified deployment of new and varied workloads
12. Simple transition from single platform to multi-platform hosting environments which then allow for the addition of options such as cloud deployments or high availability platform systems
13. Redeployment of workloads to allow for easy physical scaling

In today’s computing environments, server-side workloads should be universally virtualized for these reasons.  The benefits of virtualization are extreme while the downsides are few and trivial.  The two common scenarios where virtualization still needs to be avoided are in situations where there is specialty hardware that must be used directly on the server (this has become very rare today, but does still exist from time to time) and extremely low latency systems where sub-millisecond latencies are critical.  The second of these is common only in extremely niche business situations such as low latency investment trading systems.  Systems with these requirements will also have incredible networking and geolocational requirements such as low-latency Infiniband with fiber to the trading floor of less than five miles.

Some people will point out that high performance computing clusters do not use virtualization, but this is a grey area as any form of clustering is, in fact, a form of virtualization.  It is simply that this is a “super-system” level of virtualization instead of being strictly at the system level.

It is safe to assume that any scenario in which you might find yourself in which you should not use virtualization you will know it beyond a shadow of a doubt and will be able to empirically demonstrate why virtualization is either physically or practically impossible.  For all other cases, virtualize.  Virtualize if you have only one physical server and one physically workload and just one user.  Virtualize if you are a Fortune 100 with the most demanding workloads.  And virtualize if you are anyone in between.  Size is not a factor in virtualization; we virtualize out of a desire to have a more effective and stable computing environment both today and into the future.

Today, four key RAID types exist for the majority of purposes: RAID 0, RAID 1, RAID 6 and RAID 10.  Each has a place where it makes the most sense.  RAID 1 and RAID 10, one simply being an application of the other, can handily be considered as a single RAID type with the only significant difference being the size of the array.  Many vendors refer to RAID 1 incorrectly as RAID 10 today because of this and, while this is clearly a semantic mistake, we will call them RAID 1/10 here to make decision making less complicated.  Together they can be considered the “mirrored RAID” family and the differentiation between them is based solely on the number of pairs in the array.  One pair is RAID 1, more than one pair is RAID 10.

RAID 0: RAID without redundancy.  RAID 0 is very fast and very fragile.  It has practically no overhead and requires the fewest hard disks in order to accomplish capacity and performance goals.  RAID 0 is perfect for situations where data is volatile (such as temporary caches) and where data is read only and there are solid backups and where accessibility is not a key concern.  RAID 0 should never be used for live or critical data.

RAID 6: RAID 6 is the market standard today for parity RAID, the successor to RAID 5.  As such, RAID 6 is cost effective in larger arrays (five drives minimum, normally six or more drives) where performance and reliability are secondary concerns to cost.  RAID 6 is focused on cost effective capacity for near-line data.

RAID 1/10: Mirrored RAID provides the best speed and reliability making it ideally suited for online data – any data where speed and reliability are of the top concern.  It is the only reasonable choice for arrays of four or fewer drives where the data is non-volatile.  With rare exception, mirrored RAID should be the defacto choice for any RAID array where specific technical needs do not clearly mandate a RAID 1 or RAID 6 solution.

It is a rare circumstance where RAID 0 is required, very rare.  RAID 6 has a place in many organizations but almost never on its own.  Almost every organization should be relying on RAID 1 or 10 for its primary storage and potentially using other RAID types for special cases, such as backups, archives and caches.  It is a very, very rare business that wouldn’t not have RAID 10 as the primary storage for the bulk of its systems.

I can see where this concern arises.  Virtualization allows for many guest operating systems to be contained in a single physical system which, in the event of a hardware failure, causes all guest systems residing on it to fail together, all at once.  This sounds bad, but perhaps it is not as bad as we would first presume.

The idea of the eggs and baskets idiom is that we should not put all of our resources at risk at the same time.  This is generally applied to investing, encouraging investors to diversify and invest in many different companies and types of securities like bonds, stocks, funds and commodities.  In the case of eggs (or money) we are talking about an interchangeable commodity.  One egg is as good as another.  A set of eggs are naturally redundant.

If we have a dozen eggs and we break six, we can still make an omelette, maybe a smaller one, but we can still eat.  Eating a smaller omelette is likely to be nearly as satisfying as a larger one – we are not going hungry in any case.  Putting our already redundant eggs into multiple baskets allows us to hedge our bets.  Yes, carrying two baskets means that we have less time to pay attention to either one so it increases the risk of losing some of the eggs but reduces the chances of losing all of the eggs.  In the case of eggs, a wise proposition indeed.  Likewise, a smart way to prepare for your retirement.

This theory, because it is repeated as an idiom without careful analysis or proper understanding, is then applied to unrelated areas such as server virtualization.  Servers, however, are not like eggs.  Servers, especially in smaller businesses, are rarely interchangeable commodities where having six working, instead of the usual twelve, is good enough.  Typically servers each play a unique role and all are relatively critical to the functioning of the business.  If a server is not critical then it is unlikely to be able to justify the cost of acquiring and maintaining itself in the first place and so would probably not exist.  When servers are interchangeable, such as in a large, stateless web farm or compute cluster, they are configured as such as a means to expanding capacity beyond the confines of a single, physical box and so fall outside the scope of this discussion.

IT services in a business are usually, at least to some degree, a “chain dependency.”  That is, they are interdependent and the loss of a single service may impact other services either because they are technically interdependent (such as a line of business application being dependent on a database) or because they are workflow interdependent (such as an office worker needing the file server working in order to provide a file which he needs to edit with information from an email while discussing the changes over the phone or instant messenger.)  In these cases, the loss of a single key service such as email, network authentication or file services may create a disproportionate loss of working ability.  If there are ten key services and one goes down, company productivity from an IT services perspective likely drops by far more than ten percent, possibly nearing one hundred percent in extreme cases.   This is not always true, in some unique cases workers are able to “work around” a lost service effectively, but this is very uncommon.  Even if people can remain working, they are likely far less productive than usual.

When dealing with physical servers, each server represents its own point of failure.  So if we have ten servers, we have ten times the likelihood of outage than if we had only one of those same servers.  Each server that we add brings with it its own risk.  If each failure has an outage factor of 2.5 – that is financially impacting the business for twenty five percent of revenue for, say, one day then our total average impact over a decade is the equivalent of two and a half total site outages.  I use the concept of factors and averages here to make this easy, determining the length of an average outage or impact of an average outage is not necessary as we only need to determine relative impact in this case to compare the scenarios.  It’s just a means of comparing cumulative outage financial impact of one event type compared to another without needing specific figures – this doesn’t help you determine what your spend should be, just relative reliability.

With virtualization we have the obvious ability to consolidate.  In this example we will assume that we can collapse all ten of these existing servers down into a single server.  When we do this we often trigger the “all our eggs in one basket” response.  But if we run some risk analysis we will see that this is usually just fear and uncertainty and not a mathematically supported risk.  If we assume the same risks as the example above our single server will, on average, incur just a single total site outage, once per decade.

Compare this to the first example which did the damage equivalent to two and a half total site outages – the risk of the virtualized, consolidated solution is only forty percent that of the traditional solution.

Now keep in mind that this is based on the assumption that losing some services means a financial loss greater than the strict value of the service that was lost, which is almost always the case.  Even if the service lost is no more than the loss of an individual service we are only at break even and need not worry.  In rare cases impact from losing a single system can be less than its “slice of the pie”, normally because people are flexible and can work around the failed system – like if instant messaging fails and people simple switch to using email until instant messaging is restore, but these cases are rare and are normally isolated to a few systems out of many with the majority of systems, say ERP, CRM and email, having disproportionally large impacts in the event of an outage.

So what we see here is that under normal circumstances moving ten services from ten servers to ten services on one server will generally lower our risk, not increase it – in direct contrast to the “eggs in a basket” theory.  And this is purely from a hardware failure perspective.  Consolidation offers several other important reliability factors, though, that can have a significant impact to our case study.

With consolidation we reduce the amount of hardware that needs to be monitored and managed by the IT department.  Fewer servers means that more time and attention can be paid to those that remain.  More attention means a better chance of catching issues early and more opportunity to keep parts on hand.  Better monitoring and maintenance leads to better reliability.

Possibly the most important factor, however, with consolidation is that there is significant cost savings and this, if approached correctly, can provide opportunities for improved reliability.  With the dramatic reduction in total cost for servers it can be tempting to continue to keep budgets tight and attempt to purely leverage the cost savings directly.   Understandable and for some businesses this may be the correct approach.  But it is not the approach that I would recommend when struggling against the notion of eggs and baskets.

Instead by applying a more moderate approach keeping significant cost savings but still spending more, relatively speaking, on a single server you can acquire a higher end (read: more reliable) server, use better parts, have on-site spares, etc.  The cost savings of virtualization can often be turned directly into increased reliability further shifting the equation in favor of the single server approach.

As I stated in another article, one brick house is more likely to survive a wind storm than either one or two straw houses.  Having more of something doesn’t necessarily make it the more reliable choice.

These benefits come purely from the consolidation aspect of virtualization and not from the virtualization itself.  Virtualization provides extended risk mitigation features separately as well.  System imaging and rapid restores, as well as restores to different hardware, are major advantages of most any virtualization platform.  This can play an important role in a disaster recovery strategy.

Of course, all of these concepts are purely to demonstrate that single box virtualization and consolidation can beat the legacy “one app to one server” approach and still save money – showing that the example of eggs and baskets is misleading and does not apply in this scenario.    There should be little trepidation in moving from a traditional environment directly to a virtualized one based on these factors.

It should be noted that virtualization can then extend the reliability of traditional commodity hardware providing mainframe-like failover features that are above and beyond what non-virtualized platforms are able to provide.  This moves commodity hardware more firmly into line with the larger, more expensive RISC platforms.  These features can bring an extreme level of protection but are often above and beyond what is appropriate for IT shops initially migrating from a non-failover, legacy hardware server environment.  High availability is a great feature but is often costly and very often unnecessary, especially as companies move from, as we have seen, relatively unreliable environments in the past to more reliable environments today.  Given that we have already increased reliability over what was considered necessary in the past there is a very good chance that an extreme jump in reliability is not needed now, but due to the large drop in the cost of high availability, it is quite possible that it will he cost justified where previously it could not be.

In the same vein, virtualization is often feared because it is seen as a new, unproven technology.  This is certainly untrue but there is an impression of this in the small business and commodity server space.  In reality, though, virtualization was first introduced by IBM in the 1960s and ever since then has been a mainstay of high end mainframe and RISC servers – those systems demanding the best reliability.  In the commodity server space virtualization was a larger technical challenge and took a very long time before it could be implemented efficiently enough to make it effective to use in the real world.  But even in the commodity server space virtualization has been available since the late 1990s and so is approximately fifteen years old today which is very far past the point of being a nascent technology – in the world of IT it is positively venerable.  Commodity platform virtualization is a mature field with several highly respected, extremely advanced vendors and products.  The use of virtualization as a standard for all or nearly all server applications is a long established and accepted “enterprise pattern” and one that now can easily be adopted by companies of any and every size.

Virtualization, perhaps counter-intuitively, is actually a very critical component of a reliability strategy.  Instead of adding risk, virtualization can almost be approached as a risk mitigation platform – a toolkit for increasing the reliability of your computing platforms through many avenues.

To simplify the list, RAID 0 will be left off of it.  RAID 0 is a viable choice for certain niche business scenarios in any count of drives.  So there is no need to display it on the list.  Also, the list assumes that a hot spare, if it exists, is not included in the count as that is “outside” of the RAID array and so would not be a part of the array drive count.

2 Drives: RAID 1

3 Drives: RAID 1 *

4 Drives: RAID 10

5 Drives: RAID 6

6 Drives: RAID 6 or RAID 10

7 Drives: RAID 6 or RAID 7

8 Drives: RAID 6 or RAID 7 or RAID 10 **

9 Drives: RAID 6 or RAID 7

10 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61

11 Drives: RAID 6 or RAID 7 

12 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61

13 Drives: RAID 6 or RAID 7

14 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61or RAID 70/71

15 Drives: RAID 6 or RAID 7 or RAID 60

16 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61 or RAID 70/71

17 Drives: RAID 6 or RAID 7

18 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61 or RAID 70/71

19 Drives: RAID 6 or RAID 7

20 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61 or RAID 70/71

21 Drives: RAID 6 or RAID 7 or RAID 60 or RAID 70

22 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61 or RAID 70/71

23 Drives: RAID 6 or RAID 7

24 Drives: RAID 6 or RAID 7 or RAID 10 or RAID 60/61 or RAID 70/71

25 Drives: RAID 6 or RAID 7 or RAID 60

………

* RAID 1 is technically viable at any drive count of two or more.  I have included it only up to three drives because using it beyond that point is generally considered absurd and is completely unheard of in the real world.  But technically it would continue to provide equal write performance while continuing to increase in read performance and reliability as more drives are added to the mirror.  But for reasons of practicality I have included it only twice on the list where it would actually be useful.

** At six drives and higher both RAID 6 and RAID 10 are viable options for arrays of even drive counts and RAID 6 alone is a viable option for odd numbered drive array counts.

For this list I have only considered the standard RAID levels of 0, 1, 4, 5, 6 and 10.  I left 0 off of the list because it is always viable for certain use cases.  RAID 5 never appears because there is no time on spindle hard drives today that it should be used, as RAID 5 is an enhancement of RAID 4, it too does not appear on the list.  Non-standard double parity RAID solutions such as Netapp’s RAID-DP and Oracle’s RAIDZ2 can be treated as derivations of RAID 6 and apply accordingly.  Oracle’s triple parity RAIDZ3 (sometimes called RAID 7) would apply at seven drives and higher but is a non-standard level and extremely rare so I included it in italics.

More commonly, RAID 6 makes sense at six drives or more and RAID 7 at eight drives or more.

Like RAID 4 and 5, RAID levels based on them (RAID 40, 50, 41, 51, 55, etc.) are not appropriate any longer due to the failure and fragility modes of spindle-based hard drives.  Complex RAID levels based on RAID 6 and 7 (60, 61, 70, 71, etc.) have a place but are exceedingly rare as they generally have very little cost savings compared to RAID 10 but suffer from performance issues and increased risk.  RAID 61 and 71 are almost exclusively effective when the highest order RAID, the mirror component, is over a network rather than local on the system.

In the small business space, where Intel and AMD architecture systems and Windows operating systems rule, hardware RAID is so common that a lot of confusion has arisen around software RAID due, as we will see, in no small part to the wealth of scam software RAID products touted as dedicated hardware and known colloquially as “Fake RAID.”

When RAID was first developed, it was used, in software, on high end enterprise servers running things like proprietary UNIX where the systems were extremely stable and the hardware was very powerful and robust making software RAID work very well.  Early RAID was primarily focused on mirrored RAID or very simplistic parity RAID (like RAID 2) which had little overhead.

As the need for RAID began to spill into the smaller server space and as parity RAID began to grow in popularity requiring greater processing power to support it became an issue that the underpowered processors in the x86 space were significantly impacted by the processing load of RAID, especially RAID 5.  This, combined with almost no operating systems heavily used on these platforms having software RAID implementations, lead to the natural development of hardware RAID – an offload processor board (similar to a GPU for graphics) that had its own complete computer on board with CPU and memory and firmware all of its own.

Hardware RAID worked very well at solving the RAID overhead problem in the x86 server space.  As CPUs gained more power and memory became less scarce popular x86 operating systems like Windows Server began to offer software RAID options.  Specifically Windows software RAID was known as a poor RAID implementation and was available only on server operating system versions causing a lack of appreciation for software RAID in the community of system administrators working primarily with Windows.

Because of historical implementations in the enterprise server space and the commodity x86 space there became a natural separation between the two markets supported initially by technology and later purely by ideology.  If you talk to a system administrator in the commodity space you will almost universally hear that hardware RAID is the only option.  Conversely if you talk to a system administrator in the mainframe, RISC (Sparc, Power, ARM) or EPIC (Itanium) server (sometimes called UNIX server) space you will often be met with surprise as hardware RAID isn’t available for those classes of systems – software RAID is simply a forgone conclusion.  Neither camp seems to have a real knowledge of the situation in the opposite one and crossovers in skill sets between these two is relatively rare until recently as enterprise UNIX platforms like Linux, Solaris and FreeBSD have started to become very popular and well understood on commodity hardware platforms.

To make matters more confusing for the commodity server space, in order to fill the vacuum left by the dominate operating system vendor’s lack of software RAID for the non-server operating system market while attempting to market to a less technically savvy target audience, a large number of vendors began selling non-RAID controller cards along with a “driver” that was actually software RAID and pretending that the resulting product was actually hardware RAID.  This created a large amount of confusion at best and an incredible disdain for software RAID at worse as almost universally any system whose core function is to protect data whose market is built upon deception and confusion will result in disaster.  Fake RAID systems routinely have issues with performance and reliability.  While, in theory, a third party software RAID package is a reasonable option, the reality of the software RAID market is that essentially all quality software RAID implementations are native components of either the operating system itself (Linux, Mac OSX, Solaris, Windows)  or of the filesystem (ZFS, VxFS, BtrFS) and are provided and maintained by primary vendors leaving little room or purpose for third party products outside of the Windows desktop space where a few, small legitimate software RAID players do exist but are often overshadowed by the Fake RAID players.

Today there is almost no need for hardware RAID as commodity platforms are incredibly powerful and there is almost always a dramatic excess of both computational and memory resources.  Hardware RAID instead competes mostly based on features rather than on reducing resource load.  Selection of hardware RAID versus software RAID in the commodity server space is almost completely one of preference and market momentum rather than of specific performance or features – both platforms essentially are equal with individual implementations being far more important in considering product options rather than hardware and software approaches are on their own.

Today hardware RAID offerings tend to be more “generic” with rather vanilla implementations of standard RAID levels.  Hardware RAID tends to earn its value through resource utilization reduction (CPU and memory offload), ability to “blind swap” failed drives, simplified storage management, block level storage agnostically abstracted from the operating system, fast cache close to the drives and battery or flash backed cache.  Software RAID tends to earn its value through lower power consumption, lower cost of acquisition, integrated management with the operating system, unique or advanced RAID features (such as ZFS’ RAIDZ that doesn’t suffer from the standard RAID 5 write hole) and generally better overall performance.  It is truly not a discussion of better or worse but of better or worse for a very specific situation with the most important factor often being familiarity and comfort and/or default vendor offering.

One of the most overlooked but important differentiators between hardware and software RAID is the change in the job role associated with RAID array management.  Hardware RAID moves the handling of the array to the server administrator (the support role that works on the physical server and is  stationed in the datacenter) whereas software RAID moves the handling of the array to the system administrator (the support role working on the operating system and above and rarely sitting in the datacenter.)  In the SMB market this factor might be completely overlooked but in a Fortune 500 the difference in job role can be very significant.  In many cases with hardware RAID disk replacements and system setup can be done without the need for system administrator intervention.  Datacenter server administrators can discover failed drives either through alerts or by looking for “amber lights” during walkthroughs and do replacements on the fly without needing to contact anyone or know what the server is even running.  Software RAID almost always would require the system administrator to be involved in managing the offlining of a failed disk, coordinating the replacement process with the datacenter and onlining the new one once the replacement process was completed.

Because of the way that CPU offloading and performance works and because of some advantages in the way that non-standard RAID implementations often handle parity RAID reconstruction there is a tendency for mirrored RAID levels to favor hardware RAID and software RAID levels to favor parity RAID.  Parity RAID is drastically more CPU intensive and so having access to the high power central CPU resources can be a major factor in speeding up RAID calculations.  But with mirrored RAID where RAID reconstruction is far safer than with parity RAID and where automated rebuilds are more important then hardware RAID brings the benefit of allowing blind drive replacement very easily.

One aspect of the hardware and software RAID discussion that is extremely paradoxical is that the same market that often dismisses software RAID out of hand as being inferior to hardware RAID is almost completely overlapping (you can picture the Venn Diagram in your head here) with the market that feels that file servers are inferior to commodity NAS appliances yet those NAS appliances in the SMB range are almost universally based on the same software RAID implementations being casually dismissed.  So it is often considered both inferior and superior simultaneously.  Some NAS devices in the SMB range, and NAS appliance software, that are software RAID based include: Netgear ReadyNAS, Netgear ReadyData, Buffalo Terastation, QNAP, Synology, OpenFiler FreeNAS, Nexenta and NAS4Free.

There is truly no “always use one way or the other” with hardware and software RAID.  Even giant, six figure enterprise NAS and SAN appliances are undecided as to which to use with part of the industry going each direction.  The real answer is it depends on your specific situation – your job role separation, your technical needs, your experience, your budget, etc.  Both options are completely viable in any organization.

In the software engineering community in recent years the concept of “You Aren’t Gonna Need It” or YAGNI has become a popular one.  YAGNI arose from the Extreme Programming (XP) group of Agile developers and is stated as this rule: “Always implement things when you actually need them, never when you just foresee that you need them.”

I like to rephrase YAGNI in development to “Don’t invest in something until you know that you need it.”  But the concept is the same – if you spend time and money building pieces that you aren’t sure that you will ever need you take on risks such as not getting value as early as possible (by focusing on the things that don’t matter yet while neglecting the things that do) and investing in technology that will never be used (because requirements change, project gets canceled, etc.)

This concept ports over to IT extremely well.  Design and purchasing are both heavily influences, or should be, by YAGNI.  Storage is a great example.  Don’t invest in storage today that you think that you will use tomorrow.  We can list a lot of reasons for why early storage investment is bad: the business has little to no ability to accurately predict its own growth, IT is poor at predicting storage growth based on business growth, the time-value of money and buying storage today is more costly than buying the same storage tomorrow.  Anytime that we buy based on predictions we take on risk.  Predictions rarely come true.

If we over buy storage today we are paying a premium for that storage because storage costs drop dramatically over time.  If we buy with 100% headroom and it takes three years or more before we use that headroom we are paying too much for the storage and getting older technology when buying later would give us better insight into what we actually need at that time (not just capacity but speed, reliability, features, etc.), lower cost and more options.

Overbuying is one risk, underbuying is another.  Underbuying is, obviously, less of a risk but still a concern.  If you buy today for needs three years out and at two years suddenly have a burst of need you may have overinvested in a platform or technology that cannot meet your needs.

Storage is one example but this can apply anywhere from software licenses, to CPU capacity, memory, high availability technologies even desktops.  Few shops would over buy desktops by a hundred percent just to be ready for a predicted head count increase in three years, but strangely they won’t hesitate doing it elsewhere.

By buying what is required for the immediate need and holding purchasing decisions until later there is a significant opportunity for cost savings and technology improvements.  In some cases it may be that the future need never arises whether because of bad predictions, changes in market or strategy or a change in technology direction either internally or externally.

Beyond purchasing, YAGNI can apply to network design.  It is not uncommon for large, complex designs to be proposed and implemented based on anticipated growth often years away and, to be honest, seldom very likely in a realistic world.  Building, for example, a complex high availability environment with expensive licensing, complex networking, lots of storage for an expected company growth in the future when just two servers and a nice backup plan is all that is cost justified today is dangerous.  Not only must the necessary growth happen to justify the IT spend but it must happen so quickly that the time-value of the money is justified and the cost of the technology does not drop so much as to have made implementing two systems more cost effective.  It is surprising how easily it can happen that putting in a smaller, stop-gap system and then implementing a larger scale system when needed can be far cheaper just because the cost of building the larger, more complex system has dropped in price so much since the first system was put in place and that is before taking into account the risk of bad predictions.

Spending early has an additional risk – it ties up corporate finances in unused architecture.  That money could be invested in other parts of the business in order to grow the business.  In extreme cases, overinvestment in infrastructure could be a contributor to a company failing completely – a self fulfilling situation where not using YAGNI in and of itself created the situation where YAGNI most applied.  The architected solution was never needed as the company failed.

YAGNI is a risk mitigation process.  Working with the needs that you know versus the needs that you anticipate.

Maybe IT shops over buy today because they are given specific budgets.  This is understandable that IT ends up in a technology grab attempting to implement whatever they can when the whims of the business smile upon them.  This, however, is an extremely poor business practice.  Businesses need to realize that large sums of money are being wasted on IT because IT is forced to implement systems with the assumption of clairvoyancey based on arbitrary budgets from the business with no basis in the real world.  IT is stuck buying what they can “sell” to the business based on often very unclear factors and the business often funds IT quite capriciously.  The creates a very unhealthy business and IT relationship where IT is wasting money because it has little choice and the business sees IT as a waste because they are not allowed to operate efficiently.

To fix this situation the business and IT need to work together.  IT needs to act more like a business-savvy unit and the business needs to lean on IT for guidance and not use prediction-based budgeting or become entangled in picking technological approaches without the technical understanding of the ramifications of those choices.  IT needs to be able to trust the business to make logical business financial decisions and the business needs to be able to trust IT to make logical technological decisions for the business.  The business drives IT, IT enables the business.  It is a symbiotic relationship.  If the business insists on making IT predict and operate on fixed budgets, IT will continue to be forced to overspend and overarchitect whenever possible in the hopes of being prepared for tomorrow when the budget may not be approved.  If IT was trusted to request what is needed and the business was trusted to fund technology needs at the appropriate time both could operate more effectively for the common good.

Takeaway: Don’t invest early, you don’t know what technology or the business will do tomorrow.

To really look at Windows 8 we need to look at the historic value of Windows as a desktop operation system.  For many years, even as far back as the late 1990s, Windows has competed against other desktop options on two fundamental premises.  The first premise is that moving to the next version of Windows is less disruptive and requires less retraining of staff than moving to a competitive platform allowing the majority of end users to remain comfortable and efficient even when going through major desktop upgrades.  The second is that the majority of business applications are written for Windows and moving to another platform severely limits application options.

Windows provides ancillary benefits of course such as a tight security model, well known support processes, massive user community, IT departments well versed in supporting it, excellent training and certification programs and good change processes.  But to a business selecting its next computing platform, continuity of usability and application support are the features traditionally driving the almost blind adoption of subsequent Windows versions year after year.

What makes Windows 8 unique in the long history of Windows desktop operating environments is that for the very first time even since the Windows 3.1 days there is a major change in the look, feel and usability of the desktop environment leaving many users stranded and confused in extreme cases and at least, in most cases, inefficient and frustrated.  Windows has never before departed from the basic need to make sure that users felt as little pain moving from version to version before and the need for retraining was basically out of the question beyond quick highlights showing where something has moved or showing off new features.  Windows 95 was the most extreme change of the past ~20+ years of Windows desktops and compared to Windows 8 it was relatively trivial.

With Windows 8 the move to the latest Windows edition is so dramatic that it begs comparison to the competition.  It is not that Windows 8 is bad, it is quite good, but that it doesn’t delivery the traditional Windows desktop upgrade value proposition either in user experience or in being a unique application platform as the majority of modern business applications are desktop platform agnostic running in the web browser leaving Windows in a very precarious position.  There are Linux desktops, for example, that offer user experiences far closer to Windows 7 than Windows 8 offers.  This, combined with the widespread use of enterprise web-based applications, means that, in theory, Windows 8 is no longer the simple upgrade path for desktops but it, in fact, the more difficult option requiring more training, more time and more pain for users and, from what we have seen, more long term loss of productivity as Windows 8 simply lacks the end-user efficiencies of most non-Windows platforms (Linux, Mac OSX and BSD.)

I’ve heard many people attempt to defend Windows but but the defense seems, to me, to be universally centered around mitigating Windows 8 flaws rather than finding places where it excels.  That users should not avoid it because they “haven’t taken time to learn how to deal with it”, that users should learn “keyboard shortcuts to make up for GUI deficiencies”, that they should “work hard to customize the Metro interface to make it less painful” or that “users should remove and/or replace troublesome Windows applications with more functional third party components” all, to me, sound like failures of the platform rather than reasons why Windows 8 is a good choice.  Yes, Windows 8 can certainly be made to be functional.  But Mac OSX or Linux Mint, as examples, solve all of these issues out of the box.  Users can hit the ground running and remain productive in the future.

From an IT support perspective there is a lot of pressure to maintain a status quo.  While Windows 8 is a departure it does not represent any significant change from supporting past Windows versions.  The tools and techniques are the same.  The set of experience and skills acquired for many years can be leveraged against Windows 8 and everyone comes to Windows 8 fresh so if there are new skills to be learned existing Windows desktop administrators and supporters are in the best position to learn them first.  Windows 8 continues to be the best job retention gamble and leverages best the in place support teams.  Moving to any new platform means that completely new skills and approaches need to be learned, new vendors need to be engaged and the risk of large portions of the department being replaced with outsides already possessing those skills looms large.

For end users, though, pressures might be the opposite.  IT needs to keep perspective that IT is not the end user of technology but the supplier of it.  The business and the business users are the end users of technology and it is the role of the IT department to support those needs.  If Windows 8 fails to deliver business value in comparison to competing options then it is IT’s job to deliver alternatives even if that means retraining for IT in order to make the business run more smoothly and more cost effectively.

When we step back and do a business by business analysis, Windows 8 is going to continue to dominate, there is no question.  But a shift is clear, Windows desktops are no longer the clear and obvious choice for end user easy of use and continued efficiency.  Microsoft is playing a dangerous game of alienating those whom they have courted for the longest.  Users looking for an easy transition will have to think twice about Windows 8 and the future of the Windows desktop.  Windows is already suffering from having lost the mobile and tablet space to the iOS and Android camps and has seen heavy market attrition in netbooks to Linux and the traditional desktop and laptop space to Mac OSX.  Windows’ areas of market dominance are growing fewer and those remaining are shrinking.  Ten years ago running a company without Windows on the desktop was unthinkable.  Today it is a very real consideration and both Mac OSX and many Linux distros have an opportunity to go through one or even several iterations before Windows 8′s replacement OS arrives from Microsoft giving them time to polish, advance and attract users who will be considering the Windows 8 move over the next few years.

Windows 8 fails to continue providing the Windows desktop’s traditional value.  Windows 8 fails to deliver new benefits to justify it on its own right.  Windows 8 fails to convince users and businesses of Microsoft long term vision.

One thing that I have found repeatedly creating a stumbling block for IT professionals is that they come from a stance of traditional, legacy knowledge (a natural situation since all of our knowledge has to have come from sometime in the past) and attempting to justify new techniques or technologies in relationship to the existing, established assumptions of “normal.”  This is to be expected.

IT is a field of change, however, and it is critical that IT professionals accept change as normal and not react to it as an undermining of traditional values.  It is not uncommon for people to feel that decisions that they have made in the past will be judged by the standards of today.  They feel that because there is a better option now that their old decision is somehow invalid or inadequate.  This is not the case.  This is exacerbated in IT because decisions made in the past that have been dramatically overturned in favour of new knowledge might only be a few years old and the people who made them still doing the same job.  Change in IT is much more rapid than in most fields and we can often feel betrayed by good decisions that we have made not long ago.

This reaction puts us into a natural, defensive position that we must rationally overcome in order to make objective decisions about our systems.

One trick that I have found is to reverse questions involved assumed norms.  That is to say, if you believe that you must justify a new technique against an old and find that while convincing you are not totally sways, perhaps you should try the opposite – justify the old, accepted approach versus the new one.  I will give some examples that I see in the real world regularly.

Example one, in which we consider virtualization where none existed before.  Typically someone looking to do this will look for virtualization to provide some benefit that they consider to be significant.  Generally this results in someone feeling that virtualization either doesn’t offer adequate benefits or that they must incorporate other changes and end up going dramatically overboard for what should have been a smaller decision.  Instead, attempt to justify not using virtualization.  Treat virtualization as the accepted pattern (actually, it long has been, just not the in SMB space) and try to justify going with physical servers instead.

What we find is that, normally, our minds accepted that the physical machine only had to be “nearly as good” or “acceptable” in order to be chosen even though virtualization was, in nearly all cases, “better”.  Why would be decide to use something that is not “better”?  Because we approached one as change and one as not change.  Our minds play tricks on us.

Example two, in which traditional server storage is two arrays with the operating system on one RAID 1 array and the data partition on a second RAID 5 array versus the new standard of a single RAID 10 array holding both operating system and data.  If we argue from the aspect of the traditional approach we can make decent arguments, at times, that we can make the old system adequate for our needs.  Adequate seems good enough to not change our approach.  But argue from the other direction.  If we assume RAID 10 is the established, accepted norm (again, it is today) then it is clear that it comes out as dramatically superior in nearly all scenarios.  If we try to justify why we would chose a split array with RAID 1 and RAID 5 we would quickly see that they never provide a compelling value.  So sticking with RAID 10 is a clear win.

This reversal of thinking can provide for a dramatic, eye-opening effect on decision making.  Making assumptions about starting points and forcing new ideas to significantly “unseat” incumbent thinking is dangerous.  This keeps us from moving forward.  In reality, most approaches should start from equal ground and the “best” option should win.  It is far too often than a solution is considered “adequate” when it is not the best.  Yes, a solution may very well work in a given situation but why would we ever intentionally choose a less than superior solution (we assume that cost is factored into the definition of best?)

As IT professionals attempting to solve problems for a business we should be striving to recommend and implement the best possible solutions, but making due with less than ideal ones simply because we forget to equally consider the reasonable options against one another.  And it is important to remember that cost is inclusive in deciding when a solution is best or adequate.  The best solution is not a perfect solution but the best for the company, for the money.  But very often solutions are chosen that cost more and do less simply because they are considering the de facto starting point and the alternatives are expected to dramatically outperform them rather than simply being “better”.

Taking a fresh look at decision making can help us become better professionals.

The quick rule of thumb for storage should be: Local before DAS, DAS before NAS, NAS before SAN.  Or as I like to write it:

Local Disks -> DAS -> NAS -> SAN

To use this rule you simply start with your storage requirements in hand and begin on the left hand side.  If local disks meet your requirements, then almost certainly they are your best choice.  If they don’t meet your requirements move to the right and check if DAS will meet your requirements.  If so, great, if not continue the process.

That’s the rule of thumb, so if that is all you need, there you go.  But we will dive into the “why” of the rule below. The quick overview is that on the left we get speed and reliability at the lowest cost.  As we move to the right complexity increases as does price typically.  The last two, while very different, are actually the most alike in many ways due to their networked nature.

Local Disks:  Local drives inside your server chassis are your best bet for most tasks.  Being inside the chassis means the least amount of money spent on extra containers to hold and power the drives, least physical risk, most solid connection technologies, shortest distance and least amount of potential bottlenecks. Being raw disks, local disks are block devices.

Direct Attached Storage:  DAS is, more or less, local drives housed outside of the server chassis.  The server itself will see them exactly like any other local drives making them very easy to use.  DAS is simple but still has extra external containers and extra cables.  This adds cost and some complexity.  DAS makes it easier to attach multiple servers to the same set of drives as this is almost impossible, and always cumbersome, with local disks.  So DAS is effectively our first type of physically sharable storage.  Being identical to local disks, DAS is a form of block device.

Network Attached Storage: NAS is unique in that it is the only non-block device from which we have to choose.  A NAS, or a traditional file server – they are truly one and the same, is the first of our technologies designed to run over a network.  This adds a lot of complication.  NAS shares storage out at the filesystem level.  A NAS is an intelligent device that allows users over the network to easily and safely share storage because the NAS has the necessary logic on board to handle multiple users at one time.  NAS is very easy for anyone to use and is even commonly used by people at home.

Storage Area Network: SAN is an adaptation of DAS with the addition of a network infrastructure allowing the SAN to behave as a remote hard drive (block device) that an operating system sees as no different from any other hard drive attached to it.  SANs require advanced networking knowledge, are surrounded by a large amount of myth and rumor, are poorly understood by the average IT professional, are generally complex to use and understand and because they lack the logic of a NAS they effectively expose a hard drive directly to the network making it trivially easy to corrupt and destroy data.  It is, in fact, so easy to lose data on a SAN due to misconfiguration that the most commonly expected use of a SAN is a use case for which a SAN cannot be used.

Of course there is much grey area.  What is normally considered a DAS can be turned into a SAN.  A SAN can be direct connected.  NAS can be direct connected.  Local storage can act as either NAS or SAN depending on configuration such as with a VSA (Virtual Storage Appliance.)  Many devices are simultaneously NAS and SAN and the determination is by configuration, not by the physical device itself.  But in generally accepted use, the terms are mostly straightforward.

The point being that as we move from left to right in our list we move from simple and easy to difficult and complex.  SAN itself is a rock solid technology; it is the introduction of humans and their tendency to do dangerous things easily with SAN that makes it a dangerous storage technique for the average user.  As with everything in IT, keeping our technologies and processes simple brings stability and security and, often, cost savings as well.

There are many times when movement to “the right” is necessary.  Local disks do not scale well and can become too expensive to maintain for certain types of larger deployments.  DAS, likewise, doesn’t scale well in many cases.  NAS scales well but being a non-block protocol is a bit unique and doesn’t always work for our purposes, a good example being HyperV that requires a block device for storage.  SAN is the final catchall of storage.  If nothing else works, SAN is always there to fall back on – or, as I like to say, SAN is the storage of last resort.

This is a very high level look at the basics of choosing a storage approach.  This is a common IT task that must be done with great regularity.  I did not intend this post, in any way, to explain any deep knowledge of storage but simply to provide a handy guide to understanding where to start looking at storage options.  Exceptions and special cases abound, but it is extremely common to simply skip the best option and go straight to considering something big, expensive and complex and rapidly forget that something much more simple might do the same job in a far superior manner.  The underlying concept is the simplest solution that meets the need is usually the best.

Is this really the case?  In a world where security and control of data are becoming increasingly regulated and exposed and as the public takes a growing interest in how companies are securing their data it is safe to assume that the movement of the IT field is not going to be towards a loss of control.  And, in my experience, BYOD means exactly the opposite.

There is no ignoring the fact that BYOD signals many changes and demands IT departments rethink traditional approaches.  But is that such a bad thing?  The old model was one of a network castle.  The firewalls were the moat and all of our devices from servers to desktops sat huddled together inside the castle courtyard talking freely one to another.  One of the greatest fears was that one of those desktops were to become “compromised” and would unleash a fifth column attack from within the castle where there were practically no defenses of which to speak.

The old model created a quagmire of issues and required complicated workarounds in order to accommodate modern changes in computing environments.  When businesses existed in only a single location or when businesses would regularly purchase leased lines connecting all of their offices the model worked rather well.  Once workers began to need to work remotely, whether at home or when on the road, the model became difficult to support and the concept of VPNs were introduced in order to extend the castle wherever it was needed.  VPNs changed how companies could physically exist but did so without addressing some fundamental issues with the architecture of a traditional IT infrastructure.

The solution to this infrastructure reinvention has been coming for a long time now.  The movement towards web applications, “cloud services”, hosted applications, Software as a Service and other terms for the new ways in which people were thinking about applications.  Slowly we started exposing applications to the “outside”.  We started simply with email, then basic web applications and slowly more and more components of business infrastructure start to be exposed externally without requiring the use of a VPN.

The advent of smartphones accelerated this process as certain applications, email and calendaring being the biggest drivers, absolutely demanded extension to these mobile devices.  For the most part, IT departments did not even see a significant shift occurring.  Instead it was little pinholes, small changes as more and more of the tools used in the business were available without connecting to the VPN, without sitting inside the office.

Today a new business might legitimately ask its CIO: “Why do we even need a LAN?  What benefit do we get from everyone sitting on a single, physical network?”  There are still plenty of good reasons why a LAN might be needed.  But it is a valuable question to ask and the answer might surprise you.  I was asked this myself and the answer was that we didn’t need a LAN, every app was available through its own, secure channel, without a need for VPNs or a local network.

Where LANs continue to shine brightest is in desktop management.  If you need to lock down and control the actual end user equipment then LANs work their best here – currently.  This too will change in time.  But this is where BYOD becomes the secret weapon of the IT department.

BYOD, while creating its own raft of obvious complications, especially around end user support expected after decades of total IT control of end user devices, offers the opportunity to eliminate the LAN, pull back the walls of the castle to surround only the core infrastructure where no end user ever need venture and to drop the support of end users devices solidly into the lap of the end users themselves.  With modern LAN-less application publishing strategies (this includes web apps, remote desktop technologies and others) end user devices are effectively thin clients often providing no more processing capacity than is necessary to display the application.  They are a window into the infrastructure, not a gateway.  They look at the servers, they aren’t sitting inside the castle with them.

Thinking of end user devices as view panels or windows rather than computing devices is the key to making BYOD an advantage to the IT department rather than its bane.  Of course, this plays into the usual ebb and flow and fat and thin clients over the history of computing.  The tide will change again, but for now, this is our current opportunity.  End users want the illusion of control and the reality of picking the device that is best suited to their needs – which are almost strictly physical needs whether of fashion or function.  IT departments want the reality of control and should be happy to allow end users to pick their own devices.  Everyone can win.

The key, of course, is eliminating legacy applications or finding workarounds.  Technological approaches such as VDI, terminal servers or even racks of datacenter-housed desktops potentially provide fallback strategies that can be accessed from nearly any device while “view” layer technologies like HTML 5 look to provide elegant, modern options for exposing applications, shifting display-related processing to the end user device and standardizing on a protocol that is likely to exist ubiquitously in the very near future.  The technologies are there today.

With the corporate network shrunk down to being only the infrastructure servers and associated networking gear suddenly IT departments have the potential for greater control and more flexibility while giving up little.  End users are happy, IT is happy.  BYOD is an opportunity for IT to exert greater control, tighter security all while giving the impression of being approachable and flexible.

What is important is that Windows comes out in a flip-flop fashion with every other release being a “long term support, heavily stable” release and the alternate releases being the “new technology preview” releases.  This is not to say that any particular release is good or bad, but that one release is based around introducing a new system to the public and the next is a more polished release with fewer changes than its predecessor focused on long term adoption.

The goal of this release pattern should be obvious.  Whenever major changes from to such a widely used platform the average user, even the average IT professional, tends to resist the change and be unhappy with it.  But after a while the new look, feel and features start to feel natural.  Then a slightly updated, slightly more polished version of the same features can be released and the general public feels like Microsoft has “learned its lesson” and they appreciate the same features that they disliked a few years before.  This approach works wonders in Microsoft’s mixed consumer and business world where they get home users to adopt the latest and greatest at home with OEM licenses bundled with the computers that they buy and businesses can, and usually do, wait for the “every other” cycle to allow them to utilize only the more mature of the two releases to their users who have already lived through the pain of the changes at home.

Outside of the Windows world you can witness the same sort of adoption with the much maligned MS Office 2007 and MS Office 2010.  The former was universally hated because of the then new Ribbon interface.  The later was much loved mostly because people had already adapted to the Ribbon interface and now appreciated it but also because Microsoft had time to learn from the 2007 release and tweak the Ribbon to be improved by 2010.

This pattern started long ago and can be seen happening, to some degree, even in the DOS-based Windows era (the Windows family starting from the very beginning and running up through Windows ME.)  Of the more recent family members Windows 3 was the preview, Windows 3.1 was the long term release, Windows 95 was the preview, Windows 98 the long term release and Windows ME was the preview.  Each one of the previews had poor reception, comparatively, due to the introduction of new ideas and interfaces.  Each of the long term releases outlived its counterpart preview release on the market and were widely loved.  It is a successful pattern.

In the modern era of Windows NT, starting with Windows NT 3.1 in 1993, the overarching pattern continued with NT 3.1 itself being the “preview” member of the new Windows NT family.  Just one year later Windows NT 3.5 released and was popular for its time.  Windows NT 3.51 came out and provided the first support for the new world of interoperability with Windows 95 from the DOS family which released just a few months after NT 3.51 itself did.  Then the stable, long term Windows NT 4 released in 1996 and dominated the Windows world for the next half decade.  Windows NT 4 leveraged both the cycle from the Windows NT family as well as the cycle from the DOS/Windows family to great effect.

In 2000 when Windows 2000 released it was a dramatic shift for the Windows NT family and was poorly received.  The changes, both to the desktop and the coinciding Server product with the introduction of Active Directory were massive and disruptive.  Windows 2000 was the quintessential preview release.  It took just one year before Windows XP replaced it on the desktop.  Windows XP, per its place in the cycle, turned out to be the quintessential long term release making even Windows NT 4 look short lived.  Windows XP expanded very little on Windows 2000 Workstation but it brought additional polish and no significant changes making it exactly what businesses and most home users, were looking for as their main operating system for a very long time.

When Microsoft was ready to disrupt the desktop again with new changes, like the additional security of UAC, they did so in Windows Vista.  Vista, like Windows 2000, was not well received and was possibly the most hated Windows release of all time.  But Vista did its job perfectly.  Shortly after the release of Windows Vista came the nominally different Windows 7 with some minor UAC changes and some improved polish and was very well received.  Vista paved the way so that Windows 7 could be loved and used for many years.

Now we stand on the verge of the Windows 8 release.  Like Vista, 2000, Office 2007 and Windows 95, Windows 8 represents a dramatic departure for the platform and already, before even being released, has generated massive amounts of bad press and animosity.  If we study the history of the platform, though, we would have expected this in the Windows 8 release regardless of what changes were going to be announced.  Windows 8 is the “preview” release.  We know that a new operating system, perhaps called Windows 9, is at most two years away and will bring a slightly tweaked, more polished version of Windows 8 that end users will love and the issues with Windows 8, like its predecessors, will soon be forgotten.  The cycle is well established and very successful.  There is very little chance that it will be changing anytime soon.

First, I’d like to address what I personally feel is a mistake in the naming conventions. What we refer to as a hot spare should, I believe, actually be called a warm spare because it is sitting there ready to go but does not contain the necessary data to be used immediately.  A spare drive stored outside of the chassis, one that requires a human to step in and swap the drives manually, would be a cold spare.  To truly be a hot spare a drive should be full of data and, therefore, would be a participatory member of the RAID array in some capacity.  Red Hat has a good article on how this terminology applies to disaster recovery sites for reference.  This differentiation is important because what we call a hot spare does not already contain data and does not immediately step in to replace the failed drive but instead steps in to immediately begin the process of restoring the lost drive – a critical differentiation.

In order to keep concepts clear, from here on out I will refer to what vendors call hot spares as “warm spares.”  This will make sense in short order.

There are two main concerns with warm spares.  The first is the ineffectual nature of the warm spare in most use cases and the second is the “automated array destruction” risk.

Most people approach the warm spare concept as a means of mitigating the high risk of secondary drive failure on a parity RAID 5 array.  RAID 5 arrays protect only against the failure of a single disk within the array.  Once a single disk has failed the array is left with no form of parity and any additional drive failure results in the total loss of the array.  RAID 5 is chosen because it is very low cost for the given capacity and sacrifices reliability in order to achieve this cost effectiveness.   Because RAID 5 is therefore risky in comparison to other RAID options, such as RAID 6 or RAID 10, it is common to implement a warm spare in order to minimize the time that the array is left in a degraded state allowing the array to begin resilvering itself as quickly as possible.

So the takeaway here that is more relevant is that warm spares are generally used as a buffer against using less reliable RAID array types as a cost saving measure.  Warm spares are dramatically more common in RAID 5 arrays followed by RAID 6 arrays.  Both of which are chosen over RAID 10 due to cost for capacity, not for reliability or performance.  There is one case where the warm spare idea truly does make sense for added reliability, and that is in RAID 10 with a warm spare, but we will come to that.  Outside of that scenario I feel that warm spares make little sense in the real world.

We will start by examining RAID 1 with a warm spare.  RAID 1 consists of two drives, or more, in a mirror.  Adding a warm spare is nice in that if one of the mirrored pairs dies the warm spare will immediately begin mirroring the remaining drive and you will be protected again in short order.  That is wonderful.  Except for one minor flaw, instead of using a warm spare that same drive could have been added to the RAID 1 array all along where it would have been a tertiary mirror.  In this tertiary mirror capacity the drive would have added to the overall performance of the array giving a nearly fifty percent read performance boost with write performance staying level and providing instant protection in case of a drive failure rather than “as soon as it remirrors” protection.  Basically it would have been a true “hot spare” rather than a warm spare.  So without spending a penny more the system would have had better drive array performance and better reliability simply by having the extra drive in a hot “in the array” capacity rather than sitting warm and idle waiting for disaster to strike.

With RAID 5 we see an even more dramatic warning against the warm spare concept, here where it is more common than anywhere else.  RAID 5 is single parity RAID with the ability to rebuild, using the parity, any drive in the array that fails.  This is where the real problems begin.  Unlike in RAID 1 where a remirroring operation might be quite quick, a RAID 5 resilver (rebuild) has the potential to take quite a long time.  The warm spare will not assist in protecting the array until this resilver process completes successfully – this is commonly many hours and is easily days and possibly weeks or months depending on the size of the array and how busy the array is.  If we took that same warm spare drive and instead tasked it with being a member of the array with an additional parity stripe we would achieve RAID 6.  The same set of drives that we have for RAID 5 plus warm spare would create a RAID 6 array of the exact same capacity.  Again, like the RAID 1 example above, this would be much like having a hot spare, where the drive is participating in the array with live data rather than sitting idly by waiting for another drive to fail before kicking in to begin the process of taking over.  In this capacity the array degrades to a RAID 5 equivalent in case of a failure but without any rebuild time, so the additional drive is useful immediately rather than only after a possible very lengthy resilver process.  So for the same money, same capacity the choice of setting up the drives in RAID 6 rather than RAID 5 plus warm spare is a complete win.

We can continue this example with RAID 6 plus warm spare.  This one is a little less easy to define because in most RAID systems, except for the somewhat uncommon RAIDZ3 from ZFS, there is no triple parity system available one step above RAID 6 (imagine if there was a RAID 7, for example.)  If there were the exact argument made for RAID 5 plus warm spare would apply to RAID 6 plus warm spare.  In the majority of cases RAID 6 with a warm spare must justify itself against a RAID 10 array.  RAID 10 is more performant and far more reliable than a RAID 6 array but RAID 6 is generally chosen to save money in comparison to RAID 10.  But to offset RAID 6′s fragility warm spares are sometimes employed.  In some cases, such as a small five disk RAID 6 array with a warm spare, this is dollar for dollar equivalent to a six disk RAID 10 array without a warm spare.  In larger arrays the cost benefit of RAID 6 does become apparent but the larger the cost savings the larger the risk differential as parity RAID systems increase risk with array size much more quickly than do mirror based RAID systems like RAID 10.  Any money saved today is done at the risk of outage or data loss tomorrow.

Where a warm spare comes into play effectively is in a RAID 10 array where a warm spare rebuild is a mirror rebuild, like in RAID 1, which does not carry parity risks, where there is no logical extension RAID system above RAID 10 from which we are trying to save money by going with a more fragile system.  Here adding a warm spare may make sense for critical arrays because there is no more cost effective way to gain the same additional reliability.  However, RAID 10 is so reliable without a warm spare that any shop contemplating RAID 5 or RAID 6 with a warm spare would logically stop at simple RAID 10 having already surpassed the reliability they were considering settling for previously.  So only shops not considering those more fragile systems and looking for the most robust possible option would logically look to RAID 10 plus warm spare as their solution.

Just for technical accuracy, RAID 10 can be expanded for better read performance and dramatic improvement in reliability (but with a fifty percent cost increase) by moving to three disk RAID 1 mirrors in its RAID 0 stripe rather than standard two disk RAID 1 mirrors just like we showed in our RAID 1 example.  This is a level of reliability seldom sought in the real world but can exist and is an option.  Normally this is curtailed by drive count limitations in physical array chassis as well as competing poorly against building a completely separate secondary RAID 10 array in a different chassis and then mirroring these at a high level effectively created RAID 101 – which is the effective result of common, high end storage array clusters today.

Our second concern is that of “automated array destruction.”  This applies only to the parity RAID scenarios of RAID 5 and RAID 6 (or the rare RAID 2, RAID 3, RAID 4 and RAIDZ3.)  With the warm spare concept, the idea is that when a drive fails the warm spare is automatically and instantly swapped in by the array controller and the process of resilvering the array begins immediately.  If resilvering was a completely reliable process this would be obviouslyd highly welcomed.  The reality is, sadly, quite different.

During a resilver process a parity RAID array is at risk of Unrecoverable Read Errors (UREs) cropping up.  If a URE occurs in a single parity RAID resilver (that is RAID 2 –  5) then the resilvering process fails and the array is lost completely.  This is critical to understand because no additional drive has failed.  So if the warm spare had not been present then the resilvering would have not commenced and the data would still be intact and available – just not as quickly as usual and at the small risk of secondary drive failure.  URE rates are very high with today’s large drives and with large arrays the risks can become so high as to move from “possible” to “expected” during a standard resilvering operation.

So in many cases the warm spare itself might actually be the trigger for the loss of data rather than the savior of the data as expected.  An array that would have survived might be destroyed by the resilvering process before the human who manages it is even alerted to the first drive having failed.  Had a human been involved they could have, at the very least, taken the step to make a fresh backup of the array before kicking off the resilver knowing that the latest copy of the data would be available in case the resilver process was unsuccessful.  It would also allow the human to schedule when the resilver should begin, possibly waiting until business hours are over or the weekend has begun when the array is less likely to experience heavy load.

Dual and triple parity RAID (RAID 6 and RAIDZ3 respectively) share URE risks as well as they too are based on parity.  They mitigate this risk through the additional levels of parity and do so successfully for the most part.  The risk still exists, especially in very large RAID 6 arrays, but for the next several years the risks remain generally quite low for the majority of storage arrays until far larger spindle-based storage media is available on the market.

The biggest problem with parity RAID and the URE risk is that the driver towards parity RAID (willing to face additional data integrity risks in order to lower cost) is the same driver that introduces heightened URE risk (purchasing lower cost, non-enterprise SATA hard drives.)  Shops facing parity RAID generally do so with large, low cost SATA drives bringing two very dangerous factors together for an explosive combination.  Using non-parity RAID 1 or RAID 10 will completely eliminate the issue and using highly reliable enterprise SAS drives will drastically reduce the risk factor by an order of magnitude (not an expression, it is actually a change of one order of magnitude.)

Additionally during resilver operations it is possible for performance to degrade on parity systems so drastically as to equate to a long-term outage.  The resilver process, especially on large arrays, can be so intensive that end users cannot differentiate between a completely failed array and a resilvering array.  In fact, resilvering at its extreme can take so long and be so disruptive that the cost to the business can be higher than if the array had simply failed completely and a restore from backup had been done instead.  This resilver issue does not affect RAID 1 and RAID 10, again, because they are mirrored, not parity, RAID systems and their resilver process is trivial and the performance degradation of the system is minimal and short lived.  At its most extreme, a parity resilver could take weeks or months during which time the systems act as though they are offline – and at any point during this process there is the potential for the URE errors to arise as mentioned above which would end the resilver and force the restore from backup anyway.  (Typical resilvers do not take weeks but do take many hours and to take days is not at all uncommon.)

Our final overview can be broken down to the following (conventional term “hot spare” used again): RAID 10 without a “hot spare” is almost always a better choice than RAID 6 with a “hot spare.”  RAID 6 without a “hot spare” is always better than RAID 5 with a “hot spare.”  RAID 1 with additional mirror member is always better than RAID 1 with a “hot spare.”  So whatever RAID level with a hot spare you decide upon, simply move up one level of RAID reliability and drop the “hot spare” to maximize both performance and reliability for equal or nearly equal cost.

Warm spares, like parity RAID, had they day in the sun.  In fact it was when parity RAID still made sense for widespread use – when URE errors were unlikely and disk costs were high – that warm spare drives made sense as well.  They were well paired, when one made sense the other often did too.  What is often overlooked is that as parity RAID, especially RAID 5, has lost effectiveness it has pulled the warm spare along with it in unexpected ways.

To me the compelling story of Windows 8 comes not in the fit and finish but the under the hood rewiring that hint at a promising new future for the platform.  In the past Microsoft has attempted shipping Windows Server OS on some alternative architectures including, for those who remember, the Digital Alpha processor and more recently the Intel Itanium.  In these previous cases, the focus was on the highest end Microsoft platforms being run on hardware above and beyond what the Windows world normally sees.

Windows 8 promises to tackle the world of multiple architectures in a completely different way – starting with the lowest end operating system and focusing on a platform that is lighter and less powerful than the typical Intel or AMD offering, the low power ARM RISC architecture with the newly named Windows RT (previously WoA, Windows on ARM.)

The ARM architecture is making its headlines as Microsoft attempts to drive deep into handheld and low power devices.  Windows RT could signal a unification between the Windows desktop codebase and the mobile smartphone codebase down the road.  Windows RT could mean strong competition from Microsoft in the handheld tablet market where the iPad dominates so completely today.  Windows RT could be a real competitor to the Android platforms.

Certainly, as it stands today, Windows RT has a lot of potential to be really interesting, if not quite disruptive, with where it will stand upon release.  But I think that the interesting story lies beneath the surface in what Windows RT can potentially mean for the datacenter.  What might Microsoft have in store for us in the future?

The datacenter today is moving in many directions.  Virtualization is one driving factor as are low power server options such as Hewlett-Packard’s Project Moonshot which is designed to bring ARM-based, low power consumption servers into high end, horizontally scaling datacenter applications.

Currently, today, the number of server operating systems available to run on ARM servers, like those coming soon from HP, are few and far between and are mostly only available from the BSD family of operating systems.  The Linux community, for example, is scrambled to assemble even a single, enterprise-supported ARM-based distribution and it appears that Ubuntu will be the first out of the gate there.  But this paucity of server operating systems on ARM leaves an obvious market gap and one that Microsoft may be well thinking of filling.

Windows Server on ARM could be a big win for Microsoft in the datacenter.  A lower cost offering broadening their platform portfolio without the need for heavy kernel reworking since they are already providing this effort for the kernel on their handheld devices.  This could be a significant push for Windows into the growingly popular green datacenter arena where ARM processors are expected to play a central role.

Microsoft has long fought to gain a foothold in the datacenter and today is as comfortable there as anyone but Windows Servers continue to play in a segregated world where email, authentication and some internal applications are housed on Windows platforms but the majority of heavy processing, web hosting, storage and other roles are almost universally given to UNIX family members.  Windows’ availability on the ARM platform could push it to the forefront of options for horizontally scaling server forms like web servers, application servers and other tasks which will rise to the top of the ARM computing pool – possibly even green high performance compute grids.

ARM might mean exciting things for the future of the Windows Server platform, probably at least one, if not two releases out.  And, likewise, Windows might mean something exciting for ARM.

RAID is a great example of where a lack of holistic risk thinking can lead to some strange decision making.  If we look at a not uncommon scenario we will see where the goal of protecting against drive failure can actually lead to an increase in risk even when additional redundancy is applied.  In this scenario we will compare a twelve drive array consisting of twelve three terabyte SATA hard drives in a single array.  It is not uncommon to hear of people choosing RAID 5 for this scenario to get “maximum capacity and performance” while having “adequate protection against failure.”

The idea here is that RAID 5 protects against the loss of a single drive which can be replaced and the array will rebuild itself before a second drive fails.  That is great in theory, but the real risks of an array of this size, thirty six terabytes of drive capacity, come not from multiple drive failures as people generally suspect but from an inability to reliably rebuild the array after a single drive failure or from a failure of the array itself with no individual drives failing.  The risk of a second drive failing is low, not non-existent, but quite low.  Drives today are highly reliable. Once one drives fails it does increase the likelihood of a second drive failing, which is well documented, but I don’t want this risk to mislead us from looking at the true risks – the risk of a failed resilvering operation.

What happens that scares us during a RAID 5 resilver operation is that an unrecoverable read error (URE) can occur.  When it does the resilver operation halts and the array is left in a useless state – all data on the array is lost.  On common SATA drives the rate of URE is 10^14, or once every twelve terabytes of read operations.  That means that a six terabyte array being resilvered has a roughly fifty percent chance of hitting a URE and failing.  Fifty percent chance of failure is insanely high.  Imagine if your car had a fifty percent chance of the wheels falling off every time that you drove it.  So with a small (by today’s standards) six terabyte RAID 5 array using 10^14 URE SATA drives, if we were to lose a single drive, we have only a fifty percent chance that the array will recover assuming the drive is replaced immediately.  That doesn’t include the risk of a second drive failing, only the risk of a URE failure.  It also assumes that the drive is completely idle other than the resilver operation.  If the drives are busily being used for other tasks at the same time then the chances of something bad happening, either a URE or a second drive failure, begin to increase dramatically.

With a twelve terabyte array the chances of complete data loss during a resilver operation begin to approach one hundred percent – meaning that RAID 5 has no functionality whatsoever in that case.  There is always a chance of survival, but it is very low.  At six terabytes you can compare a resilver operation to a game of Russian roulette with one bullet and six chambers and you have to pull the trigger three times.  With twelve terabytes you have to pull it six times!  Those are not good odds.

But we are not talking about a twelve terabyte array.  We are talking about a thirty six terabyte array – which sounds large but this is a size that someone could easily have at home today, let alone in a business.  Every major server manufacturer, as well as nearly all low cost storage vendors, make sub $10,000 storage systems in this capacity range today.  Resilvering a RAID 5 array with a single drive failure on a thirty six terabyte array is like playing Russian roulette, one bullet, six chambers and pulling the trigger eighteen times!  Your data doesn’t stand much of a chance.  Add to that the incredible amount of time needed to resilver an array of that size and the risk of a second disk failing during that resilver window starts to become a rather significant threat.  I’ve seen estimates of resilver times climbing into weeks or months on some systems.  That is a long time to run without being able to lose another drive.  When we are talking hours or days the risks are pretty low, but still present.  When we are talking weeks or months of continuous abuse, as resilver operations are extremely drive intensive, the failure rates climb dramatically.

With an array of this size we can effectively assume that the loss of a single drive means the loss of the complete array leaving us with no drive failure protection at all.  Now if we look at a drive of the same or better performance with the same or better capacity under RAID 0, which also has no protection against drive loss, we need only use eleven of the same drives that we needed twelve of for our RAID 5 array.  What this means is that instead of twelve hard drives, each of which has a roughly three percent chance of annual failure, we have only eleven.  That alone makes our RAID 0 array more reliable as there are fewer drives to fail.  Not only do we have fewer drives but there is no need to write the parity block nor skip parity blocks when reading back lowering, ever so slightly, the mechanical wear and tear on the RAID 0 array for the same utilization giving it a very slight additional reliability edge.  The RAID 0 array of eleven drives will be identical in capacity to the twelve drive RAID 5 array but will have slightly better throughput and latency.  A win all around.  Plus the cost savings of not needing an additional drive.

So what we see here is that in large arrays (large in capacity, not in spindle count) that RAID 0 actually passes RAID 5 in certain scenarios.  When using common SATA drives this happens at capacities experienced even by power users at home and by many small businesses.  If we move to enterprise SATA drives or SAS drives then the capacity number where this occurs becomes very high and is not a concern today but will be in just a few years when drive capacities get larger still.  But this highlights how dangerous RAID 5 is in the sizes that we see today.  Everyone understands the incredible risks of RAID 0 but it can be difficult to put into perspective that RAID 5′s issues are so extreme that it might actually be less reliable than RAID 0.

That RAID 5 might be less reliable than RAID 0 in an array of this size based on resilver operations alone is just the beginning.  In a massive array like this the resilver time can take so long and exact such a toll on the drives that second drive failure starts to become a measurable risk as well.  And then there are additional risks caused by array controller errors that can utilize resilver algorithms to destroy an entire array even when no drive failure has occurred.  As RAID 0 (or RAID 1 or RAID 10) do not have resilver algorithms they do not suffer this additional risk.  These are hard risks to quantify but what is important is that they are additional risks that accumulate when using a more complex system when a simpler system, without the redundancy, was more reliable from the outset.

Now that we have established that RAID 5 can be less reliable than RAID 0 I will point out the obvious dangers of RAID 0.  RAID in general is used to mitigate the risk of a single, lone hard drive failing.  We all fear a single drive simply failing and all data being lost.  RAID 0, being a large stripe of drives without any form of redundancy, takes the risk of data loss of a single drive failing and multiplies it across a number of drives where any drive failing causes total loss of data to all drives.  So in our eleven disk example above, if any of the eleven disks fails all is lost.  It is clear to see where this is dramatically more dangerous than just using a single drive, all alone.

What I am trying to point out here is that redundancy does not mean reliability.  Just because something is redundant, like RAID 5, provides no guarantee that it will always be more reliable than something that is not redundant.

My favourite analogy here is to look at houses in a tornado.  In one scenario we build a house of brick and mortar.  On the second scenario we build two redundant house, east out of straw (our builders are pigs, apparently.)  When the tornado (or big bad wolf) comes along which is more likely to leave us with a standing house?  Clearing one brick and mortar house has some significant reliability advantages over redundant straw houses.  Redundancy didn’t matter, reliability mattered in the end.

Redundancy is often misleading because it is easy to quantify but hard to qualify.  Redundancy is a black or white question: Is it redundant?  Yes or no.  Simple.  Reliability is not so simple.  Reliability is about failure rates and likelihoods.  It is about statistics and analysis.  As it is hard to quantify reliability in a meaningful way, especially when selling a project to the business people, redundancy often becomes a simple substitute for this complex concept.

The concept of using redundancy to misdirect questions of reliability also ends up applying to subsystems in very convoluted ways.  Instead of making a “system” redundant it has become common to make a highly reliable, and low cost, subsystem redundant and treat subsystem redundancy as applying to the whole system.  The most common example of this is RAID controllers in SAN products.  Rather than having a redundant SAN (meaning two SANs) manufacturers will often make that one component not often redundant in normal servers redundant  and then calling the SAN redundant – meaning a SAN that contains redundancy, which is not at all the same thing.

A good analogy here would be to compare having redundant cars meaning two complete, working cars and having a single car with a spare water pump in the trunk in case the main one fails.  Clearly, a spare water pump is not a bad thing.  But it is also a trivial amount of protection against car failure compared to having a second car ready to go.  In one case the entire system is redundant, including the chassis.  In the other we are making just one, highly reliable component redundant inside the chassis.  It’s not even on par with having a spare tire which, at least, is a car component with a higher likelihood of failure.

Just like the myth of RAID 5 reliability and system/subsystem reliability, shared storage technologies like SANs and NAS often get treated in the same way, especially in regards to virtualization.  There is a common scenario where a virtualization project is undertaken and people instinctively panic because a single virtualization host represents a single point of failure where, if it fails, many systems will all fail at once.

Using the term “single point of failure” causes a panic feeling and is a great means of steering a conversation.  But a SPOF, as we like to call it, while something we like to remove when possible may not be the end of the world.  Think about our brick house.  It is a SPOF.  Our two houses of straw are not.  Yet a single breeze takes out our redundant solutions faster than our reliable SPOF.  Looking for SPOFs is a great way to find points of fragility in a system, but do not feel that every SPOF must be made redundant in every scenario.  Most businesses will find their best value having many SPOFs in place.  Our real goal is reliability at appropriate cost, redundancy, as we have seen, is no substitute for reliability, it is simply a tool that we can use to achieve reliability.

The theory that many people follow when virtualizing is that they take their virtualization host and say “This host is a SPOF, so I need to have two of them and use High Availability features to allow for transparent failover!”  This is spurred by the leading virtualization vendor making their money firstly by selling expensive HA add on products and secondly by being owned by a large storage vendor – so selling unnecessary or even dangerous additional shared storage is a big monetary win for them and could easily be the reason that they have championed the virtualization space from the beginning.  Redundant virtualization hosts with shared storage sounds great but can be extremely misguided for several reasons.

The first reason is that removing the initial SPOF, the virtualization host, is replaced with a new SPOF, the shared storage.  This accomplishes nothing.  Assuming that we are using comparable quality servers and shared storage all we’ve done is move where the risk is, not change how big it is.  The likelihood of the storage system failing is roughly equal to the likelihood of the original server failing.  But in addition to shuffling the SPOF around like in a shell game we’ve also done something far, far worse – we have introduced chained or cascading failure dependencies.

In our original scenario we had a single server.  If the server stayed working we are good, if it failed we were not.  Simple.  Now we have two virtualization hosts, a single storage server (SAN, NAS, whatever) and a network connecting them together.  We have already determined that the risk of the shared storage failing is approximately equal to our total system risk in the original scenario.  But now we have the additional dependencies of the network and the two front end virtualization nodes.  Each of these components is more reliable than the fragile shared storage (anything with mechanical drives is going to be fragile) but that they are lower risk is not the issue, the issue is that the risks are combinatorial.

If any of these three components (storage, network or the front end nodes) fail then everything fails.  The solution to this is to make the shared storage redundant on its own and to make the network redundant on its own.  With enough work we can overcome the fragility and risk that we introduced by adding shared storage but the shared storage on its own is not a form of risk mitigation but is a risk itself which must be mitigated.  The spiral of complexity begins and the cost associated with bringing this new system up on par with the reliability of the original, single server system can be astronomic.

Now that we have all of this redundancy we have one more risk to worry about.  Managing all of this redundancy, all of these moving parts, requires a lot more knowledge, skill and preparation than does managing a simple, single server.  We have moved from a simple solution to a very complex one.  In my own anecdotal experience the real dangers of solutions like this come not from the hardware failing but from human error.  Not only has little been done to avoid human error causing this new system to fail but we’ve added countless points where a human might accidentally bring the entire system, redundancy and all, right down.  I’ve seen it first hand; I’ve heard the horror stories.  The more complex the system the more likely a human is going to accidentally break everything.

It is critical that as IT professionals that we step back and look at complete systems and consider reliability and risk and think of redundancy simply as a tool to use in the pursuit of reliability.  Redundancy itself is not a panacea.  Neither is simplicity.  Reliability is a complex problem to tackle.  Avoiding simplistic replacements is an important first step in moving from covering up reliability issues to facing and solving them.

Choosing the right operating system for a storage server is important and not always that easy.  I work extensively in this space and people often ask me what I recommend and the recommendations vary, based on scenario, and often seem confusing.  But the factors are actually relatively easy, if you just know the limitations that create the choices and paths in the decision tree.

Before choosing an OS we must stop and consider what our needs are going to be.  Some areas that need to be considered are: capacity, performance, ease of administration, budget, connection technology, cust and clustering.  There are two main categories of systems that we will consider as well, standard operating system or storage appliance operating system.  The standard operating systems are Windows, Linux, Solaris and FreeBSD.  The storage appliance operating systems are OpenFiler, FreeNAS and NexentaStor.  There are others in both categories but these are the main players currently.

The first decision to be made is whether or not you or your organization is comfortable supporting a normal operating system operating in a storage server role.  If you are looking at NAS then simply ask yourself if you could administer a file server.  Administrating a block storage server (SAN) is a little more complex or, at least, unusual, so this might induce a small amount of concern but is really in line with other administration tasks.  If the answer is yes, that using normal operating system tools and interfaces is acceptable to you, then simply rule out the “appliance” category right away.

Storage appliance operating systems exist only to provide a pre-packaged, “easy to use” view into running a storage server.  In concept this is nice, but there are real problems with this method.  The biggest problems come from the packaging process which pulls you a step away from the enterprise OS vendors themselves making your system more fragile, further behind in updates and features and less secure than the traditional OS counterparts.  It also leaves you at the mercy of a very small company for OEM-level support when something goes wrong rather than with a large enterprise vendor with a massive user base and community.  The appliancization process also strips features and options from the systems by necessity.  In the end, you lose.

Appliances are nice because you get a convenient web interface from which “anyone” can administer your storage.  At least in theory.  But in reality there are two concerns.  The first is that there is always a need to drop into the operating system itself and fix things every once in a while.  Having the custom web interface of the appliance makes this dramatically harder than normal so at the time when you most need the appliance nature of the system is when you do not have it.  The second is that making something as critical as storage available for “anyone” to work on is a terrifying thought.  There are few pieces of your infrastructure where you want more experience, planning and care taken than in storage.  Making the system harder to use is not always a bad thing.

If you are in need of the appliance system then primarily you are looking at FreeNAS and OpenFiler.  NexentaStor offers a compelling product but it is not available in a free version and the cost can be onerous.  The freely downloadable version appears to be free for the first 18TB of raw storage but the license states otherwise making this rarely the popular choice.

FreeNAS, outside of clustering, is the storage platform of choice in an appliancized package.  It has the much touted ZFS filesystem which gives it flexibility and ease of use lacking in OpenFiler.  It also has a working iSCSI implementation so you can use FreeNAS safely as either a NAS or a SAN.  Support for FreeNAS appears to be increasing with new developments being made regularly and features being retained.  FreeNAS offers a large range of features and supported protocols.  It is believed that clustering will be coming to FreeNAS in the future as well as this has recently been added to the underlying FreeBSD operating system.  If so, FreeNAS will completely eliminate the need for OpenFiler in the marketplace.  FreeNAS is completely free.

OpenFiler lacks a reliable iSCSI SAN implementation (unless you pay a fortune to have that part of the system replaced with a working component ) and is far more out of date than its competitors but does offer full block-level real-time replication allowing it to operate in a clustered mode for reliability.  The issue here being that the handy web interface of the NAS appliance does not address this scenario and if you want to do this you will need to get your hands dirty on the command line, very dirty indeed.  This is expert level stuff and anyone capable of even considering a project to make OpenFiler into a reliable cluster will be just as comfortable, and likely far more comfortable, building the entire cluster from scratch on their Linux distribution of choice.  OpenFiler is built on the rather unpopular rPath Linux using the Conary packaging system both which are niche players, to say the least, in the Linux world.  You’ll find little rPath support from other administrators and many packages and features that you may wish access to are unavailable.  OpenFiler’s singular advantage of any significance is the availability of DRBD for clustering.  Support for OpenFiler appears to be waning with new features being non-existant and, in fact, key features like the AFP have been dropped rather than new features having been added.  OpenFiler is free but key features, like iSCSI, are not.

If you do not need to have your storage operating system appliancized then you are left with more and better choices, but a far more complex decision tree.    Unlike the appliance OS market which is filled with potholes (NexentaStor has surprise costs, OpenFiler appears to support iSCSI but causes data loss, features get removed from new versions) all four operating systems mentioned here are extremely robust and feature rich.  Three of them have OEM vendor support which can be a major deciding factor and all have great third party support options far broader than what is available for the appliance market.

The first decision is whether or not Windows only features, notably NTFS ACLs, are needed.  It is common for new NAS users to be surprised when the SMB protocol does not provide all of the granular filesystem control that they are used to in Windows.  This is because those controls are actually handled by the filesystem, not the network protocol, and Windows alone provides these via NTFS.  So if that granular Windows file control is needed, Windows is your only option.

The other three entrants, Linux, Solaris and FreeBSD, all share basic capabilities with the notable exception of clustering.  All have good software RAID, all have powerful and robust filesystems, all have powerful logical volume management and all provide a variety of NAS and SAN connection options.  Many versions of Linux and FreeBSD are available completely freely.  Solaris, while free for testing, is not available for free for production use.

The biggest differentiator between these three OS options is clustering.  Linux has had DRBD for a long time now and this is a robust filesystem clustering technology.  FreeBSD has recently (as of 9.0) added HAST to serve the same purpose.  So, in theory, FreeBSD has the same clustering options as Linux but this is much newer and much less well known.  Solaris lacks filesystem clustering in the base OS and requires commercial add-ons to handle this at this time.

Solaris and FreeBSD share the powerful and battle tested ZFS filesystem.  ZFS is extremely powerful and flexible and has long been the key selling point of these platforms.  Linux’ support for filesystems is more convoluted.  Nearly any Linux distribution (we care principally about RHEL/CentOS, Oracle Unbreakable Linux, Suse/OpenSuse and Ubuntu here) supports EXT4 which is powerful and fast but lacks some of the really nice ZFS features.  However, Linux is rapidly adopting BtrFS which is very competitive with ZFS but is nascent and currently only available in Suse and Oracle Linux distros.  We expect to see it from the others soon for production use but at this time it is still experimental.

Outside of clustering, likely the choice of OS of these three will come down primarily to experience and comfort.  Solaris is generally known for providing the best throughput and FreeBSD the worst.  But all three are quite close.  Once BtrFS is widely available and stable on Linux, Linux will likely become the de facto choice as it has been in the past.

Without external influence, my recommendation for storage platform are FreeBSD and then Linux with Solaris eliminated on the basis that rarely is anyone looking for commercial support and so it is ruled out automatically.  This is based almost entirely on the availability of Copy-on-Write filesystems and assuming no clustering which is not common.  If clustering is needed then Linux first then FreeBSD and Solaris is ruled out, again.

Linux and FreeBSD are rapidly approaching each other in functionality.  As BtrFS matures on Linux and HAST matures on FreeBSD they seem to be meeting in the middle with the choice being little more than a toss up.

There is no single, simple answer.  Choosing a storage OS is all about balancing myriad factors from performance, resources, features, support, stability, etc.  There are a few factors that can be used to rule out many contenders and knowing these hard delimiters is key.  Knowing exactly how you plan to use the system and what factors are important to you are important in weeding through the available options.

Even once you pick a platform there are many decisions to make.  Some platforms include multiple file systems.  There is SAN and NAS.  There are multiple SAN and NAS protocols.  There is network bonding (or teaming, the Windows world.)  There is Multipathing.  There are snapshots, volumes, RAID.  The list goes on and on.

When considering the cost of printing we often calculate the cost of the printer itself along with the consumables: paper and ink.  These things alone rack up a pretty serious per-page cost for an average business.  Planning for an appropriate lifespan and duty cycle of a printer are critical to making printing remain cost effective.  And do not forget the cost of parts replacement as well as stock-piled ink and paper.  These may seem minor, but printers often cause an investment in inventory that is never recovered.  When the printer dies, supplies for that printer are often useless.

The big, hidden cost of printing is none of these things. The big cost is in supporting the printers, both upfront with the initial deployment but even moreso in continuing support.  This is especially true in a smaller shop where the trend is to use many small printers rather than fewer large ones.  Deploying and supporting a five thousand dollar central office printer is no more than, and possibly lower than, the cost of deploying a two hundred dollar desktop inkjet.  The bigger the printer the better the support in drivers and support from the vendor that can usually be expected making normal support tasks easier and more reliable.

At a minimum, rolling out a new desktop printer is going to take half an hour.  Realistically it is far more likely to take closer to an hour.  Go ahead, count up the time: time to deliver printer to station, time to unpack printer, time to physically set up printer, time to plug in printer, time to install printer drivers and software, time to set up printer and time to print a test page.  If it was a one time race, you could probably do these steps pretty quickly.  But printer support is not a production line and rarely, if ever, do you have someone with these exact steps being performed in a rapidly repeatable manner.  Likely installing a printer is a “one off” activity that requires learning the new printer, tracking down the current driver and troubleshooting potential issues.

An hour to deploy a two hundred dollar printer could add fifty percent to the cost of the printer quite easily.  There are a lot of factors that can cause this number to skyrocket from a long travel distance between receiving location and the desk to missing cables to incompatible drivers.  Any given printer could take the better part of a day to deploy when things go wrong.  We are not even considering “disruption time” – that time in which the person receiving the printer is unable to work since someone is setting up a printer at their workstation.

Now that the printer has been set up and is, presumably, working just fine we need to consider the ongoing cost of printer support.  It is not uncommon for a printer to sit, undisturbed, for years chugging along just fine.  But printers have a surprisingly high breakage rate caused by the nature of ink, the nature of paper, a propensity for printers to be reassigned to different physical locations or for the machine to which they are attached to be changed or updated introducing driver breakage.  Add these things together and the ongoing support cost of a printer can be surprisingly high.

I recently witnessed the support of a company with a handful of high profile printers.  In a run of documentation, physical cabling and driver issues the printers were averaging between four and eight hours of technician time, per printer, to set up correctly.  Calculate out the per hour cost for that support and those printers, likely already costly, just became outrageously expensive.

I regularly hear of shops that decide to re-purpose printers and spend many times the cost of the printers in labor hours as older printers are massaged into working with newer computer setups or vice versa. Driver incompatibility or unavailability is far more common than people realize.

Printers have the additional complication of being used in many different modes such as directly attached to a workstation, directly attach and shared, directly attached to a print server, directly attached to the network or attached to a print server over the network.  While this complexity hardly creates roadblocks it does significantly slow work done on printers in a majority of businesses.

Printers, by their nature, are very difficult to support remotely.  Getting a print driver installed remotely is easy.  Knowing that something has printed successfully is something completely different.  Considering that printer support should be one of the lower cost support tasks this need for physical on-site presence for nearly every printer support task dramatically increases the cost of support if only because it increases the time to perform a task and receive appropriate feedback.

When we take these costs and combine them with the volume of printing normally performed by a printer we can start to acquire a picture of what printing is really costing.  The value to centralized printing suddenly takes on a new level of significance when seen through the eyes of support rather than through the eyes of purchasing.  Even beyond centralizing printing when possible it is important to eliminate unnecessary printing.

Good planning, strategic purchasing and a holistic approach can mitigate the potential for surprise costs in printing.

Virtualization itself, I should point out, I feel should be a default decision today for those working in the x64 computing space with systems being deployed sans virtualization only when a clear and obvious necessity exists such as specific hardware needs, latency sensitive applications, etc.  Baring any specific need, virtualization is free to implement from many vendors and offers many benefits both today and in future-proofing the environment.

That being said, what I often see today is companies deploying virtualization not as a best practice but as a panacea to all perceived IT problems.  This it certainly is not.  Virtualization is a very important tool to have in the IT toolbox and one that we will reach for very often, but it does not solve every problem and should be treated like every other tool that we posses and used only when appropriate.

I see several things recurring when virtualization discussions come up as a topic.  Many companies today are moving towards virtualization not because they have identified a business need but because it is the currently trending topic and people feel that if they do not implement virtualization that somehow they will be left behind or miss out on some mythical functionality.  This is generally good as it is increasing virtualization adoption, but it is bad because good IT and business decision making processes are being bypassed.  What happens is often that in the wave of virtualization hype IT departments feel that not only do they have to implement virtualization itself but do so in ways that may not be appropriate for their business.

There are four things that I often see tied to virtualization, often accepted as virtualization requirements, whether or not they make sense in a given business environment.  These are server consolidation, blade servers, SAN storage and high availability or live failover.

Consolidation is so often vaunted as the benefit of virtualization that I think most IT departments forget that there are other important reasons for doing implementing it.  Clearly consolidation is a great benefit for nearly all deployments (mileage may vary, of course) and is nearly always able to be achieved simply through better utilization of existing resources.  It is a pretty rare company that runs more than a single physical server that cannot shave some amount of cost through limited consolidation and it is not uncommon to see datacenter footprints decimated in larger organizations.

In extreme cases, though, it is not necessary to abandon virtualization projects just because consolidation proves to be out of the question.  These cases exist for companies with high utilization systems and little budget for a preemptive consolidation investment.  But these shops can still virtualize “in place” systems on a one to one basis to gain other benefits of virtualization today and look to consolidate when hardware needs to be replaced tomorrow or when larger, more powerful servers become more cost effective in the future.  It is important to not rule out virtualization just because its most heralded benefit may not apply at the current time in your environment.

Blade servers are often seen as the choice for virtualization environments.  Blades may play better in a standard virtualization environment than they do with more traditional computational workloads but this is both highly disputable and not necessarily applicable data.  Being a good scenario for blades themselves does not make it a good scenario for a business.  Just because the blades perform better than normal when used in this way does not imply that they perform better than traditional servers – only that they have potentially closed the gap.

Blades needs to be evaluated using the same harsh criteria when virtualizing as when not and, very often, they will continue to fail to provide the long term business value needed to choose them over the more flexible alternatives.  Blades remain far from a necessity for virtualization and often, in my opinion, a very poor choice indeed.

One of the most common misconceptions is that by moving to virtualization one must also move to shared storage such as SAN.  This mindset is the obvious reaction to the desire to also achieve other benefits from virtualization which, if they don’t require SAN, benefit greatly from it.  The ability to load balance or failover between systems is heavily facilitated by having a shared storage backend.  It is a myth that this is a hard requirement, but replicated local storage brings its own complexities and limitations.

But shared storage is far from a necessity of virtualization itself and, like everything, needs to be evaluated on its own.  If virtualization makes sense for your environment but you need no features that require SAN, then virtualize without shared storage.  There are many cases where local storage backed virtualization is an ideal deployment scenario.  There is no need to dismiss this approach without first giving it serious consideration.

The last major assumed necessary feature of virtualization is system level high availability or instant failover for your operating system.  Without a doubt, high availability at the system layer is a phenomenal benefit that virtualization brings us.  However, few companies needed high availability at this level prior to implementing virtualization and the price tag of the necessary infrastructure and software to do it with virtualization is often so high as to make it too expensive to justify.

High availability systems are complex and often overkill.  It is a very rare business system that requires transparent failover for even the most critical systems and those companies with that requirement would almost certainly already have failover processes in place.  I see companies moving towards high availability all of the time when looking at virtualization simply because a vendor saw an opportunity to dramatically oversell the original requirements.  The cost of high availability is seldom justified by the potential loss of revenue from the associated reduction in downtime.  With non-highly available virtualization, downtime for a failed hardware device might be measured in minutes if backups are handled well.  This means that high availability has to justify its cost in potentially eliminating just a few minutes of unplanned downtime per year minus any additional risks assumed by the added system complexity.  Even in the biggest organizations this is seldom justified on any large scale and in a more moderately sized company it is uncommon altogether.  But today we find many small businesses implementing high availability systems at extreme cost on systems that could easily suffer multi-day outages with minimal financial loss simply because the marketing literature promoted the concept.

Like anything, virtualization and all of the associated possibilities that it brings to the table need to be evaluated individually in the context of the organization considering them.  If the individual feature does not make sense for your business do not assume that you have to purchase or implement that feature.  Many organizations virtualize but use only a few, if any, of these “assumed” features.  Don’t look at virtualization as a black box, look at the parts and consider them like you would consider any other technology project.

What often happens in a snowball effect where one feature, likely high availability, is assumed to be necessary without the proper business assessment being performed.  Then a shared storage system, often assumed to be required for high availability, is added as another assumed cost.  Even if high availability features are not purchased the decision to use SAN might already be made and fail to be revisited after changes to the plan are made.  It is very common, in my experience, to find projects of this nature with sometimes more than fifty percent of the total expenditure on the project being spent on products that the purchaser is unable to even describe the reason for having purchased.

This concept does not stop at virtualization.  Extend it to everything that you do.  Keep IT in perspective of the business and don’t assume that going with one technology automatically assumes that you must adopt other technologies that are popularly associated with it.