Survey reveals extensive server disasters and data loss among enterprises

December 14 2005

A VERY interesting press release from Lucid8:

A recent survey provides clear evidence that a large number of organizations are experiencing disastrous Exchange Server failures. The survey was conducted by Lucid8, a provider of automated disaster prevention, optimization and recovery software for Microsoft Exchange Server and was directed at organizations that rely heavily on e-mail as a driving force in communications. More than 360 IT managers, messaging and Exchange specialists participated in the study, providing a highly representative base of respondents.
Seems like a pretty good sample size and solid methodology...let's take a look at the results.
- Nearly three quarters of the respondents had suffered from an Exchange disaster. 70 percent of these firms stated that their Exchange Server was down for at least eight hours, and 45 percent said it was down for more than 24 hours per incident.

- Almost half of the respondents stated the cost of lost opportunity was more than $5,000 per incident, and 23 percent reported that they had experienced permanent data loss from server disasters.
70% of firms have had at least eight hours downtime per incident, and almost half have had more than one day downtime per incident.  One quarter have experienced permanent data loss.

It almost would be too easy to talk about Domino's multiplatform characteristics, single mailbox per user deployments, and shared-nothing clustering at this point.  The survey results would be substantially different for IBM Lotus Domino customers.  Hmmm...

Ed Brill | In the snow Category:  Microsoft  Add/View Comments [41]

Post a Comment

  1. 1  Alan Bell http://www.dominux.co.uk | 12/14/2005 9:24:07 AM

    like most surveys they got the result they wanted, these guys sell stuff which makes Exchange suck less for disaster recovery so it makes sense to talk up the bad things that happen if you don't pay lots of money to Lucid8. They look to be very focussed on Exchange, it would be good to repeat the survey somehow for Domino, but it wouldn't work for IBM to do so, they don't have the same agenda as Lucid8. I can't think of anyone who makes a living doing DR for Domino which says a lot in itself. Perhaps Lucid8 would like to do the survey for Domino as a market research exercise to see if they would like to expand their offerings.

  1. 2  Danny Lawrence http://www.tiassatech.com/domino/tiahome.nsf/83e9f45c11caa9d58525647300561fe6/6bf5e9d28c26fa508525709700542fcc?OpenDocument | 12/14/2005 9:28:26 AM

    Ed, can I plug my e-pro article (now on my website since e-pro has been moved to dev/null) again?

    Note that while we had a machine down for close to 72 hours, there was no data loss, indeed the userbase didn't even know we had a problem. And to those who say "yes, but your servers were clustered", 2 points:

    1. AFAIK Exchanged still shares the RAID array in clustered servers, if the RAID controller on an Exchanged server died as ours did, the Exchanged server would be dead.

    2. Even if our server wasn't clustered, we could have built another box from scratch, and run it as a stopgap while we were rebuilding the original box, and let it replicate when we got the procudtion server back up. There might have been some data loss in this scenario, but not to the degree you see when the Exchanged DB goes south.

  1. 3  Dave http://www.weblayouts.net | 12/14/2005 9:31:10 AM

    There is information missing from this study:

    In all of the respondents' DR plans, how many of them planned on Exchange having an RTO of less than 8 or less than 24 hours?

    For my company, I think we are shooting for a week in a DR situation. Lack of e-mail would be frustrating, but it wouldn't actually stop our business. It would only hinder communication. For those statistics to mean anything, we need to know what percentage considered Exchange to be a "critical" application.

    (And that means more than "rely heavily" on it for communications. Everyone does that -- the question is whether or not that communication actually has a material impact on your business.)

  1. 4  Wild Bill http://www.billbuchan.com | 12/14/2005 9:34:30 AM

    Yup - Exchange is not a solid technical base. And where's the Kodiak - Exchange on SQL upgrade - their customers were promised years ago ?

    What terrifies me is that you can now bump up the datastore size in Exchange from the 16gig or so, to something like 72gig. I guess folks were complaining that they wanted more capacity in their inboxes - I've seen exhange sites that absolutely nailed users to 50mb, and I think thats what the internal limit was at Redmond for a long time. (Though I think their eMail retention policy was dictated by their court appearances more than anything else).

    The biggest mailfile I've seen in the wild was 12gb - working quite happily. And clustered.

    Exchange ?

    Love the new Notes "Boxing gloves" advertising campaign. Any chance of putting some blood on those gloves now in terms of highlighting Exchanges considerable stability problems ?

    ---* Bill

  1. 5  Wild Bill http://www.billbuchan.com | 12/14/2005 9:35:29 AM

    Sorry. To clarify. The biggest mailfile I've seen in the wild was 12gb - and it was on Domino. Of course.

    ---* Bill

  1. 6  Bruce Elgort http://www.bruceelgort.com | 12/14/2005 9:46:59 AM

    During the day I manage an Exchange shop with 300 users. Our server hasn't gone down once in the last two years. I am also in the process of designing a new DR system for our Exchange 2003 migration and there are many rock solid solutions out there for Exchange DR (Double-Take for one). Yes, I know it's not built in like Domino has with clustering/replication but it works.

    For all of you out there who say Exchange sucks have you ever actually used it?

  1. 7  Chris Miller http://www.IdoNotes.com | 12/14/2005 9:55:43 AM

    @1 - Do we make a living providing DR for Domino? No, not a living. But we do have way more than a handful of them online for numerous business reasons.

  1. 8  Ben Rose http://www.jaffacake.net | 12/14/2005 10:02:06 AM

    @6 - Bruce

    I look forward to the OpenNTF mail template for Outlook/Exchange.

  1. 9  Bruce Elgort http://www.bruceelgort.com | 12/14/2005 10:05:59 AM

    @Ben,

    You mean the one that makes Outlook look like Notes Mail :-)

  1. 10  Paul Robichaux http://www.e2ksecurity.com | 12/14/2005 10:12:06 AM

    @1: what Alan said. Lucid8 is known for using FUD to sell their solutions, so it's no surprise that they're still doing it. See e.g. { Link }

  1. 11  Ben Rose http://www.jaffacake.net | 12/14/2005 10:14:44 AM

    @9 - Oooh, that would be nice. Can it put new emails at the bottom too? ;O)

  1. 12  Bill Pappert  | 12/14/2005 10:37:00 AM

    @6 --

    Have we used Exchange? To lay out some bona fides, first. I've worked with earlier releases (5.5) and migrated some from Exchange to Domino. I've run a very small Exchange 2003 environment at home to learn a bit about it. I'm an IBM Certified Advanced System Admin (ND 6/6.5) and an MCSA: Messaging (Win 2003).

    Does Exchange suck? At some things, undoubtedly it does. By storing messages and attachments for all users in one giant pot, it sucks. In having no active/active clustering, it does.

    I've never managed a large or enterprise-scale Exchange network (although my good friend Simon is about to embark on that journey). I have found that it is easy enough to administer in a small scale. I don't know how disaster recovery is in Exchange with a large environment.

    I do know, thanks to a tiny disk controller failure in my current environment, just how nicely an active/active cluster in Domino recovers. See { Link } for more on that.

    Knowing what I know about how Exchange works, I'd not want to see a repeat of that in an Exchange environment of similar scale (the mail server hosting 1200 users, including the company's biggest of guns).

  1. 13  Darren http://www.dadams.co.uk | 12/14/2005 10:54:09 AM

    @6 - like Bill (@12) I've never managed an Exchange infrastructure, only set it up for my own amusement and education. But I have spoken to enough Exchange administrators and IT Managers to make me believe there is some substance in the Lucid8 article.

  1. 14  Bruce Elgort http://www.bruceelgort.com | 12/14/2005 11:27:32 AM

    I do manage a relatively small Exchange environment and in my environment it works. You know I love Notes/Domino, but instead of reading analyst reports it would be better to hear real stories from real people :-).

    For companies (like the one I work for)that aren't into collaborative computing (outside of email) Exchange fits the bill. I would have a hard time putting a Notes/Domino installation in an organization for simply email at this point (being honest here guys). For customrs wanting a collaborative environment where they needed the rich application framework the Notes client provides I would recommend ND.

    How many of you have put Notes/Domino into shops that simply use ND for email?

    How many Exchange shops do you know of who have adopted M$'s collaborative apps such as Sharepoint, LCS etc?

  1. 15  Bill Pappert  | 12/14/2005 12:32:21 PM

    @14 - Bruce, I'm with you on the "would have a hard time putting in Notes/Domino in a small environment that just wants mail." I've said for years (sorry, Ed!) that if you're just buying mail and you've invested in a Domino infrastructure that you've done something akin to a single man buying a school bus when all he needs is basic transport.

  1. 16  Ed Brill www.edbrill.com | 12/14/2005 12:49:59 PM

    @15 Bill, no need to apologize. While I think Notes/Domino has some world-class features in a "mail only" configuration, it's definitely at the high-end features-wise for that. Exchange as a mail/calendar system is up against a lot more competitors in Groupwise, Oracle, Scalix, Sendmail, Bynari, IPSwitch, Mirapoint, and everybody else in that boat. Vast majority of Notes/Domino customers run "more than mail" which is why MS desperately tries to separate the mail and applications discussions... they can't beat us in the "more than" category.

  1. 17  Greg Wojcicki  | 12/14/2005 1:13:04 PM

    @14 and @15 SMB can/should still consider Domino for mail. If a small organization doesn't want/need the functionality beyond mail they can license a server as "mail only". Ed, wasn't this licensing type created to allow SMB to consider Domino as an option for messaging?

  1. 18  Ed Brill www.edbrill.com | 12/14/2005 1:15:23 PM

    Yes, Domino Messaging Express.

    { Link }

  1. 19  Peter de Haas http://www.peterdehaas.com | 12/14/2005 1:32:32 PM

    Not as thorough as the Exchange survey perhaps ...

    { Link }

  1. 20  Alan Bell http://www.dominux.co.uk | 12/14/2005 2:34:02 PM

    @19 this isn't about crash frequency, which neither are immune from and the root cause could be anything from a buggy driver bringing down the operating system to decaying hardware. The point is the impact of a crash and the length of time it takes to do anything about it.

  1. 21  Christopher Byrne http://www.controlscaddy.com/ | 12/14/2005 3:05:37 PM

    "Get your facts first, and then you can distort them as much as you please"

    == Mark Twain

    I am sorry, I have a hard time buying any data that is put together by a vendor trying to sell a product. That is pure FUD.

  1. 22  Wild Bill http://www.billbuchan.com | 12/14/2005 3:07:42 PM

    Ah - Peter. When a Domino server crashes, it comes back up automatically (assuming you've switched that on), figures out which separate databases require maintenance, performs it and keeps going.

    Sometimes, if the crash is a hard drive, etc, then you have to restore. Domino's good, but it cant yet mysically retrieve data off of a Dead Drive. Ed - is that going to be in v8 ?

    Often, as you will have put replica copies of your mail file around the environment, even if a single database (say, a single mailfile) is not recoverable, then its fairly easy to get back from another on-line copy. At this point, you've not even got out the key for the tape-store.

    And of course, with clustered domino servers - very common in the bigger sites, the users barely notice that the server has crashed.

    So - we have a single mail-in survey that says that 14% of servers crash daily. If only they'd sent their names in - I'd be round there in a flash to get big fat consultancy checks off them for nothing more than reviewing their housekeeping routines, cleaning dead accounts off and perhaps defragging their windows servers. Money for old rope.

    Now - when an Exchange server crashes.. Well, there's a BIIIG datastore (or "stores"), thats almost certainly going to require maintenance. So everyone has a half-day off their eMail and shouts at the poor Exchange admin instead. They MIGHT try and work offline on local Outlooks, but more likely they'll just revert to their DR strategy - their yahoo/gMail/hotsnail accounts. Out goes any corporate eMail compliance.

    Smaller sites - such as Bruces' - might tick along for years with minimal effort. However, a similar sized site I know of in the UK was down for a whole week (Ironically that one was a group of Exchange consultants). So its a bit like russian roulette in that respect.

    The worst thing of course is that if you ARE going for the DR options - paying through the nose for Exchange clustering, special hardware, etc, you STILL have to share a single hard drive subsystem. And may God have mercy on your soul if you try running that over a WAN link for offsite DR...

    Whereas with Domino Replication - its very easy to ensure that ALL your data is copied up over inexpensive lines.

    Now.

    @all. Domino in a small site. I really dont know what you chaps are scared of. I've just been working on a 50-70 user office who have been running Domino v5 on a SUSE Linux server for at least THREE years. We swapped out their hardware for a new xSeries, Domino 7 and SUSE enterprise 9. (I think we spent more time racking and de-racking the kit than we did migrating domino across. No user changes, of course)

    On that sort of configuration, the SUSE only costs 350 STERLING - so there's quite a bit of cost saving on Windows 2000/2003. The Domino server, once up and running, ticks along without any oversight, and the local admin person things the domino admin client - although its got more controls in ND7 than v5, is still perfectly usable.

    So whats not SMB about Domino ? Bluewave.ie in Ireland for instance look after a number of "small" domino accounts and are really only there for the rare, big problems.

    Or just get them hosted by someone like prominic.net, for instance. Cheap, reliable, extensable, predictable.

    (Peter - your Exchange product is getting a real kicking of late. Is there any *good* news about this product ? For instance, a complete re-architecture in E12 to get it off the increasingly aged/flaky Jet database, or to provide reliable DR facilities ? It must be a real bugger to pursuade folks used to 99.9% uptime to switch...)

    ---* Bill

  1. 23  david racicot  | 12/14/2005 4:13:17 PM

    @22 thanks Bill. I was going to type the same thing. Domino is for SMB. Do what Bill outlines and in addition to mail you can build apps for these customers and run on their existing server. Ed. We need a boxing glove ad with that on it. Suse on one mit, Domino on the other. Wham, wham.

  1. 24  david racicot  | 12/14/2005 4:18:59 PM

    @3. he he ... Nice defense ... "maybe they expect it to be down for a week." Ed. We should use that too.

    Dear: Mr. CEO. Did we forget to tell you that we expect a 50%-70% chance that we'll be without our Exchange system (email to you sir) for a week at a time.

    Signed

    Mr. I can't get fired for recommending MS.

  1. 25  Peter de Haas http://www.peterdehaas.com | 12/15/2005 1:35:33 AM

    @22

    Bill, Good news about MS Exchange ? on Ed's blog ?

    That was not what I expected ;-)

    Ed claims : "Seems like a pretty good sample size and solid methodology...let's take a look at the results."

    I can honoustly not tell this from thea *press release*. What was the methodology ? Why is the sample size good ? Who are these respondents (what demographics ?)

    I must say I agree with Christopher (comment 21). Perhaps not of it beying FUD, I haven't seen the details, but you should clearly put this suvey into proper perspective of who is asking the questions for what specific reason.

  1. 26  Ben Rose http://www.jaffacake.net | 12/15/2005 3:04:38 AM

    Sylvester Stallone is currently filming Rocky VI if you really want a heavyweight campaign :O)

  1. 27  Ben Rose http://www.jaffacake.net | 12/15/2005 3:21:42 AM

    @18 - But you can't cluster the Domino Express products, so IBM are shooting themself in the foot a bit. One of the key features of Domino, it's excellent cross-platform clustering, and you can't use it in the express version.

  1. 28  Nathan T. Freeman  | 12/15/2005 4:25:31 AM

    @27, BS. You *CAN* cluster, it's just not legal according to your license. So you have to upgrade your license. You don't have to change anything on the software whatsoever. It's just that, legally, in order to click on "Add to Cluster" in your Directory, you need to have something above the Express license. If you want to click that button, you're obligated to send a check to IBM. That is all.

  1. 29  Ben Rose http://www.jaffacake.net | 12/15/2005 5:40:39 AM

    @28 - Yes, I was aware that it's the same software technically...I'm referring the license limitation. They're selling them a cheap product with a vital key competitive function missing. We all know that "shared nothing clustering" is at the top of any marketing cribsheet and it's not in the license for express.

  1. 30  GarryL  | 12/15/2005 6:20:28 AM

    @27 I wasn't aware of that. Interesting.

    Of course, the Express version is licenced per user, so you can have other servers sitting around replicating. If one goes down then you can switch to the other. Not quite the same as clustering, but possibly ok for SME's, which the product is aimed at.

    Still seems odd that its not available mind.

  1. 31  Nathan T. Freeman  | 12/15/2005 7:12:25 AM

    @29 - If you don't like that it doesn't come with the licensing option, fair enough. But that's not a technical comparison -- it's a licensing question. Let's all encourage IBM to offer an option in the Express range that allows clustering. Maybe a piece meal extension that allows just clustering for messaging servers or something.

  1. 32  Wild Bill http://www.billbuchan.com | 12/15/2005 9:52:28 AM

    @all - Dunno why we're unduly concerned about the lack of clustering in Domino Express. After all, the lowest version of Exchange cant cluster either.. ?

    And hey - if they want to go up the feature tree, its not necessarily even a machine software change - its just a license fee. Whilst it costs money, it doesnt sound crippling.

    ---* Bill

  1. 33  Bill Pappert  | 12/15/2005 10:32:58 AM

    @32 -- Some would argue, Bill, that the way Exchange does clustering, no version of Exchange really does clustering.

  1. 34  Tom Nichols http://www.TomsRant.com | 12/15/2005 11:18:52 AM

    Our company has had Exchange for almost 5 years and we have never been down for more than a couple of hours. Also, we've never had a total outage- we have @ 40,000 users.

    We also don't use Exchange for any collaborative efforts. We have web- based Domino apps to handle quite a bit of that.

    I love Domino, but I've not missed the mail or calendaring at all. I either got lucky, or Exchange is not as unstable as I'd heard in the past. When we went to Exchange (from "green screen" mainframe e-mail), I thought that we'd be down every week and would be riddled with viruses. Neither has happened.

    The only thing that really bothers me is that we have to rename certain file extensions or zip them with a password to get them through, otherwise Exchange will strip them.

  1. 35  Paul Robichaux http://www.e2ksecurity.com | 12/15/2005 3:01:29 PM

    @34: Tom, you probably already know this, but that kind of filtering is adjustable. You can customize the list of filtered file types fairly easily. Contact me privately for details (or, heck, just see { Link }

  1. 36  Danny Lawrence http://www.tiassatech.com/domino/tiahome.nsf/83e9f45c11caa9d58525647300561fe6/6bf5e9d28c26fa508525709700542fcc?OpenDocument | 12/15/2005 8:43:38 PM

    OK, so you can't cluster Express licenses. However there is nothing in the license agreement that prevents you from having another server and keeping replicas on that one. You don't get automatic failover, but you do have a decent recovery system.

    @34 "I thought that we'd be down every week and would be riddled with viruses. Neither has happened.

    The only thing that really bothers me is that we have to rename certain file extensions or zip them with a password to get them through, otherwise Exchange will strip them."

    Those 2 statements go hand in hand. The reason that Exchanged requires you to rename or zip your attachments is because that is the princiapl virus vector used by lookOut hackers. MSFT can't (or won't) fix the root cause so they cripple the utility of the product in order to cover up the security holes.

  1. 37  Bill Brown  | 12/16/2005 7:19:38 AM

    We've had Domino servers fail, and it resulted in extended periods of downtime for some of the school districts we serve.

    Sure, Domino can be clustered, but in education, resources are tight and they'll usually live with the possibility of downtime before paying for redundant servers.

    It took a day of downtime in our own office to convince them that we shouldcluster our own mail server. The application server still isn't clustered.

    Re: Filtering executable - Our spam filter is configured to block the extensions M$ documents as "unsafe." { Link } Yeah, we can override the defaults in this list, but why would anyone do that? It's opening a security hole.

  1. 38  drew shobbrook www.sydit.com | 12/16/2005 7:49:39 AM

    @6

    i have and our company has both notes and exchange clients and techs.. guess what exchange site go down more times than notes.. all our techs are MCSE .. and the notes servers problems are allways DNS or non notes issues..

  1. 39  Paul Robichaux http://www.e2ksecurity.com | 12/17/2005 5:00:51 AM

    @36: the root cause is that some file types are executable, and users who double-click them are running potentially malicious code on their boxes. This has nothing to do with Exchange, Outlook, Domino, Eudora, Thunderbird, or whatever.

  1. 40  Nathan T. Freeman  | 12/19/2005 1:58:45 AM

    @39 - Some clients make it easier to manage the behavior of executables than others.

  1. 41  Mike Robinson http://www.invcs.com | 12/20/2005 10:45:37 PM

    @6, it's not so much that it doesn't go down (yes that's important) but what happens when it *does* go down (and all servers do at some point- even an as/400). @21 looking for independent verification on Exchange recovery issues that's not FUD (which I totally agree with you on that).

    Here's an independent article written by OutlookPower.com. It's entitled "My 13 days in Exchange Hell". Here's the link { Link }

    Honestly I was impressed that he published an accurate account of what happens. The site *runs* Exchange for it's own mail. I find it pretty much interesting that other MVPs empathized and felt his pain- but couldn't offer a solution.

    Regarding Domino in an SMB environment and the express product? It's great- use the Outlook connector if the end user really wants Outlook. In fact using Domino, Outlook connector, you can access your email/calendar using just TCP/IP, no VPN. In an Exchange world, to do this you have to be 2003 (Windows, Exchange, and Outlook).

    (Note: I use this article very frequently with customers regarding Exchange vs. Domino. I email it to them, leave it on the chair of the CFO, etc. I normally get a cell phone call on the way home needing to talk about it...)