Guest blogger: Collin Murray: Domino storage versus lies, damn lies, and statistics

January 7 2010

Collin Murray is the Program Director for Lotus Domino, part of my product management team.  After hearing some recent competitive FUD, he asked for the opportunity to share some realities about cost-effective storage for Domino.

Storage.  I've been hearing about it a lot lately, and not just about cleaning up the garage to get the snow blower some exercise.  I've been hearing a lot of positive acclaim about the storage savings in Domino 8.5.x along with questions about Exchange storage improvements.  

The initial 'grabber' for most people to talk about storage has been disk savings of 40 – 60%.  Imagine not having to purchase new storage, not having to backup large volumes of data, and not having to archive or break up mail files into smaller psts (oops, did I say that?).  That's just for starters...

Now, imagine being able to use less expensive disks to store information.  For example, while a 30 day old email message may be periodically referenced, it is unlikely that an attachment in that message will be accessed with any significant frequency.  Domino 8.5's DAOS feature securely stores attachments outside of a Notes/Domino application or mail file.  This capability allows for the separation of static and dynamic data.  Storage acquisition costs can be dramatically reduced by using less expensive storage for older, or infrequently accessed data.  Considering each higher tier of storage can increase by a factor of 3, we're talking real money here.

Still not a believer?  Since Domino 7, we have reduced I/O consumption by more than 50%!  That's a significant reduction of a precious resource.  Reducing the amount of IO consumed allows for greater scalability (e.g.: more users) and corresponding cost reductions (e.g.: lower cost disks).  Perhaps we've been a bit too reserved about publicizing the significant IO reductions that Domino has achieved.  Our west coast competitor has been talking about the IO improvements in 2010.  I'm not sure if they woke up to the fact that Domino has been addressing this angle for the past several releases, or if they realized that in order to use their product for hosting -- they need to be much more efficient with storage.  While they're off re-designing mail file storage (again), we've been steadily investing in NSF improvements and optimizations.  That 50% IO improvement mentioned earlier is on top of already efficient IO handling.  Given where Microsoft is starting from, they must see any progress as good progress, but comparatively, it's still thin.  Eliminating 90% of the U.S. national debt is progress but it still leaves more than $1 trillion of debt!

As many of you know, this is really nothing new for Lotus -- we've been on the storage related savings 'kick' for some time now -- witness document compression, attachment compression, DAOS, optimized view updates, etc.  Without stealing too much thunder of what will be discussed at Lotusphere 2010, there are efforts underway to further reduce storage related costs (physical storage, reduced IO use, addressing peak IO times, administration, etc.).

Collin Murray Cambridge, MA USA Category:  Notes and Domino 8.5  Add/View Comments [35]

Post a Comment

  1. 1  Volker Weber http://vowe.net/about | 1/7/2010 11:37:01 AM

    Maybe it's obvious from the inside, but there are a few questions:

    - What is the competitive FUD?

    - If you can improve by 50% after 20 years, didn't you waste 100% along the way?

    - Would you be happier if you could save 90%?

    - How does that influence the national debt?

  1. 2  Keith Brooks http://www.vanessabrooks.com | 1/7/2010 2:22:17 PM

    Competitive discussions require clear details, when possible, or at least published information to compare. Specifics are important.

    MS has posted discussions along these lines, like this one: { Link }

    Not saying the information is valid but that is what you would go up against. Your post, aside from the title, doesn't provide much help.

  1. 3  Bill Geimer  | 1/7/2010 2:34:41 PM

    not to mention Exchange 2007 message stores crash on a daily basis.

  1. 4  Collin Murray http://www-01.ibm.com/software/lotus/products/domino/ | 1/7/2010 2:48:53 PM

    @1 - One of the current FUD tactics is: "Disk is cheap, IO is the cost point...". Domino has made improvements to IO and other aspects of storage - we could probably do a better job touting them to the rest of the world.

    The 50% is from version 7. There have been other incremental improvements in the past.

    I'd be happier if I could save 90% on something meaningful...

    @2 - Fair. Perhaps a link or two would help with context.

    { Link }

    { Link }

  1. 5  Ed Brill http://www.edbrill.com | 1/7/2010 2:50:04 PM

    and I was hesitant to link to the Microsoft FUD because it doesn't deserve my endorsement, but you can certainly find some of it on the Exchange website.

  1. 6  Volker Weber http://vowe.net/about | 1/7/2010 3:15:57 PM

    Collin, I am afraid that is not a FUD (FEAR UNCERTAINTY DOUBT) point. That is a statement you might not agree with. If you are looking for FUD, you will find it in comment #3.

    I/O is certainly one of the major pain points in storage today. We back up data to disk and not to tape because I/O is of more concern than disk cost. If you want to battle the backup issue, why would anyone need to backup derivative data like view or full text indexes?

    Please explain. What's with the 90% reduction of the national debt and the one trillion. How does that relate to your storage?

  1. 7  Ed Brill http://www.edbrill.com | 1/7/2010 3:39:20 PM

    The Microsoft document absolutely has FUD. I can send you the link offline.

  1. 8  Collin Murray http://www-01.ibm.com/software/lotus/products/domino/ | 1/7/2010 4:00:15 PM

    Volker, we are evaluating the aspect of storing derivative, or hot/dynamic data such as indexes separately. DAOS is a step towards that, and provides the potential to further distinguish between dynamic & static data (for backup, tiered storage, etc.).

    The 90% angle is an (attempt at) analogy between their stated IO improvement and the significance of the improvement. Its an improvement, yes, but does it provide sufficient relief? With over $1 trillion in debt, will it provide tax breaks, will the currency rise, etc.? (Not intended to be a political discussion or a country-centric view)

  1. 9  David Jeyachandran http://www.openntf.org/Projects/pmt.nsf/ProjectLookup/Data%20Moving%20Animal | 1/7/2010 5:03:06 PM

    Thanks Collin, it is good to be reminded of these figures. Great to see that Domino keeps getting better!

  1. 10  Tim Haugen  | 1/7/2010 6:56:05 PM

    How about some I/O improvements on the Client side? We've had major complaints and wide variation in Client start-up time(8.0.2FP3+). After some deep-dive work on trying to optimize, one item we've proven is a near direct correlation to drive RPM - 40% reduction in start-up replacing a 5400 RPM drive with 7200 RPM on the exact same machine (ghosting the drive so fragmentation, etc. is same). If this were a few seconds, the point would be academic, but the "improved" state is still 40-45 seconds.

  1. 11  Erik Brooks  | 1/7/2010 7:59:39 PM

    @8 - "...the aspect of storing derivative, or hot/dynamic data such as indexes separately..."

    Oooooh, tell me you've been reading my blog. :-)

    @10 - Collins is the Program Director for Domino, not Notes, so it's probably outside his area.

    Your problem is the fact that, since the move to Eclipse, Notes has to load a bazillion tiny little .jar files, causing your drive heads to seek around like a starving hamster with ADHD trapped in a grocery store.

    If you want to really make your machines fly, throw an SSD in there. The Notes client will start in 4-5 seconds from a *cold* boot. When your seek times improve by a factor of 1000 things get FAST.

  1. 12  Erik Brooks  | 1/7/2010 8:05:16 PM

    Oh, in case you haven't and want to read my blog on index and i/o-related discussions:

    { Link }

    Look for the articles "The Future of NSF."

  1. 13  Mike Robinson http://www.invcs.com | 1/7/2010 8:44:33 PM

    I admit I was sorta looking forward to reading some of that MS FUD, but do respect Ed's decision not to link to it. I didn't even bother clicking the link though.

    I've always believed that DAOS is "the bomb" and why on earth IBMers doing anything with Notes/Domino doen't talk about it all the time (and I know it doesn't happen b/c I'm sitting in the same room with them on multiple occasions and accounts).

    I *always* hear about Exchange SCOS from Exchange folks, and when set up properly it does work quite well.

    While disk may be cheap the cost to implement this cheap disk isn't. Personally DAOS is to the Domino Server Mgmt as Xpages is to Notes Development- it's something that is a very real differentiator and everyone (that signs off on HW procurement) gets that point immediately. Just very few decision makers know about it.

  1. 14  Collin Murray http://www-01.ibm.com/software/lotus/products/domino/ | 1/7/2010 9:09:40 PM

    Tim - thanks for the feedback - I'll bring it to the client team (thanks Erik).

    Erik - thanks for the link to the io discussions, I'm reading now...

    Mike & others - thanks for the feedback!

    We have a really good story & haven't stopped with the work in this area. Stay tuned...

  1. 15  Ed Brill http://www.edbrill.com | 1/7/2010 9:14:27 PM

    @10 / 11, that's pretty interesting feedback. I wonder whether we've made that well-known.

  1. 16  Gregg Eldred http://www.ns-tech.com/blog/geldred.nsf | 1/7/2010 9:56:38 PM

    @11 - LMAO!

    "Your problem is the fact that, since the move to Eclipse, Notes has to load a bazillion tiny little .jar files, causing your drive heads to seek around like a starving hamster with ADHD trapped in a grocery store."

    I realize the year is only a week old, but that should be put on a "Blog Comment of the Year" list. :-)

  1. 17  David Bell  | 1/7/2010 11:38:14 PM

    @10 - are you starting the client up while the OS is still loading services and other apps (via Startup) ? That is just introducing disk contention.

    Are you virus scanning on access or on change and are .jar files included or excluded ?

  1. 18  Michael  | 1/8/2010 4:22:39 AM

    @10 : if you are a Vista or 7 user (but I think you can install on XP too) the "superfetch" feature will boost your client. On my laptop, with superfetch, a cold start is 5 seconds using "standard" hard drive (no SSD). A hot start is 1-2 sec.

  1. 19  Volker Weber http://vowe.net/about | 1/8/2010 4:44:29 AM

    Ed, by all means, send me the link. If the "FUD" is so obvious, debunk it.

  1. 20  Charles Robinson http://www.cubert.net | 1/8/2010 7:53:37 AM

    If you're going to throw a tantrum at least show us what you're ranting about. Daddy can't fix it if he doesn't know what's wrong.

    @3 - I call BS. The last Exchange downtime I had was when the building caught on fire in July 2008. The fire department cut power to the building and the UPS eventually ran out of power. The fire was on a Friday, Exchange was running normally on Monday. If it weren't for the building being a burned out shell and users not having any computers, they wouldn't have known a difference.

  1. 21  Ed Brill http://www.edbrill.com | 1/8/2010 8:25:30 AM

    Here's what Microsoft is saying publicly:

    "With Microsoft Exchange 2010 customers can choose cheaper, lower-cost storage types including DAS, SATA, and JBOD without sacrificing reliability or performance. IBM provides Domino attachment object store (DAOS), but customers are not provided with a true range of storage options to reduce costs for all users."

    It's a bit deeper than that in some of customer engagements.

    Of course, the reality of how Exchange does this is by suggesting that you store data three times if you are going to use generic disk, which somehow still doesn't seem like a way to get to a cheaper solution (and isn't exactly a place to attack since Domino has had that ability for 15 years).

  1. 22  Darren Duke http://blog.darrenduke.net | 1/8/2010 8:36:28 AM

    @6, I agree with the view index backup issue. Been a pain for ages.

    It is no secret I am a DAOS fanboy. But there are still some areas IBM can make in-roads and further the technology (shared storage, attachment life cycle and moving to SATA, local Notes DAOS, etc...) but overall DAOS has, and will continue to be a huge selling point for R8 and beyond.

    Disk is "just" a cost point if you are not buying them ;)

    @10, as @11 mentions, SSD is truly mouthwatering and will really change Notes client perceptions { Link } .

    @All, I hope this guest blog is a precursor to all the things mentioned at LS09 about the DAOS work Lotus were doing in the labs. If it is, MS can't come close, FUD, truth or otherwise.

  1. 23  Darren Duke http://blog.darrenduke.net | 1/8/2010 8:37:18 AM

    Oh, and FTI's are a pain too. Can I please move them to another drive? Please?

  1. 24  Charles Robinson http://www.cubert.net | 1/8/2010 8:50:07 AM

    @21 - Oh, so you're referencing this: { Link } . Based on that, I agree that Microsoft really isn't saying anything. The three things they list aren't even marginally related in terms of performance or reliability. It's not FUD, it's nonsensical.

    One of the biggest limitations we're facing is you can't store archives on a separate file system than the mailbox. You can't tier storage very well, which may be what you're referring to when you say you have to store Exchange data three times.

    So we agree. See how easy that was? :-)

  1. 25  Keith Brooks http://www.vanessabrooks.com | 1/8/2010 9:16:50 AM

    @3 its not true Exchange fails daily, monthly in some clients cases which I work around. BUT when it fails, it is down hard for some time, unless someone spent the big bucks. Even this is getting better for them as they now keep VMs available.

    Colin, thanks for the links too.

  1. 26  Tim Haugen  | 1/8/2010 1:13:24 PM

    @11 - Yep - SSD would be great -- but does IBM want me telling senior executives that to get acceptable performance from Notes, they'll need to purchase SSDs for 70,000 PCs?

    and - yep - I forget the exact counts, but the "bazillion" .jar files, etc. is in the 10's of thousands.

    @17 - with our testing, we specifically wait until all startup processing is complete, but for Real User scenarios, Yes, they start the client immediately, so that's the performance they're going to experience. We had one user whose specific experience we were trying to review - My guys told her, "don't start Notes when it first starts up. Wait until we tell you to... NO - Wait !!" -- She literally could not overcome her muscle memory/reflex action of launching Notes immediately.

    @18 - we're using Windows XP, and according to { Link } among others, SuperFetch is not available in XP (the Vista/7 Reg entry can be added in XP, but the code is not in the XP kernal, so it's not doing anything)

    - but -- we are looking at some custom code to specifically pre-cache the Notes files (send the hampster through the grocery store ahead of time).

    @22 - Darren - thanks - couldn't recall whose blog entry that was -- When I shared that with my folks and IBM folks engaged, that got them looking closer at the testing to isolate the drive speed impact (ghosting a system).

  1. 27  Mike McP http://www.openntf.org/Projects/pmt.nsf/ProjectHome?ReadForm&Query=mPortal | 1/8/2010 3:10:59 PM

    @26 - Yeah, I wouldn't recommend anyone mention SSDs as a fix for performance:)

    There must be some places to squeeze more performance out of the client. For instance, when I'm prompted for the password in the client, the app seems to just sit there and wait for me to enter it and submit....there appears to be no activity from any Notes processes, and the remaining startup time seems to be equal if I sit at the password prompt for 20 seconds or 2 seconds.

    Could we be doing more pre-loading, with the assumption that a user is going to enter the password correctly? You would think that since 9X% of the people sign in correctly on the first attempt, it would be advantageous to assume success is imminent and securely load as much as possible while the system waits for user input. That would be an easy way to give the illusion of quicker loading.

    Anyway, just go over that freaking thing with a fine-tooth comb, and bring in people from the outside to get creative.

  1. 28  Erik Brooks  | 1/8/2010 3:35:43 PM

    @27 - I think 8.5.1 does this (more preloading during password prompting.)

  1. 29  Christer Eklundh http://lotusblogg.atea.se/webblogg.nsf/ | 1/8/2010 3:52:49 PM

    I love my Notes client but to get rid of the bad rep Notes has among the uninitiated IBM must:

    1. Improve the performance

    2. Improve the performance more

    3. Make the user interface even more appealing. I really hope that Notes 8.x.x only is the start of an user interface that users will say "damn, that's one good looking application". Spend more on a even more good-looking user interface than new features.

    Use nice fonts. Notes 8.x does not look good on Vista or Windows 7. This is because I believe that MS uses a different stardard font for these operating systems. I have written about this on my blog. There are two images, the top is running Vista with Notes 8.5.1 and the bottom also Vista with Notes 8.5.1 but I've modified the font by changing the file com.ibm.notes.branding_8.5.1.20090929-1223. jar.

    Link to blog: { Link }

    4. Make it easy, reduce the number of settings the user can set.

  1. 30  Patrick Kwinten http://quintessens.wordpress.com | 1/9/2010 2:40:29 PM

    Dear IBM, can you let Notes proper handle deletion stubs? It seems for applications this seems to be a major cause of big databases.

  1. 31  Peter Wilson  | 1/10/2010 4:09:49 PM

    @29. Oh, and bring the Workspace into the 21st century. Why do we still have clunky 16 bit icon colors after 8 releases? ... that's just being lazy using all of that Notes 1-4 code days :-0

    Pete

  1. 32  Charles Robinson http://www.cubert.net | 1/11/2010 7:26:02 AM

    @30 - Can you clarify, how are deletion stubs not handled properly? You can e-mail me directly if you prefer, charles at cubert dot net.

  1. 33  Collin Murray http://www-01.ibm.com/software/lotus/products/domino/ | 1/11/2010 8:27:54 AM

    @30 - I'm interested in that as well. What behavior are you seeing?

  1. 34  Chuck Savino http://www.csavino.com | 1/11/2010 2:05:47 PM

    My reluctance to implement DAOS is simply about backup/recovery. Last time I heard, only Tivoli will easily fully restore a DAOS-enabled database.

    If you use anything else, restoring from incremental updates is a manually intensive nightmare, particulalry the longer a db has been on DAOS.

  1. 35  Collin Murray http://www-01.ibm.com/software/lotus/products/domino/ | 1/12/2010 10:12:12 AM

    Depending on the deferred deletion interval, you may not need to do anything except the database restore. The main concern is about DAOS attachments that are deleted once there is no longer a reference to it. There was a conscious decision made to leverage existing partner solutions for backup & restore to allow organizations to take advantage of the DAOS benefits right away. We have been discussing new APIs with vendors to potentially have a simpler solution, but I expect that the majority of cases for restore concerns are addressed with an adequate deletion interval.