Ed Brill

Post a Comment

1. 1  Oliver Regelmann http://n-komm.de/blog | 2/20/2009 3:23:19 AM

   While these are very impressive results, one must say the better these are, the worse the original user behaviour is.

   The way DAOS works the amount of disk space saved is directly related to the space wasted before by sending the same files again and again to multiple recipients instead of storing them in databases like document libraries or even on a network share.

1. 2  Lars Olufsen http://www.olufsphere.com | 2/20/2009 4:21:08 AM

   @1 - I think you're right regarding user behaviour.

   The above results document that bloated databases and bad user behaviour is a very REAL problem in many organisations.

   But very often the problem is based on the lack of better tools for the job. If a company has no structured way of handling files and documents, the user will do things the easiest way they can.

   Implementing archiving tools like DAOS and/or Commonstore (sorry ... Content Collector) is only the reactive way of addressing the problem. The proactive solution is equally important. Can Quickr be your friend here?

   Back on the topic - these results are very impressive, especially considering that DAOS doesn't require new infrastructure and expensive licenses. Very, VERY interesting! :-)

1. 3  Oliver Regelmann http://n-komm.de/blog | 2/20/2009 4:30:12 AM

   @2: "DAOS doesn't require new infrastructure"

   That's not entirely correct. DAOS needs transaction logging and transaction logging needs its own disks and/or controllers (well, at least it's highly recommended). This can be an issue especially in SMB environments.

1. 4  Lars Olufsen http://www.olufsphere.com | 2/20/2009 4:38:14 AM

   @3 - True of course. Sorry.

   Still ... no new servers (like Content Collector) or licenses. :-)

1. 5  sean cull http://www.seancull.co.uk | 2/20/2009 5:15:02 AM

   I have often wondered about the question of needing a separate spindle for transaction logging in production environments.

   In the case of an SMB which has separate spindles for the program files / OS and a separate spindle for the data would using the program files /OS spindle for the transaction logs not be a good half way house - how IO is there on the OS / program files side in steady state operation ?

   There is a danger that the separate spindle preference is being seen as an absolute requirement when this may not be the case in a SMB with a lightly loaded server

1. 6  Ports http://www.mrports.com/ | 2/20/2009 6:30:35 AM

   @5 It would not really be a good idea to reuse the Program File / OS spindle for the transaction logs. The reason being that the OS disk use is random access (in other words the read/write head is jumping all over the disk platter) whist a trasaction log writes serially (esentially it starts writing at the begining of the disk and just continues like an old LP Record). The performance of both would be significantly compromised by mixing these two usage types on the same disk.

1. 7  George Paglia  | 2/20/2009 8:04:04 AM

   In regards to the separate spindle, is transaction logging supported on the Foundation server with a separate spindle? Or is it done differently? (please forgive my lack of knowledge)

1. 8  Glen Urban  | 2/20/2009 8:12:46 AM

   regarding hardware requirements for transaction logging.

   When I specified the hardware/OS requirements for my Domino server I asked our server/OS guy to setup transaction logging to a seperate RAID1 with it's own disk controller. Months later he realised he'd set it up wrong and logging was taking place on the same RAID5 disk set as the the Notes data dir.

   I'm not reccomending you do it this way but there was no noticable effect on performance (about 250 users on this server). However, I cam imagine on high load/high use servers it would have a significant impact on performance.

1. 9  Glen Urban  | 2/20/2009 8:12:46 AM

   regarding hardware requirements for transaction logging.

   When I specified the hardware/OS requirements for my Domino server I asked our server/OS guy to setup transaction logging to a seperate RAID1 with it's own disk controller. Months later he realised he'd set it up wrong and logging was taking place on the same RAID5 disk set as the the Notes data dir.

   I'm not reccomending you do it this way but there was no noticable effect on performance (about 250 users on this server). However, I cam imagine on high load/high use servers it would have a significant impact on performance.

1. 10  keith Brooks http://www.vanessabrooks.com | 2/20/2009 8:14:09 AM

   Ed, I posted my findings back on Jan 8th.

   { Link }

1. 11  Irv Schor  | 2/20/2009 8:47:17 AM

   @3 - This is indeed what I had to explain to management, who had previously read all the initial reports of huge disk space savings, etc. without understanding the architectural changes needed for it to happen. One thing we could do, however, was enable compression on our mail files and upgraded from ODS v43 to v48 which saved about 15% across the board (very easy to do).

   The following knowledge collection may help the SMB's thinking about enabling transaction logging. { Link }

1. 12  Thomas Schulte http://www.welovenotesut.com/blog | 2/20/2009 9:12:22 AM

   @1 and @2 Yes we know that the users are misbehaving. ut dont dare to believe that you could change your users attitude and them "living" in their mail file by throwing apropriate tools at them.

   The company i work for has done that as you can read in my post. Our users have every possibility to use central file systems. Teamrooms. Document databases and even quickr.

   THEY SIMPLY WONT DO THAT.

   And believe me IT IS FAR WORSE than that. Most of the files found with DAOS will be in the file systems too.

   Perhaps if you would put an evil looking guy with a baseball bat behind them, who tells them with an sarcastic smile on his face and a dark rough voice "this will have consequences", everytime they are saving something in their mail file, this behaviour perhaps could be changed.

1. 13  Tripp Black http://www.mindwatering.com | 2/20/2009 11:17:10 AM

   We have found that often there is no such thing as a separate RAID or bus anymore. The SMBs are rolling into VMs for the ease of backups and restores of VMs compared. The separate drive is simply another big honking file either in local storage or on an iSCSI or less often fiberchannel.

   As the drives are virtual, if VMotion is not going to be used, they can use local storage instead of a iSCSI storage for the translogs. Obviously, that saves some iSCSI network traffic. If that is not an option, they typically will setup a disk that actually uses a local (on that server) drive. (Typically the SMBs only have 1 or 2 VMWare servers and have a starter pack license w/o vMotion.)

   The result is that the load difference for 250-500 users is not noticeable when translogging is enabled. (You will see increase on the iSCSI network though if you are not using local storage.)

   When the server is still on "iron" rather than in a VM, we have seen little difference on a client with about 150 users where the translogs were on a xseries 1U server w/local drives in RAID 1. The translogs were simply a different partition. If fact, as typcial w/translogging's delayed writes, the user exerience was actually a slightly zippier server.

   That said, we have not turned on DAOS for these clients. Their desire for translogging was primarily faster startup.

   Thanks,

   Tripp

1. 14  Tim Rand  | 2/20/2009 11:36:35 AM

   Nice collection of "results" but be aware that some of these are the DAOS Estimator values, not the true "before vs after" comparison.

1. 15  Darren Duke http://blog.darrenduke.net | 2/20/2009 12:49:45 PM

   Mine from Jan 6th { Link }

   Was that the earliest (at least for 8.5 gold)? Do I get a prize? ;)

1. 16  Henry Ferlauto http://www.geniusinside.com | 2/20/2009 1:58:47 PM

   I would hope that IBM publicizes this in many ways. With the economy not doing well and this providing real, measurable savings IBM should be able to get some very positive buzz going.

   I could think of a couple of good ad concepts for a viral video based on this.

   e.g. Big boss tells CIO / IT Director to cut costs. CIO then works with his team and strikes a line in budget through "extra storage for mail servers." (As opposed to something more important like new software, raises, pizza, etc.)

   Team thinks he's suicidal because every employee will want his head on a platter for the new, smaller mail quotas...... You can figure out the rest...

1. 17  Carsten  | 2/20/2009 2:48:57 PM

   It is not a bad idea, but what company are going to stop migrating to Exchange, because they can save a few disks. (Read: are you kidding, just buy a few extra disks or expand your SAN)

   I suggest focussing on elements that actually encourage companies to stay on Domino and to attract new customers to Domino.

1. 18  Henry Ferlauto http://www.geniusinside.com | 2/20/2009 5:41:34 PM

   @17 - That same message could also be targeted at the existing customer base to get them to upgrade. Once a customer is on ND 8.5, the likelihood of migration off Domino is dramatically reduced for all the user interface reasons.

1. 19  Vincent  | 2/20/2009 10:00:22 PM

   Although we are rolling out 8.0.2 (FP1), 8.5 is already planned in our next phase. The DAOS estimate tool give us, so far, 23% saving. Consider we are talking about 120GB mail size, anything beyond 15% has significant impact on archiving.

   I am glad to hear that IBM are still doing more performance tuneup on 8.5 client and beyond.

1. 20  Henning Heinz  | 2/21/2009 4:08:59 AM

   @17 Carsten The problem often is not the few extra disks but the daily backup. I am still surprised how many companies do full backups of the Domino data directory and not using an incremental nsf backup agent. That means if the mail files are 120 GB they have 120GB of data saved every day.

   DAOS will play nice with traditional backup software so even if you do full backups your attachments are now stored incrementally. I still would like to have DAOS work without transactional logging.

1. 21  Carsten  | 2/21/2009 9:03:57 AM

   @20. Agree, and sure it is important to keep your databases down in size .... it also speeds up the server, indexing etc. Backing up "just" 120 Gb of mail databases is trivial.

   It is a technical argument how many disks you need in the server to store your mail databases, and not an argument that would win the hearts of Exchange users.

1. 22  paul http://no web | 2/24/2009 4:46:21 AM

   OK, estimation data sounds good but people don't see what will happen after DAOS is implemented.

   You will typically setup the prune period to 30-60 days (it is recommended by IBM due to backup issues). It means that in these 60 days you will have in your disk many many attachments that should not be there anymore. These attachments come from deleted emails and from SPAMS (even if your antispam software deletes the email, DAOS will put the attachments there).

   So it will in practice increase the disk space and you will backup garbage.

   Transaction logging is needed because in case of a crash the DAOS resynchronization could take literally days.

   My boss asked me to implement DAOS but after researching in deep about it, we talked and we decided not to go ahead.

   Anyway I agree that the estimator results shows how "uneducated" the users are... even with DAOS if users send big atts to many many persons it will mean loss of bandwidth thus server performance issues. And DAOS estimator does not show emails sent to internet!

1. 23  Andrew Pollack http://www.thenorth.com | 2/24/2009 7:54:44 PM

   One of my (many) pet peeves is how I.T. people forget that the end users and the business they do is the reason for our work -- not the other way around.

   Earlier in the comments, a few people point out that the big savings from DAOS results from poor behavior on the part of the users.

   I disagree. Users just want to do their work. The way they're working hasn't been well matched by the tools we've provided them. To deal with that, we've had to impose arbitrary limits to how they work. DAOS is an advancement that helps us get rid of the need for those arbitrary limits and rules.

   Using DAOS takes us one step closer to matching the technology to the needs of the users --- our job.

1. 24  Andrew Pollack http://www.thenorth.com | 2/24/2009 8:05:54 PM

   @Paul -- you're as wrong here as you are on every other site you're posting this stuff on.

   I'll bet you a steak dinner at Lotusphere 2010 (the great steak restaurant at the Y&B is my favorite there -- plan to spend around a hundred bucks for dinner for two) that DAOS, properly installed in any "reasonably typical" Domino/Notes mail environment of more than 50 users that doesn't have 'draconian' limits on mail file use, will substantially reduce disk space usage and disk I/O traffic.

   Care to take me up on that bet? We'll let the community be the arbiters of what "draconian" and "substantially" mean. If you think you can find real world -- not contrived -- examples to prove me wrong and its a matter of interpretation, we'll post and let the community decide.

   I'll put my 15+ years of working with this stuff to the test on this one. I think most people who know my work will attest that performance is pretty high on my agenda. How about you?

1. 25  Darren Duke http://blog.darrenduke.net | 2/25/2009 3:07:42 AM

   @22, why such a long prune? The prune should be your last full backup timeframe + 1 day.

   For instance, if you do weekly FULL backups your timeframe = 7 days. Therefore your prune should = 7 + 1 = 8.

   FWIW, I agree with Pollack. There are very few circumstances where DAOS will not provide significant I/O savings. And TX Logging SHOULD ALREADY BE ENABLED for any production system anyway, so that fact that DAOS "requires" this should be arbitrary.

1. 26  paul http://no web | 2/25/2009 5:16:53 AM

   @24: When I talk I give strong technical arguments like "even if you have an antispam the attachments are put in the DAOS repository", or "does it makes any sense to include in the spam repository emails sent from one internal user to outside and this attachment is not stored in the sent folder" and many other arguments.

   When you answer you give answers like "hey, I have more than 15 years of experience and DAOS rocks!"

   So next time you answer to my questions, pls answer with technical arguments refuting my arguments... else your 15 years of experience may come from nothing... I also have 15 years of experience in programming but I don't mention it each time I post...

1. 27  Andrew Pollack http://www.thenorth.com/apblog | 2/25/2009 5:37:12 AM

   @26 : Paul, because your argument isn't based on facts - its speculative based on incorrect assumptions of how the product works and what the real world use files is. I pointed this out here: { Link } Misquoting me won't change that.

1. 28  Kerr  | 2/25/2009 6:08:51 AM

   @26, Paul, why would you DAOS enable the mail.box where your spam filtering takes place? Don't enable DAOS on your spam filtering mail.box if you let spam get that far. Why would you wait 59 days to do a full backup? Back up more frequently. Your users habitually send round hundreds of mails and then everyone deletes them? Seriously?

   If your determined to contrive situations where DOAS doesn't make sense then, go ahead, well done, but what's the point? Trolling? Outside these weirdo scenarios you're coming up with DOAS saves space and lots of people are seeing it.

1. 29  paul http://no web | 2/25/2009 7:21:38 AM

   @28: Outside these weirdo scenarios you're coming up with DOAS saves space and lots of people are seeing it

   >People are seeing what the daos report says... but this is the instant savings they get... this report doesn't say what will happen after you enable daos.

   After you enable daos, there will be many atts in the daos repository that are not really needed because user deletes them, or because they were spams, or because user sent an email to outside with atts and the user didn't save this email, and other many situations.

   So you will have these files there, not deleted...

   and you will also backup these unused files!

   What would be great is a product that will tell the admin how much MB or GB was wasted in deleted atts in the DAOS repository just before you make a backup...

1. 30  Kerr  | 2/25/2009 9:06:19 AM

   @29, Paul, actually I'm talking about people running DAOS in real production environments. Which is more than you are doing. You are speculating about what DAOS will do in a scenario quite unlike anything I've ever heard of and then declaring the feature a dud. Others are actually running it in production and seeing good space saving.

   If you were actually running it in a production environment and reporting on what you actually saw and took advice on how to configure it for optimal performance, then you might have someone listen to you. As it is, you sound like a troll.

1. 31  Ed Brill http://www.edbrill.com | 2/25/2009 9:13:06 AM

   @29/30 Paul, it would certainly be helpful if you provided a full name and valid e-mail address per this blog's policy.

1. 32  paul http://no web | 2/25/2009 11:45:34 AM

   @31: I provided a valid email address...

1. 33  Ben Rose http://www.jaffacake.net | 2/25/2009 12:07:25 PM

   Please don't feed the troll!

1. 34  Russ Holden  | 3/10/2009 12:30:45 PM

   Obviously the numbers that Ed is showing are spectacular. But please remember that DAOS is about a lot more than disk space savings. Moving a large amount of (generally) low-usage content out of mail files allows use of lower-cost disks for that content with potential savings that makes the disk space-based savings look boring. And we don't have to argue about whether users are good or bad.

   We also allow for huge backup cost reduction - DAOS (".NLO") files are read-only after creation (excepting deletion).

   We also get huge reductions in mail file maintenance operations - notably compact.

   So, for me, disk space savings is a nice (side) benefit of DAOS but not the main point.