Ed Brill

Post a Comment

1. 1  Bill http://www.billbuchan.com | 12/3/2008 2:51:57 AM

   I found that the Exchange object store was often thrown in my face. As usually everyone in the same object store was in the same workgroup, then only one copy of a binary file would be kept. And this hasnt changed since Exchange v5.5 - back in the late 1990s'.

   However, how does that work in large scale environments ? Well, a recent Domino 6.5 -> Exchange 2003 migration took mail file usage from 13th (25k users) to 57tb. Eh ? How did that work ?

   Well, in order to get Exchange reliable, you've gotta keep those object stores small - after all, *when* (not *if*) they corrupt, it takes over an hour per gigabyte to recover. So although they claim that you can get 72gb into an object store, a certain degree of insanity is required to do so.

   So what we've seen in medium to large sized environments is that in order to even have some idea of stability, its a case of a low number of users in a high number of object stores. Which means that it gets increasingly less likely that you will have duplicate files to save space on.

   So Domino - up to v8.5 - makes no attempt to save space, gives you fantastic reliability. And it transpires, against a similarly configured Exchange system - takes less space in any case. And this is without expensive site-specific shared SAN drives and Clarion subsystems. (Dont try and 'cluster' exchange servers over a WAN, it would appear)..

   Domino 8.5 DAMO is a game changer in this respect - giving you single file storage across ALL databases (potentially) on your server. So you get both massive reliability, *and* saved disk space.

   Oh, we all say, 'Disk Space is Cheap'. Ah. No. Backup Window isn't cheap. Backup tape isn't cheap. High performance SAN definitely isn't cheap.

   I think the old Garner Advice - 'dont bother migrating' - is now swinging in favour of Domino - especially as far as managed operations are going.

   I'd love to see IBM Global Services - who host a *lot* of customers - upgrade all of those customers to 8.5 in the near future and publish stats on disk/tape savings.

   Its about time the rest of IBM got behind Lotus for a change, eh ?

   ---* Bill

1. 2  Adam Osborne http://www.preemptive.com.au | 12/3/2008 3:55:22 AM

   Bill is spot on that 8.5 with DAOS - it is a game changer. We have had similar functionality in the Preemptive Email Vault for a while now, and it saves a massive amount of storage.

   The key is how are we (the Lotus community) going to get this message out there ? We need to get mind share back so people consider Domino 8.5. I'm sick of people saying we are going to Exchange, oh it is a political decision - pure madness.

   I'm really looking forward to seeing how IBM is going to market this fantastic product.

1. 3  Jan Van Puyvelde  | 12/3/2008 4:04:26 AM

   >> you could turn on single copy template

   What is the IBM advice ? Use only if really needed or is it very stable and recommended to use ? What % of customers use it ?

1. 4  Olaf Boerner http://www.bcc.biz | 12/3/2008 5:23:14 AM

   We all have to communicate DAOS as a to the market that this is a great feature and not Shared Object Store 2.0.

   We do this with our customers.

   But as always we BPs can only enforce an IBM market communication.

   And saving 40% diskspace when moving to Domino is nice on.

1. 5  Jim Casale http://www.jimcasale.net | 12/3/2008 5:51:11 AM

   It's a bummer that this feature will probably not be taken advantage by organizations with high compliance needs (unless it can be given a seal of approval by auditors,etc). It would sure help in my organization since we have several servers that are closing in on 1TB of data each - most of which I was say is attachments.

1. 6  Nathan T. Freeman http://nathan.lotus911.com | 12/3/2008 7:30:12 AM

   @5 - "It's a bummer that this feature will probably not be taken advantage by organizations with high compliance needs"

   Why would you say that? I honestly can't think of any way that DAOS or non-Summary compression relates to compliance requirements.

   @3 - I've used SCT in production before, but it's pretty rare in the wild. Particularly with Design Compression enabled, you only get about 15MB of space back per user -- which isn't usually worth the trouble (even when you account for costs of backups.) I'm not even sure it's worth mentioning here.

   The real advantage to SCT, in my opinion, is that you can rollout template changes instantly, instead of waiting for the Design task to update thousands of individual NSF design elements.

   It's great to see the progress the Domino server team has made on reducing cost of ownership by using disk space more effectively. Backup and restore times are really becoming the bane of data center administrators, and Lotus is one of the few companies in the industry that's actively pursuing storage savings like this.

   Less drive consumption = less energy consumption. DAOS is GREEN, man!

   And you know what's even cooler? That there's still more improvements available. For instance, moving FT indexes out of the Domino directory and into a configurable path could help a lot of operators. Or moving the $Collection object for a view outside the NSF itself, and allowing those objects to be stored and updated directly on a drive. We certainly haven't seen the end of what's possible with NSF storage reduction. I can't wait to see what the team does next. :-)

1. 7  Keith Brooks http://www.vanessabrooks.com | 12/3/2008 8:10:49 AM

   @6 are we going green on top this year in place of yellow?

1. 8  Darren  | 12/3/2008 8:11:22 AM

1. 9  Darren http://www.dadams.co.uk | 12/3/2008 8:18:12 AM

   Ooops... I should turn the heating on, my hands are cold and hitting keys that shouldn't be hit.

   This discussion is very interesting as I'm just writing some words for a proposal that discusses Domino's consistent roadmap and addresses how an existing customer can reduce cost as a result of upgrading and tidying up the infrastructure. Approx 30% space savings with the 8.0.1 compression, and then DAOS to come (not DAMO Bill, that's something else) - they have a £££ cost attached to every gigabyte so it's quite easy to quantify.

   I'm sure that Microsoft are working on ways to drive down storage requirements and costs too... if only they had a roadmap... c'mon Microsoft, even I'm bored with saying that now.

1. 10  Dan Holzrichter  | 12/3/2008 8:22:03 AM

   I don't think you save that much with the single copy store in exchange for it to really make a difference. I'm pretty sure that it only applies to the same people in the same database or storage group so it's not really single copy unless the people you send to happen to be on the same server/storage group/database. When you consider that mail converted over to exchange from Notes uses 10%-15% more storage space Notes is a clear winner.

   What Lotus really need to hit back at microsoft with is disk I/O performance. From our experience going from domino 7.0.2 to exchange 2007 the disk spindle requirements are substantially higher. I know our spindle count is much higher than it was with domino and performance is still a problem.

1. 11  Darren Duke http://blog.darrenduke.net | 12/3/2008 9:48:34 AM

   Thanks for the props Ed. Wow, that was a can-of-worms post ;)

   Regarding SCT, I've never seen it used in production. ONe of these days we will enabled it in STS and see what happens. It IS NOT enabled on the server that the blog posting is about. With 8.0.1+ compression the design size goes from 25MB to 9MB in most cases, that is usually good enough for me.

   @1, there was a post on the blog indicating just that, "disk is cheap". Not if you have an IBM blade with SAS drives ;). Anyway, your point about backup times/windows is where the selling point is here. Imagine reducing your backup time by 30-40% just by upgrading to 8.5! Bye bye 6.5 and 7.0, hello R8.

   @9, 30% is high for the 8.0.1 switch, real world you will see 7 to 20% depending on the db content { Link } but it is "free" so use it. Once we get 8.5 gold I'll also see what % reduction we get with that in the real world.

1. 12  Oliver Regelmann http://n-komm.de/blog | 12/3/2008 9:49:33 AM

   I once migrated a some 30 users shop from Exchange (which exploded reaching its 16GB license limit coincidentally that day) and had 12GB Domino mail data afterwards.

   @6: I have only one customer using SCT. But I'd say that 15MB space saved per mail file is 15% more space for every user when the usual quota is at 100MB.

1. 13  Charles Robinson http://www.cubert.net | 12/3/2008 10:17:58 AM

   I'm going to be doing a test migration at work to see how well Domino would work. I'll be sure to share what I find. We're a very small company, though.

   @3/6 - I have used SCT in production on mail files for a couple of years. I honestly don't know why more people don't. It's no trouble at all to implement and 12 - 13MB per user in space savings is nothing to sneeze at.

   @1 - I think you mean DAOS, not DAMO. :-) And you're right, that is what Shared Mail should have been. It would behoove you to spend a few minutes learning about Exchange before you bash it. Your terminology and numbers are way off. :-)

1. 14  Henry Ferlauto http://www.geniusinside.com | 12/3/2008 10:57:06 AM

   @6 & 7 - We can steal the old GM t-shirts from last year, "Yellow is the new Green." :)

   This is a great piece of news. It would be good to see more examples of this (to be statistically relevant).

   Additionally, I would like to see IBM go after the other half of the hardware equation, the CPU. Meaning how many (Exchange) server farms could be replaced by 1 or 2 (always need a backup) really large pieces of IBM hardware (e.g. i, p or zSeries)

   And even on the more modest scale, what can Domino on Linux on IBM blades offer over Windows on any hardware platform of choice?

   And it's not just the hardware itself but the cost to maintain it all. Physical space, electricity, and of course number of bodies to care and feed it all.

1. 15  Darren Duke http://blog.darrenduke.net | 12/3/2008 11:49:23 AM

   @14, until Domino Linux gets 64 bit support it is difficult to convince IT depts to go with it. I was hoping to see that 64 bit support come in 8.5 but the last I checked it was only Linux on Z. It is good for Sametime thought and I would love to see "Domino" Quickr on Linux too.

   @all, I guess the consensus is to enable SCT then. Free space is free space ;)

1. 16  Randy Bye  | 12/3/2008 11:52:20 AM

   @3 & 6 & 13 - According to IBM technote # 1140911,

   "The DWA7.ntf, iNotes6.ntf, iNotes60.ntf, and iNotes5.ntf template designs all contain the Database property to indicate that the template can be used as a "single copy template". Therefore, these templates can be used or selected for use in this manner. However, IBM does not recommend use of SCT along with Domino Web Access because of browser, HTTP, and server performance concerns that may arise with its use. Please consider these areas if you plan to implement one of the iNotes templates with the SCT feature."

   I've heard this recommended over the years and always refer people back to this technote. If you aren't using iNotes it's probably worth trying.

1. 17  Bill Brown  | 12/3/2008 2:13:15 PM

   @14, why not compare apples to apples. How many mailboxes can the same x86 hardware support in Exchange v. Domino.

   Calculating the energy use of moving to a different platform (a choice only available to Domino) would be an interesting exercise, but I suspect that it wouldn't make much difference in most shops since they already understand X86, Windows, and perhaps Linux on that platform. Learning an entirely new platform would be too steep a curve.

   Many shops don't have redundant big iron (iSeries or zSeries) systems unless they are really, really big. Those things just don't seem to break!

1. 18  Erik Brooks  | 12/3/2008 2:40:06 PM

   @Ed - the original blog's spam filter ate my comments, so I'll post here.

   Re: FT index storage, view indexes, etc...

   I'm not an expert in Exchange indexing/searching. My "Apples to Apples comparison" remark was playing Devil's Advocate. If this 40% stat turns into an IBM headline (and I hope it will), it needs to be defendable with respect to functionality and impact of true space -- not just "the mailbox", which can be defined as somebody sees fit.

   For example, if I have a mail db with 5GB of attachments and 20 MB of data and I go to 8.5 with DAOS, I could claim "Domino 8.5 reduces mail files by 95%+" since the attachments are now "out" of "the mailbox." But that's a bit misleading, don't you think? I would think that an MS rep could deflect that argument pretty easily.

   By "Apples-to-apples" would mean that all user-initiated indexes and functionality are built and available. I would think that an appropriate comparison would be to open Outlook and click once on every user action (view/open/search/etc.) Then repeat in Domino, and look at the total *disk* space consumed.

   The amount of space reduction available to 8.5 is staggering, though:

   - DAOS

   - Non-Summary item compression

   - Design compression

   - View sort index deferral

   - and don't forget good ole' "Use LZ1 compression for attachments!"

   ...that's a pretty hefty list.

1. 19  Henry Ferlauto http://www.geniusinside.com | 12/3/2008 4:47:49 PM

   @17 - I agree with that too. But it has been the large "household names" that have been migrating away from Domino with some making economic arguments.

   So I would say do both.

   One would think that on the very large scale (greater than 10,000) users, that IBM should be able to pull away by considerable margins on whatever platform of the customer's choosing.

1. 20  Ed Brill http://www.edbrill.com | 12/3/2008 4:50:13 PM

   @18 I have found that it is much better to make a strawman assertion and have it attacked then not to make the claim. IBM Legal will make sure that if/when we do so, it's based on realistic representation of the possibilities. Then, you'll know if we start making such a claim as "Domino 8.5 can save on disk space by 40%!" that it is supportable and legit.

1. 21  Paul Mooney http://www.pmooney.net | 12/3/2008 11:09:30 PM

   Guys, if anyone reading this is interested, there is an SNT session at lotusphere that will show people how to deploy DAOS in an existing environment, step by step, picture by picture.. with discussion on pros/cons.

1. 22  Mark Dowling http://cork2toronto.blogspot.com | 12/4/2008 10:40:52 AM

   At the moment, it's difficult to get excited about new features in Notes/Domino like DAOS because one fears that by the time deployment happens, caveats will mean that the feature isn't useful to one's own site while everybody else is blogging about how awesome this all is.

   During the lead-up to 8 we were very excited by the Standard Client and Out of Office as a Service and it turns out we can't use either because the former doesn't support Domino roaming in the 8.0 codestream and the latter isn't supported by Blackberry. Now we're on D8 and wondering why we bothered, which is not a good thing to be thinking as an admin.

   (And yes, I know OOOS BES support is RIM's fault but given that we're not going to toss our 100+ Blackberry and BES investment, the loser ends up being Domino functionality. How many other enterprise environments have had to make the same call we did on that one?)

1. 23  NeilT  | 12/4/2008 1:21:27 PM

   I'm sitting here wondering about the whole "Validated" argument. As I understand it DAOS takes the files from the mail and stores "1" copy on the file system of the Domino server.

   Questions:

   - What happens when you send an encrypted document to 10 people on the same server?

   - Who has access to these documents and can you use EFS to keep them secure? Where would the certificate reside and who would have access to it?

   - Where does the much vaunted 7 layer Notes security access model go when these files are dumped on a Windows server drive?

   - What happens when some admin deletes the attachment by mistake and the original sender has deleted the original? Unless someone has a local replica the information is lost.

   OK you can restore from tape, but what happens when it's between backup windows (same day). Can you recover from translog or is it lost?

   Acting As a FDA auditor (I'm not but I have to think like one), I would have some very serious questions about this.

   It sounds very good, but there are reasons why many architects chose not to deploy SCOS or other technologies to achieve this kind of saving. Without a very good answer to each of these questions above, I would be extremely wary of deploying DAOS in any environment where the penalties for "Seemingly" losing sensitive data (or opening it up to compromise), could be in the tens or hundreds of millions.

1. 24  Nathan T. Freeman http://nathan.lotus911.com | 12/5/2008 4:49:00 AM

   What happens when you send an encrypted document to 10 people on the same server?

   This is a good question and I'm curious to know the answer myself. But the worst case scenario is that it stores the attachment just as Domino has for the last 20 years: duplicated in each person's mail file. You're no worse off than you were before.

   The best case scenario is that because an email to 10 people actually generates a single symmetric key for all 10 of them and seals that inside the message with their individual public keys, there really is only one version of the attachment that you need to store. Because DAOS is integrated down to the ODS level instead of being application level, it might very well keep a single copy of that file. It's not like the DAOS engine even looks at the contents anyway. It would be a BLOB just like any other.

   Who has access to these documents and can you use EFS to keep them secure? Where would the certificate reside and who would have access to it?

   There's a checkbox on the configuration screen, on by default if I remember correctly, that causes the server to encrypt the DAOS object using the server ID before the single copy is written to disk. So if your server ID is secure, the DAOS repository itself is also secure, even with physical access to the drive.

   Where does the much vaunted 7 layer Notes security access model go when these files are dumped on a Windows server drive?

   If you've encrypted them, then the contents can only be read when you request them through the process of the Domino server. In which case, the security model applies just fine.

   The process is almost entirely like encrypting local NSFs on a server. Except it has less impact on performance.

   What happens when some admin deletes the attachment by mistake and the original sender has deleted the original? Unless someone has a local replica the information is lost.

   What happens when some admin deletes a user's mail NSF by mistake on the server? Unless someone has a local replica, you have to go to backup tapes. You do have backup tapes, right? Right?

   So the real question is: why do you have admins deleting files from the Domino server via the OS?

   Can you recover from translog or is it lost?

   I don't know. But randomly deleting files from the server OS is a practice I would think would get an FDA audit result of: EPIC FAIL.

   Without a very good answer to each of these questions above, I would be extremely wary of deploying DAOS

   People find all kinds of creative ways to avoid saving themselves money.

   More info on common misconceptions about DAOS can be found here { Link }

1. 25  dan lynch  | 12/5/2008 10:22:04 AM

   @17 "....How many mailboxes can the same x86 hardware support in Exchange v. Domino"

   One key factor in this calculation is whether or not the Exchange environment uses BES or Winmmobile. Microsoft has publicly stated, here, in April of this year, that a BES user against exchange places the load of 4 normal Outlook mapi users in the box, therefore from a server load perpective that has a dramatic impact on the number of mailboxes Exchange vs Domino can host on identical hardware. Because Domino and BES get stacked on the BES server, the impact to the Domino mailbox server in a BES scenario with a client using BES is negligible as compared to Exchange/BES scenario.

1. 26  NeilT  | 12/8/2008 2:20:58 PM

   Thanks for the Input and the Nathan.

   Sadly in life we know that server admins do crazy things. There appear to be some well thought out processes here, I'd love to see that the work had been done completely end to end.

   As for Admins deleting the files on the server via the admin client, yes, it does happen. However those Admins tend to have flat spots on the back of their head and tend to avoid it. Deleting unkown files on the server outside of Domino is commonplace. Something that a whole world of experience will have to be generated with DAOS.

1. 27  Darren Duke http://blog.darrenduke.net | 1/2/2009 2:56:25 PM

   OK, so I guess I started this so lets see about ending it. DAOS is indeed all that IBM promised. On just 2 (two) email files in Domino we saw a reduction of almost 40% in mail file size (OS file size).

   See for { Link } details