Jack Dausman: Exchange Storage Is Not So Neat and Tidy

August 15 2006

The only favorable argument I've ever heard for Exchange's single instance store is disk space savings... MS proponents often claim (without data) that this is some kind of huge competitive advantage.  While I've tackled the subject generally in places like my Boss Loves Microsoft presentation, nothing's as good as real-world feedback.  Jack Dausman takes a look at this issue, in part based on an actual migration:

The Lower CPU usage I can explain in terms of the code rewrite for Domino 7, but using less disk space was completely unexpected. After all, I have always heard that the single-store architecture of Exchange provided the benefit of reducing disk resources (and, Domino has a single-store option as well for the same reasons).
Jack then uses the context of an article in Storage Magazine to find out why Domino is actually more efficient at disk usage than Exchange (go to his site to read the quotes from the article itself):
we simply run the compactor task to recover the data released from archiving. Doesn't Exchange have a compact task? Yes and no. For Exchange, the mail service has to be taken off-line for compaction.
Other reasons I think the Exchange guys are missing the point on single instance storage:

- Typically, Exchange doesn't scale to as many users per server as Domino, so there are more overall servers to deliver the message to
- Exchange servers have "storage groups" which means that messages are still written more than one time per server.  It's really the worst of both worlds -- a shared database that affects multiple people when there are problems, files that need huge amounts of memory to manage their I/O, and lots of contention on those files.
- While everyone loves to cite the example of the 10MB message sent to 500 people, these messages are the exception, rather than the rule, in any mail system..  The majority of e-mail messages are sent to one or two people, who may or may not be on the same mail database.  Have you ever looked at the tools MS provides to decide which storage group to assign a user to?  Oh wait, there aren't any.

Link: Jack Dausman: Exchange Storage Is Not So Neat and Tidy >

Ed Brill Category:  Microsoft  Add/View Comments [35]

Post a Comment

  1. 1  NeilT  | 8/15/2006 12:28:33 PM

    Yes Exchange looks pretty and is easy to get into at the surface and to be honest if you only have 25 users and one internet connection that's all you will want.

    Domino on the other hand is a completely different proposition and somewhat daunting at first look. So it gets bad press.

    I would be very cautious about the "less disk space" thing. Yes I would expect the disk usage to be somewhat similar with a brand new migration. Maybe not if the users are addicted to huge mails with attachments sent to many users, but as Ed says that is not so common a scenario as you would believe.

    However take that same server, issue a "load updall -r -c" and you will see the space utilisation grow in a spectacular way. That is because the new files will have no indexes. As soon as the indexes are built, you can kiss goodbye to a large chunk of space which will never come back again until you do a compact -d. But the first thing that will happen is that the indexes will come back again as you can't use the databases without them.

    So the question is "Does it matter?" To my mind, no it doesn't. Disk space is cheap and a properly built server can give way more performance than any Exchange server. But there are limits and Quotas should be used to retain sanity. Or your server will die a slow lingering death.

    The biggest surprise I have found, looking at a similarly sized company doing similar work with similar global distribution, is the lack of servers in Exchange. More surprising still is the very low number of Exchange 2003 servers compared to a similar mature Notes environment.

    I do migrations for a living. I don't see huge space savings from Exchange to Notes. The opposite in fact.

    What I do see is that Notes/Domino, with a sensible deployment can out value Exchange x. Also done stupidly it can overvalue Exchange x.

    Why is everyone so desperate to prove that Domino doesn't have weaknesses? It does, live with it. It is a superb product, push the strengths. I constantly have to respond to articles saying Domino is good in an area where experience shows it is weak.

    You have a fantastic product Ed, push it's strengths.

  1. 2  Ben Rose http://www.jaffacake.net | 8/15/2006 1:54:45 PM

    NeilT,

    I have never, ever seen anyone issue a "load updall -r -c" on a production server...it's non-sensical.

    Either you have indexes and they need rebuilding and you use -r or you don't and you need to create with a -c. These switches just don't have much purpose being issued together.

    "Updall -c" alone should be issued sparingly. Yes, on a huge database with thousands of documents you may want to pre-create indexes for that specific database but as a general command...never.

    View indexes can be set to be discarded over time for a reason...they aren't in use. There's no point in indexing, and re-indexing, views that a user doesn't use regularly if at all. On my user mailfiles, I estimate that only 25% of the views are in use in the production environment. All the others have either never been built or have long since been discarded. Issuing an "updall -c" will just waste space building an index that isn't required.

    "updall -r" only rebuilds indexes that already exist so won't actually use up any more space at all. In fact, as the views get rebuilt, they may even get smaller as they are optimised to take into account deleted documents etc.

    So yes, 'issue a "load updall -r -c" and you will see the space utilisation grow in a spectacular way', but who the hell would ever do that?

  1. 3  Mike Brown  | 8/15/2006 2:21:13 PM

    You don't have to issue any updall command for view indexes to be a problem though.

    For example, when users mail files are getting too big, and we're nagging them to delete some stuff, we tell them to go into their All Documents view (something that Outlook doesn't have any equivalent for, by the way) and sort by size to see the big mails. But if they haven't used their All Documents view for a while (and most users don't) then whammo! That's XX megabytes added to your database size instantly, and the users are the phone screaming "I deleted all these mails but my mailbox size has actually gone up!"

    It would be niced to have an option to not count view indexes as part of a database size; the same way you can use LotusScript to get a size that excludes white space if you want to.

    Cheers,

    - Mike

  1. 4  Ben Rose http://www.jaffacake.net | 8/15/2006 3:31:39 PM

    Interesting perspective Mike, another one I haven't seen.

    The "All Documents" view index discards after being unused for 30days, a little shorter than most of the mailfile design elements which are 45days.

    I've yet to meet a user that not only used their "All Documents" within 30days but didn't live in it.

    I see your point though - counting view indexes as part of a database size can be an issue, especially when implementing quotas, but those view indexes are proportional to the number of documents in the view so to get them down in size then the trick is still to delete documents.

    Including view indexes in a quota not only penalises users that hang on to large attachments, but also those who keep hold of many small emails too. In honesty a large number of documents is more of a resource hog than the total DB size anyway - a static mail attachment just sits there, but an inactive document delays every view rebuild.

    At this point I draw attention to the Domino Administrator client:

    Open a server and click on the files tab. Choose and select a DB, a mailfile would be a good example. On the right, expand "Database" and choose "Manage Views..."

    In the dialog that pops up, you can see the views that are currently built and the space they occupy. You can even purge view indexes if required.

    It can be quite useful to evaluate this for all users in a small setup, or just a sample on larger infrastructures. You can see which views users use, and those they don't. You can set lesser used views to expire faster or even delete them altogether.

    For those without an Admin client to hand, a picture speaks a thousand words...

    { Link }

  1. 5  Mike Brown  | 8/15/2006 3:47:01 PM

    Thanks, Ben.

    Yes, Manage Views in the Admin client is exactly where I head to when I get this problem reported. It does the trick, but that only lasts until next time the user bangs up against their quota.

    BTW, in my experience, most users don't use the All Documents view all that often. They tend to be folder junkies and know exactly which folder to look for any particular email. Me, I'm the opposite; I barely use folders at all. I have one big, fat, FU Inbox, and do an FT search on that for everything I need; or the Sent Mail view if it's somethign I...errm... sent! (I seem to recall Ed had a Blog posting on this very subject, way back when).

    Cheers,

    - Mike

  1. 6  Mark  | 8/15/2006 4:06:14 PM

    @5 curious what is your quota, and how many users do you have?

  1. 7  Mike Brown  | 8/16/2006 12:34:57 AM

    @6

    100 Meg. 1400 users.

  1. 8  NeilT  | 8/16/2006 2:50:35 AM

    @2 Well Ben,

    Updall -r updates all indexes used once or more. Updall -c updates all indexes never used and all FT indexes. So, if you want to force a full update of ALL indexes after a rebuid/Updgrade, you will ALWAYS use updall -r -c. Well if you don't want to be doing it real time during user log-in at peak periods and jamming your server up.

    As you have said, indexes which are unused are discarded after 45 days. So you create a new database/do a full compact, front load it with all indexes then let lack of use remove those which are not used.

    Which part of proactive maintenance and server load levelling did I miss?

    Remember that I'm on record as saying the disk usage of indexes are not an issue, so long as you manage it.

    What I left you to infer was that claiming this index usage doesn't exist by comparing unindexed databases leads customers to believe they don't need to expand their disk capacity. Disk is cheap, accept and plan for it is what I was saying. Doing the other leads to dissatisfaction and defectors for completely avoidable reasons.

  1. 9  Ben Rose http://www.jaffacake.net | 8/16/2006 6:09:09 AM

    @8 NeilT,

    On the technical side you're wrong, but I don't see any value in a public argument; by all means contact me off-line to discuss further if you wish.

    You're right about disk being cheap though and the costs of it being part of a Business decision seems a bit trivial to me too. Unfortunately it's one of the only rabbits in the Microsoft sales persons hat so it's daft points like this that the IBM sales team have to fight against.

  1. 10  Chris Bordeleau http://chris.bordeleau.net | 8/16/2006 6:50:41 AM

    So we actually have the worst case scenerio... We have a very small number of users, about 35, and 5 shared use mail databases for work groups... and our average mail file size is.... 2gb... with 1 user at 8gb and five users over 4gb... and one of the shared use mail databases is 5gb... Now we are on a beefy AIX 570... w/ 4gb of memory and plenty of CPU in our LPAR... We use a SAN and have transaction logs enabled... use a seperate disk for view rebuild... And our performace stinks... oh... and half our users use IMAP some for that large shared mail database...

    It used to stink for everyone... but I found if I limited the server to the amount of actual memory present then it only stinks for users of the large databases...

    So you ask why do we not do quotas and such... well the 8gb mail file is our CTO... he likes exchange and wants to go back (we migrated away about 3 years ago)... With our small number of users exchnage actually performs well... more then likely we will migrate back in the next couple of months...

    oh... and I tried archiving with the CTO... he did not like that he had to search in two places...

    I had a ticket open for a while with IBM and all I kept getting back was that our mail files were to large... I asked about tuning for the server and got pointed to some technotes that did not help....

    Now I support a campus with a similar hardware config that has 1000's of users without problems... difference... average mail file size is 25mb...

    It is hard to tell people that our enterprise email solution can not handle 2gb mail files when the keep on bring up google, hotmail, yahoo, etc...

  1. 11  NeilT  | 8/16/2006 6:57:31 AM

    @9

    Ahh technically wrong. I shall remember that :-)

    Bill says he wants to slap me when he's on my side. The last guy I publicly humiliated because he was wrong (and tyring to make me accept it), made my life a misery for 2 years. I shall instead refer you to "Updall options" in the Domino Administrators Guide and leave it at that.

    However if you are challenging the processes I use to design/support Domino servers, you have every right to do that and I won't have a public fight over it.

  1. 12  patrix  | 8/16/2006 7:47:55 AM

    @9, @11 ouch! Sparks flying... I dont know much about administrating Notes, so I had to look up this up. See below. Comments? I love to keep an arguument going :-)

    The Admin 6.5 help says:

    Updall - Rebuild options

    Rebuild: Full-text indexes only -X

    Rebuilds full-text indexes and does not rebuild views. Use to rebuild full-text indexes that are corrupted.

    Rebuild: All used views -R Rebuilds all used views.

    Using this option is resource-intensive, so use it as a last resort to solve corruption problems with a specific database.

    Rebuild: Full-text indexes and additionally: All unused views database -C

    Rebuilds unused views and a full-text index in a database. Requires you to specify a database.

  1. 13  Ben Rose http://www.jaffacake.net | 8/16/2006 8:01:39 AM

    @10 Chris,

    How do you define performance that "stinks"?

    I have a server here, the one I upgraded last weekend, with 311 mailboxes on it...85 of those are over 1GB...6 of them over 5GB.

    Performance is more than acceptable with just 3GB of RAM on the windows box. How are you measuring performance?

    @11 - Yes, I am challenging you - but this isn't the place. I've created a new thread...just for you...I hope Bill is watching :O) { Link }

  1. 14  Sean  | 8/16/2006 8:24:30 AM

    @10 Chris what version of Lotus Notes and Domino are you running? R7.01 is faster or R6.55 is good too for performance if you have a decent network and decent PC.

    I get the "I don't like to search in two places" all the time from users. They are simply lazy, and that is harder to fix than a broken IPOD. Your CTO needs

    a reality check. Export his 8 GB Notes database to a PST file and then have him try and access it using Outlook. His tune will change quickly when he sees how painfully slow Outlook really can be with large GB databases.

    Have you tuned your Domino configuration by enabling Network compression on the client and server ports, and upped the NSF pool from the default to say NSF_BUFFER_POOL_SIZE_MB=1500? I have found that these two tweaks on both the client and the server make a huge difference in Notes Client performance and should also help the Outlook users as well. Lastly be careful with full text indexes. Don't set them to update immediately, and limit the attachment option if possible.

  1. 15  stephen hood  | 8/16/2006 10:05:40 AM

    @1 We have 25 users and there isn't anything daunting about it. So Exchange isn't all you want actually.

    It's actually more important that a small company gets it right to start with. We have someone that does the our network and domino/notes admin part-time. Upgrading Domino to version 7 took him 30 minutes - most of that was file copying.

    Other admins that have come and gone have all related the same story - lost weekends, weeks on Exchange upgrades. Upgrades that failed and needed to be rolled back. Weeks and months of planning, crossing their fingers *hoping* it would go ok.

    Exchange is the daunting prospect not Domino.

  1. 16  Chris Bordeleau http://chris.bordeleau.net | 8/16/2006 10:35:25 AM

    @13 refreshes of the inbox can take several minutes (usually under a couple of hundred docs in there but lots of mail goes through it - several people are filing them away). The problem is most evident after using the database leaving it open then comming back to it later... or with first open.

    I actually found the ticket so here are some times :

    Server Date Time Task Duration

    ================================================

    lotusmail2 7/13 7:15AM First delete message after opening scadm mailbox, it was one of those accepted meeting notices, it took 1:50 to perform the operation

    lotusmail2 7/13 7:20AM Selected 2 messages to delete (using shift-down arror). The second message would not appear select for 15 seconds.

    lotusmail2 7./13 7:25AM I did Tools => Preferences => letterhead, the system hung and did not return usability for 2:25.

    lotusmail2 7/13 7:30AM I did a reply to all (the only recipients were ITEC SCADM and todd randall), it took 25 seconds for the window to come up.

    lotusmail2 7/13 8:47AM I deleted the ucereport email from the inbox, it took 2:10 to perform the operation

    lotusmail1 7/13 9:30AM Paul Hebert opened the scadm calendar, it took 3:30 to perform the operation

    lotusmail2 7/13 10AM I already had the scadm calendar opened from a couple hours prior, but when I clicked on the tab to look at the cal, it took 25 seconds to bring it to the foreground

    lotusmail2 7/13 12:57PM created a new scadm rule, took 50 seconds

    While this was going on I had some debugging on due to the problems.... so I submitted to IBM the NSD's and all other stuff they asked for... came back that the large mail files were to blame... I asked what tunning could be done... was told large mail files we to blame... kinda left it at that..

    @14 We are running 7.0.1 on that server... It is clustered witha 6.5.5 server... we were going to upgrade them both but ran into lots us locking errors so we only did one... increasing the lock memory pool to 20 mb stopped those errors...

    I have network port compression on but not the NSF pool tweak... I will have to give it a shot... I assume you mean te NSF pool on the server...

    I have most FTI's set to go hourly... attachments are off unless the users turned them on...

    Our 8gb CTO uses IMAP (which BTW kills the servers... lots of CPU lots of Memory) and outlook... doesn't seem to mind the performance... but complained a great deal about the performance of the notes client...

  1. 17  Dan Holzrichter  | 8/16/2006 10:54:20 AM

    an 8 GB mail file in itself isn't such a big deal. I have one server with 150 users with mail files in excess of 1 GB and some as large as 9 GB, plus another 500 or so with mail files averaging around 700 MB. I think the key thing is going to be what you have for disk subsystems and how you have it configured. BTW, I have one database on this server that is over 55 GB now and performance is very good on it as well, so database size in itself doesn't necessarily have to kill your server performance.

  1. 18  Alan Lepofsky http://www.alanlepofsky.net/ | 8/16/2006 11:28:32 AM

    Chris, why is your boss not using Domino Access for Microsoft Outlook?

  1. 19  Ben Rose http://www.jaffacake.net | 8/16/2006 11:53:03 AM

    @16 - Chris - This is pretty abnormal behaviour. Selecting emails in an inbox using shift+down arrow, for example, does not really utilise the server(unless you scroll of the page)...this is a client performance issue.

    As Dan highlighted @17, this sounds like a disk subsystem performance issue to some extent.

    Have you investigated the "don't overwrite free space" option in advanced DB properties. This alone has a great benefit on performance on all platforms and, imho, should be on by default.

  1. 20  Chris Bordeleau http://chris.bordeleau.net | 8/16/2006 12:47:27 PM

    @17 We have everything on an IBM shark... I do not know alot about the san but it suposed to be fast...

    The tech on the ticket I had open said the large mail files had to many lock and that was what was causing the problems... I intially had server crashes and hangs... so in the end just having it slow is a step up...

    @18 - DAMO's original problem was pst file size was limited to 2gb... then when the latest version came out we ran into the issue of the Body field going missing (this is a fun on... not only can you not see it... it gets removed from the doc - see technote 1229748) - just seemed like it was so close but not really there...

    @19 - I turned on the don't overwrite free space on all my large dbs... did not seem to make a big difference...

    I have a feeling it is disk related to... but I ran perfpmr for the tech and they never pointed at that... just large dbs...

  1. 21  David Bell  | 8/16/2006 4:08:34 PM

    @Chris - do the Notes clients have a local data directory or are they on a drive mapped to a file server ?

    Where are the transaction logs in relation to the data directory filesystem and other data in use on the SAN ?

    The SAN is not just configured in one big array is it ?

  1. 22  Chris Bordeleau http://chris.bordeleau.net | 8/16/2006 4:34:05 PM

    the clients have local data directories...

    we have all files in the san... and the disks are all laid out max spread... we learned that on the hardway when we overloaded a shelf or to and had to move things around a bit...

  1. 23  David Raciot  | 8/17/2006 8:57:46 AM

    Chris: this is not a normal experience you are seeing. We support a 200+ company with numerous large >1GB mail files and performance is fine. I think you have a throughput (network card?) issue. Also, are you using tx logging?

  1. 24  Steve Cogan http://www.linkedin.com/in/stevecogan | 8/17/2006 9:21:51 AM

    @22 Chris,

    Have you seen the best practice recommendations for working with large mail files?

    { Link }

    Regards,

    Steve.

  1. 25  Chris Bordeleau http://chris.bordeleau.net | 8/17/2006 2:27:50 PM

    on our 2gb and less dbs performance is fine... it is only in the really large 4gb+ dbs that we have issues....

    I have looked at the best practices doc but other then contacting IBM for help it does not give any real help... I understand that with large dbs everything is amplified but it would be nice to see a doc saying make these memory settings and you will see better performance...

  1. 26  NeilT  | 8/17/2006 4:01:32 PM

    @13

    I take that Challenge Ben, better have a read on your site. I'm busy (which is why the slow response), tired and have a major headache because of 2 hours of innebandy without the physical condition to play :-( so I'm a little blunt.

    This is my last word on the subject.

  1. 27  NeilT  | 8/17/2006 4:40:44 PM

    @25 Chris,

    It sounds to me like your server is paging. Make sure it isn't, that's "death" to a Domino server. Wind your page pool down until is stops. Also remember that the web ports and tasks eat memory and need to be restricted too. There's stuff on notes.net for that.

    I'm going to make some assumptions here and try to give you some help on the design.

    Firstly the SAN. I assume it's FC switched. I have just done a lot of testing with a 4Gb/s IBM TotalStorage SAN and it's way fast. However I was maxing out the 2GB/s HBA on the server at around 250 Mbytes/s. The SAN was coping at around 400 without an issue so if you aren't already 4Gb/s I suggest you get there.

    You are using AIX so I assume you are pathed something like /home/domino/data/mail.... etc

    Find yourself two trays that are not maxed and split out two or four drives among them. Make sure the pairs are on separate drive channels.

    Raid these drives as RAID1 and create a volume on them.

    Remount the exising volume, which domino is mounted on now, somewhere else for now.

    Mount the new RAID1 volume in /home/domino/data

    Migrate all the domino data directory from the existing RAID5 into this volume except for the mail dir. Move all the files and directories from the mail dir to the root of the RAID5 volume.

    Remount the existing RAID5 volume as /home/domino/data/mail

    If you are also using applications, you will need to split the one RAID5 into two and use one for apps and one for mail.

    If you have internal disks, move the rebuild dir and the translog to the internal disks. Just spread the load over as many interfaces as you can. If you don’t have internal then I would suggest using another pair of RAID1 disks for this mounted as a new volume.

    I don't know if you are using HBA failover, but personally I would be load balancing the volumes on different HBA's to max the bandwidth.

    Pretty much that will give you the best layout that you can get bar introducing another SAN into the picture to balance the load out.

    Now if you have CPU/Memory to spare and you have sorted the disk out, you may want to up your Updaters setting in the notes.ini, moving it up to 2 just to see if it helps and increasing again incrementally. Although it doesn't sound like you have much left in the way of performance.

    Finally, if you have CPU left and some memory you can play with Server_Max_Concurrent_Trans. This can have dramatic performance gains on a server which is underutilised, however it can have dramatic stability issues too. It defaults to 20. I have seen it used at 250 on an iSeries to dramatically up the number of users. It is set to -1 in notesbench tests. Personally I have seen a windows NT server fold up and refuse to do anything when this is set too high. Admittedly it was 1,000 :-D

    Finally networking. You are clustered. Do you have independent network cards or are you using one Gigabit card? If you are using one card, don’t cluster over the same port, create a new Domino port on the same card to take the clustering. Personally I would alias and use a 1.0.0.x address for the cluster. Look at your stats and see how long the cluster entries are on queue. If they are stacking up add cluster replicators until it stabilises.

    Most of this is pretty dramatic stuff and will require some downtime to accomplish. However you are clustered. So that won’t be too bad. But I sense that you are in a corner and need a way out. I don’t build conventional Domino servers, but then I don’t usually find myself in a corner unless there’s a bug….

    If you want to discuss this with me offline, you can get my mail address from Ed.

  1. 28  bernard devlin  | 8/17/2006 7:40:05 PM

    @10

    "It is hard to tell people that our enterprise email solution can not handle 2gb mail files when the keep on bring up google, hotmail, yahoo, etc"

    Chris, there is something seriously wrong with your setup. I don't see the kinds of performance problems you see, even in really unbelievably poor conditions. Let me give you an example.

    I have one mail file that is almost 2gb in size, it is housed on an R5 server running on a Redhat 7.2 server. The server is only a lowly Celeron with 256mb of Ram, and everything is running on a single IDE disk.

    But wait.. it gets worse! I access this server through an R5 client, running on Windows 98, running inside Virtual PC running on an old G3 iMac. The mac is connected to the internet over a 2mb ADSL connection (obviously the Notes client is having to pass through a virtualized NIC which would slow it down further).

    However, moving between folders (some containing approx 100,000 docs) only takes about 7 seconds, and there is no observable delay in selecting documents in views, nor any delay in deleting documents. Even when there are other tasks running simultaneously on the server there is very little impact on the speed of moving between folders.

    I find it really hard to imagine that anyone has an environment better designed to show the worst performance of Notes and Domino, yet moving around the Notes db in the situation I am describing is as fast as moving around my gmail mail box (and I tested gmail from a browser running outside of Virtual PC).

    Given that your server hardware is vastly superior to the situation I'm describing, there must be something else in your environment that is causing your problems. Maybe it's the clustering, maybe it's the SAN.

    I know that the scenario I've described has a lot less complexities than your situation. But it bares out the simple principle that Notes and Domino can quite easily manage 2gb mail files, even in unbelievably adverse technical situations.

    I think you need to have a test server, and slowly build up the complexities until you find out where the problem lies. It will be easier to test hypotheses and arrive at certainties in a simpler environment, and you can do it without disturbing your users. Sometimes the only way forward is to get back to basics, and to establish just what it is that one knows to be true.

  1. 29  Chris Bordeleau http://chris.bordeleau.net | 8/18/2006 6:59:42 AM

    Like I said... the 2gb ones are really ok... it is the 4gb+ ones where I have a problem... I only said that about the 2gb mail files as that is what google gives... and that is what they compare to... Most users here access one of the 4gb shared email databases so they see the poor performance there...

    We use VIO with our 570 so all of our IO is virtulized... I contacted IBM about a best practices doc on this and got nothing... we have 8 paths to the san (4 going to each fabric) and 8 gb network adapters (4 going to each redundant network) in this vio box... when we look at the VIO we do not use over 10-20% of either....

    We do swap from time to time which is why I limited the amount of memory using ConstrainedSHMSizeMB... this limited the amount of swapping and made performance for everything under 4gb great...

    I will read though your doc but we are mandated to put everything in the san (an IBM Shark)... we also have no say in how the san is setup... so we are max spread accross shelves but I have no control over where in the san I am or what the formating of the disk is...

    We do have CPU to spare but no memory... we have maxed out the system at 80gb (shared amoung lpars - It will handle more but we would have to replace memory... it is very expensive). I get 4gb that I share between two DPARS.

    thanks for all your help...

  1. 30  Sean  | 8/18/2006 8:03:00 AM

    Chris, you have done allot of tuning and at this point I do not think it is a server issue. If you still think it is a server issue, start a few extra update tasks. LOAD update on the console, or set the ini with updaters=3. This will make sure there are no views or folders waiting for refresh.

    You are using DAMO and synching correct? It sounds like the performace issue is the Inbox only on refresh of the PST file. Where is that PST located, on the SAN or on the PC? You must also be using Outlook 2003 with the 20 GB file size limit, since the CTO's file is 8 GB in size. You really are pushing Outlook to its limits. Make sure Outlook is current. I don't think much else can be done other than beefing up the PC and adding lots of memory, since it seems to be a Outlook client and PST issue.

    Also, there arer 42 fixes for DAMO in 7.02.

    Check them here. { Link }

  1. 31  NeilT  | 8/18/2006 9:27:16 AM

    @29 "I will read though your doc but we are mandated to put everything in the san "

    Argh, the old "leverage spare capacity" turns into "eats machine alive".

    I'd get out to Intel with Windows or Linux. You are never going to fix this without spending 20 * what it would really cost. Also you'll get incredible performance for minimal cost if you move.

    You are underspecced and without budget to fix it. Exactly the point I was trying to highlight in #1, that we should all avoid doing this by being careful what we say and reccommend.

    You could of course ask to have your LPARs priority elevated for disk, cpu and memory, but I suspect it is a lost cause.

    Sorry anything else I could suggest is hostage to the whim of your Storage and AIX owner.

  1. 32  bernard devlin  | 8/19/2006 5:40:10 PM

    @29

    Well, what can I say. For your sake, I doubled the size of that mail file so that it was over 4gb. And I still get the same response times in moving between folders, approx 7 seconds. And that is in just what is probably the most under-powered technical environment anyone would every come across. Quite frankly, when my db got to over 400,000 documents and passed 4gb, I expected my access times in changing folders to at least double. But it didn't - it remained the same. That makes me even more pleased about the way that Notes works.

    So, as I pointed out earlier - whatever problems you are having are clearly not related to the fundamental functioning of the Notes server or client. R5, even with poor quality server hardware, poor quality client hardware and low-speed network connections, is perfectly capable of moving between folders containing 100,000+ documents in a database of over 4gb in size in a matter of a few seconds. There is no delay in selecting or deleting documents, and even paging down in the 'All documents' view takes less than a second. In your small but well-powered environment you should be seeing much faster response times than I am observing here.

    You were just as capable as me in establishing these facts. But you are the only person who is really capable of isolating the other variables in your environment in order to identify the cause(s) of the problems you're experiencing. You can complain about poor performance, or you can find out why you are getting poor performance. Your problem could be clustering, SAN, your network, AIX, or several other problems. It is clearly _NOT_ a fundamental failure in the architecture of Notes.

    I worked for a company which replaced OS/2 with NT throughout the company, because the OS/2 machines kept crashing. About 90% of all hardware was beefed up to run NT. And then the newly-installed NT machines kept crashing. Because it was the network infrastructure that was faulty. Once the network moved from token-ring to ethernet (and the buildings completely rewired), all the machines were very stable (including the few remaining OS/2 machines). Of course, the staff there are probably still as bad at trouble-shooting - and are currently throwing out Notes for Exchange for similar silly reasons.

    I usually choose the humble position of assuming that the architects and engineers who designed and implmented a great, long-lived technology like Notes, are far more likely to have known what they were doing than any support staff working with that technology.

  1. 33  David Bell  | 8/20/2006 2:50:33 PM

    @29 - google may have a 2gb quota, but does anyone have any experience at what their service is like with a mailbox that big ? Saying you can have 2gb is one thing, what would happen if all their users actually used their full quota ?

    Your statement "we also have no say in how the san is setup... so we are max spread accross shelves but I have no control over where in the san I am or what the formating of the disk is.." sounds to me like the major problem.

    You cannot connect a Domino server to a SAN without having say in how your LUN's/physical disks are laid out. Storage specialists usually do not understand the intricacies and usage patterns of the applications using the storage.

    This is why I asked in an earlier post "is this all just one big array ?". I am not sure what "max spread" means, but it sounds like far too much disk is being shared at the physical level, rather than LUN's and their RAID levels being appropriate for the data they are storing and being dedicated to specific applications.

  1. 34  Volker Weber http://vowe.net | 8/20/2006 3:50:04 PM

    David, GMail currently has a quota of 2756 MB. And yes, you can fill it up and it just works. Just as well as with 20 MB.

  1. 35  Keith Brooks http://www.keithbrooks.com | 8/20/2006 4:01:24 PM

    As pointed out, your original related items of time delays were mostly related to the client machine, not the server.

    Having said that, as others have, the size of the db is not an issue.

    I know IBM said it is, but that is supports job to knock the "major" items first. They are not troubleshooters. My apologies to Ed and past co-workers.

    My current office suffers from HORRIBLE network performance.

    Which has nothing to do with Lotus or IBM.

    As was posted, ensuring you have up to date drivers for your network cards and making sure they are open to full capacity helps.

    In the past we used 2 NICs to ease the delays.

    If you put a sniffer on your LAN you can get some better ideas of true network speeds and see if your network manager is constricting Lotus packets.

    Also review all the notes.ini settings and memory settings for your NOS for mistyped or incorrect info.

    Just my 2 cents from over 12 years of Notes and Domino network management from Novell and OS/2 to AIX and mainframes.