Scalability

December 19 2005

I've been thinking about scalability a lot today.  The thought process actually kicked off with the Typepad outage from last week...even though in the end scalability wasn't exactly the problem there.  Yet it seems that "growing pains" is a common compliant about many Internet social software tools -- everything from Technorati to Orkut have been criticized for straining under increased load.

I've never really been a coder, but I've watched lots of different systems have to adjust -- or rearchitect -- due to scale challenges.  Back in 1990, I worked for FTD when they wanted to get into an early version of e-commerce.  Their approach was to develop an in-house system which used an ASCII text colon-delimited flat file for product records...which was fine for fifty items but not for 5000.  Needless to say, that system didn't last very long (though there were hundreds of additional reasons for that, which would make an interesting case study one of these days).

In my world, cc:Mail was definitely a system which eventually burst at the seams under scale issues.  As a cc:Mail administrator back in 1992, I only had solid performance when my post offices were each about 200 MB.  Now that's 1/3 the size of my own mailbox.  cc:Mail had great hardcoded limits like 200 users on a mailing list, or 20 attachments per message.  Obviously, Hubert and company designed an incredible system for its day, but its file-share architecture eventually hit the wall for modern-day messaging requirements.

Given all of that context, it's somewhat amazing that Notes has been able to evolve from 75 users per server over NETBIOS in Notes R3 to 150,000 user scalability with Domino 7, all without a fundamental rearchitecture.  It may not be sexy work, but the plumbing done in Domino 7 to get 40-400% scalability improvements on existing hardware is especially impressive when viewed this historical context.  

Ed Brill Cambridge, MA USA Category:  Notes/Domino 7  Add/View Comments [25]

Post a Comment

  1. 1  Volker Weber http://vowe.net | 12/19/2005 5:12:07 PM

    Agreed.

    With one execption. If you can improve by a factor of 4, you were not really good before. I assume this is Linux, the youngest and least optimized for plattform.

    Getting 40% out of a known plattform however means very good optimization work.

  1. 2  Randall Shimizu http://is-perspectives.blogspot.com/ | 12/19/2005 6:15:02 PM

    Scalability and reliability is without a doubt Lotus and IBM's best weapon against Microsoft. With Power and Linux MS does not stand a chance. IBM is coming out with both a Cell and Power5 blade soon. IBM is also coming out with a infiband interface for the Bladecenter.

  1. 3  Mike Robinson http://www.invcs.com | 12/19/2005 7:45:11 PM

    @1 I beg to differ. A factor of 4 is context- R3 days 100 users on a server WAS a big deal, and didn't mean you product stunk. This was early 90s, keep that in mind. There was no Exchange, it was msMail, and a bunch of other somewhat file-sharing type products.

    Also The statement is incorrect, Linux scales very well when tweaked properly. Check the NotesBench sites for the R5 reports we did for SuSE and Red Hat. There's all sorts of goodies in there for optimizing your Linux based Domino system that's applicable to R6 (oops D6) and R7.

    One thing I remember when running the Notesbench test was that the darn (Linux) server continued to take the Notesbench pounding. I purposely let it run 7500 users running a decent mail load 24/7 with no let it up. It went over a week before I just finally ran out of disk.

    That never really happened with the Windows box :)

  1. 4  Bill Geimer  | 12/19/2005 11:02:24 PM

    Ed, you know I fondly remember being able to administer multiple cc:Mail post offices from my Mac back then too. And to this day, I thank Hubert Lipinski and company for what they did back then, including your help - but not including the on/off/on/off support for the Mac by cc:Mail, Lotus and IBM. Glad to hear its on the comeback trail.

    Unlike some, I have to commend IBM / Lotus for supporting Linux on the 2.4 kernel, despite its limitations, being the new kid on the block. But I truly appreciate the improvement in 7.0 with 2.6 kernel and the new thread support. I hope that IBM / Lotus will continue in that vein and step up to 64 bit support on both Linux and Windows in the near future.

  1. 5  Volker Weber http://vowe.net | 12/20/2005 6:04:13 AM

    Mike, I was talking about improvements from 6.5 to 7.0.

  1. 6  @1&5 http://www.bart.declerq.easynet.be | 12/20/2005 8:06:08 AM

    Volker, if you read the article, you'll see that the x4 was due to removal of a hard limit (Domino on Linux only having 2GB addressable memory and 256k overhead for each connection adding up to a max of approx. 3000 users on <R7 given the test-parameters)

    So, you are pretty mucht right : this massive improvement was due to it being the "new kid" on the block, the double digit improvements seen on the other platforms is apparently due to multiple optimisations, and is quite impressive.

    (And if you see the numbers, it seems clear that Domino is now about equally 'scalable' on all platforms)

    I do think that talking about "40%-400%" improvements is a bit disingenuous, the improvements of around 40% listed in the article would be visible as reduced resource usage on existing server hardware, whereas the 400% improvement on linux will only show itself if the server is already bumping against the software limitations... (although probably some or all of that 40% reduction is also present in the Linux version of Domino)

  1. 7  Bart Declercq http://www.bart.declerq.easynet.be | 12/20/2005 8:12:20 AM

    Oops, sorry, the @1&5 above should have gone in the body, they're obviously not my name (although Resistance *is* futile!)

  1. 8  Paul Robichaux http://www.e2ksecurity.com | 12/20/2005 11:03:25 AM

    @6: so, this hard limit removal would be roughly the same as the much-dismissed scalability improvements that Exchange 12 gets from running on x64 hardware?

  1. 9  Bart Declercq http://www.bart.declerq.easynet.be | 12/20/2005 12:18:40 PM

    Ah, Paul, except that it:

    (a) doesn't require x64 Hardware

    (b) Is only relevant on one of the many Domino platforms, the others didn't have this limitation...

    So, maybe very very roughly ;)

  1. 10  Volker Weber http://vowe.net | 12/20/2005 12:27:23 PM

    Paul, Exchange 12 on x64 with the old database engine reminds me of an old engineer adage:

    "Given enough thrust, even a piano will fly"

  1. 11  Paul Robichaux http://www.e2ksecurity.com | 12/21/2005 9:06:08 AM

    @10: LOL. Of course, Exchange 2003 already scales well beyond 3K users/server, so the fact that Domino R7 is catching up is welcome news for people who aren't planning on buying all new POWER or iSeries hardware for R7 deployment. I daresay that market segment is the majority of Notes customers.

  1. 12  Ed Brill www.edbrill.com | 12/21/2005 9:11:28 AM

    Come on Paul, you know that Domino scales past 3K users per server on Wintel hardware, with 10K plus users per server not uncommon. Linux had to play catch-up because of its kernel limitations, but now is as good as any other hardware/OS combination (and better in some cases!)

  1. 13  Nathan T. Freeman  | 12/22/2005 6:45:09 AM

    LMAO. I want customer references for both Paul's & Ed's claims. I bet I'll get them from Ed.

  1. 14  Ed Brill www.edbrill.com | 12/22/2005 7:38:29 AM

    @13 Nathan, see page 16

    { Link }

    "The first LPAR was set up with two instances of Domino 5.0.10 and served a total of 7,850 users. Eventually, both instances were upgraded to Domino 6 on this LPAR. The second LPAR was set up with three instances of Domino 6.0, serving a total of 5,800 users."

    Over two years ago.

  1. 15  Ed Brill www.edbrill.com | 12/22/2005 8:04:16 AM

    oops, that was on pSeries. I'll have to go find some Wintel ones -- obviously, the IBM internal deployment isn't on Wintel :-o

  1. 16  Mike  | 12/27/2005 12:42:48 PM

    can't seem to post comments on your 'biggest...' entry -- perhaps the dot dot dots are messing up the url?

    Without specific tighter definitions, in broad numbers I have a couple of 'big' servers. Registered, vs. concurrent number of active users however can play a big part in things.

    Anyway, I've got two p570's with four partitioned servers each and 11,265 registered users. The p570's are clustered, so we can shutdown one box and all users will run from one server quite easily from their clustered mailboxes. We don't normally run in that configuration, but it can and does handle it when required (during scheduled/planned maintenance, etc.) very nicely.

  1. 17  Peter de Haas http://www.peterdehaas.com | 12/30/2005 3:49:16 AM

    For Nathan ....:-)(and Paul)({ Link }

    Some organsiation running MS Exchange 2003 on a consolidated / centralised environment server > 3000 mailboxes per server. On Wintel ofcourse...

    { Link }

    { Link }

    Also very interesting is to learn how Microsoft internally has consolidated the Exchange 2003 environment :

    { Link }

  1. 18  Ed Brill www.edbrill.com | 12/30/2005 8:11:42 AM

    I'm sorry, Peter, I don't see anywhere in those case studies that says that they have >3000 users per server.

    Siemes -- says they have 10000 users, that they cut 59 servers down by "80%" which would yield 12 servers in the Exchange 2003 environment. On average, that would mean 833 users per server.

    Nissan -- says, "Now, because Exchange Server 2003 supports up to 3,000 users per computer, Nissan can reduce the number of e-mail servers from 36 to 18 in North America. " That doesn't say that Nissan IS RUNNING 3000 users per server, just that they CAN. An example shown further in the case study has 2278 mailboxes on a server cluster (with that famous single point of failure around shared storage). This is classic Microsoft -- an implication but not a fact.

    Microsoft -- Table 5 indicates 4000 mailboxes per server (vs 3000 in Exchange 2000). Only thing is, there are 50,000 users on, according to table 4, 36 active mailbox servers (not cluster passive nodes). That makes for an average of 1333 mailboxes per server. I'm not clear on what the data in table 5 is really meant to represent. Typical? Average?

  1. 19  Peter de Haas http://www.peterdehaas.com | 12/30/2005 8:27:48 AM

    Ed,

    I'm sorry the case studies do not provide the exact level of detail required to meet the bar here.

    We could have a lenghty discussion on server roles in the Exchange configuration, because not all of the servers referred to in the case studies are actual mailbox servers.

    More detailled information on these casestudies is not (publicly) available as far as I know.

  1. 20  Peter de Haas http://www.peterdehaas.com | 12/30/2005 9:12:46 AM

    In the rebound :-)

    HP : 4000 users per server : { Link }

  1. 21  Ed Brill www.edbrill.com | 12/30/2005 11:31:14 AM

    break open the champagne :) Would be interesting to run a "who has the biggest" contest on msexchange.org or so...

  1. 22  Peter de Haas http://www.peterdehaas | 12/30/2005 12:33:31 PM

    Well no need to get excited Ed.

    IBM wins. I do not know of any MS Exchange implementation running on a piece of hardware over 4 Million USD.

  1. 23  Duffbert http://www.twduff.com | 12/30/2005 12:54:36 PM

    Ouch... *that's* gonna leave a mark... :)

  1. 24  Ed Brill www.edbrill.com | 12/30/2005 1:22:22 PM

    Well, I don't know of any Domino implementation running as the only job on a 4 million USD piece of hardware. The whole point is leveraging existing investments -- not having to go buy all new hardware just to upgrade. :)

  1. 25  Peter de Haas http://www.peterdehaas.com | 12/30/2005 1:59:05 PM

    Ed,

    As a serious note I honoustly don't think a " mine is bigger than yours" discussion will provide a good one to one comparison / is a relevant discussion.

    At the end of the day it comes down to TCO which includes things like hardware / software / implementation / management / etc.

    Scalability / the ability to consolidate servers plays into this discussion, but so is networking costs to name one important (cost)factor ...