Damien Katz: Formula engine rewrite

January 4 2005

Damien, buddy, I hope you feel better after this 5,000 word entry....

While I was on the template team I developed the original Notes Welcome page, I did work on e-mail and a lot of work on some obscure administrative applications, among other things. At the time I quit I was working on new features for the R5 point releases. But I hated it. I hated pushing around pixels all day, negotiating with the Quickplace and Sametime teams who wanted tie-ins and functionality on the Welcome page. I frequently worked 80 hours a week, was stressed and was seriously burning out. Why did I work so hard at something I hated? I don't know, I think because I wanted desperately to get away from it. So I thought if I just worked harder, someone would be impressed enough to give me a shot working on something I wanted to work on. Something more geeky and computer sciencey, instead of the touchy-feely UI crap where marketing types argue with me all day about how they think it should work and look.
Disclosure: When I moved to Lotus product management in May, 1998, my initial responsibility was for e-mail migration tools.  When I completed the licensing agreement with Binary Tree for their DUS tools, my second product management assignment was to be responsible for a Notes client feature we then called "Headlines", which became the Notes R5 Welcome Page.  I worked with these young, aggressive programmer types (well, one in particular, wink wink) who thought they knew how the new feature should work and look.   [Not to put too fine a point on it, but the fact that the feature was spec'd as "Headlines" and came out as "Welcome Page" was not a marketing decision, and with hindsight probably not the right decision.  The Workplace welcome page in Notes 6.5 and 7 comes pretty close to the original "headlines" vision. But I guess that's another blog entry ;)]
Anyway, back to Damien's story:
No one on the language services team, or even at Iris, understood the runtime engine internals completely, also known as "Compute" (the compile/decompiler was called "Compile", great names eh?). There was no documentation, the code was barely commented, many variables had one and two character names, wild pointer arithmetic was everywhere and not even the stored byte code format the compiler produced was documented. Bugs were piling up and there was no one to fix them. On top of that the engine was becoming a bottleneck for server performance, particularly for building view indexes. The engine needed to be rewritten from the ground up, it needed to be made maintainable, it needed to be faster, and it needed to be completely backwards compatible.
Damien mentions this several times in his essay -- the importance of backwards compatibility being designed in from the outset.  Damien's overall effort shows that backwards compatibility while moving the ball forward is absolutely achievable -- why some vendors always throw things out and start over is beyond me.
Now you might think that I produced a bunch of design documents and specifications and presented them to the various senior engineers and architects, but I didn't. I remember being surprised by this myself. Even Wai Ki didn't have much to say about my design or how it should be implemented. The philosophy was that if I did those things, everyone would meddle with the design and nothing would get done. It's truly easier to ask forgiveness than to ask permission, not to mention things get done a lot faster if you just do them. It sounds weird, but that was big part of the culture at Iris. Just get it done and do it right, do whatever it takes. So that's the way I worked, I was just going to go headfirst into it, damn the torpedoes.
Despite Damien's pronouncement later in his essay, I would assert some of this culture still survives in the Notes/Domino team.  Just the other day, I was offered the opportunity to code up a multi-level undo feature....oh how I wish I still had the chops to do it.  Someone will for a future release, I just know it...
Damien mentions his then-manager Wai Ki Yip several times in the essay.  Wai Ki is presenting a session on NSF-DB2 at Lotusphere 2005.  I had a chance to review Wai Ki's LS2005 presentation earlier today -- I'm very impressed.  That's one you'll want to see.
That's when I started adding all the new stuff that I wanted to add to the engine. I remember I posted a question to a forum dedicated for professionals who worked with Notes and Domino (The Business Partner Technical Forum), and I asked what sorts of features they would want to see in the new engine. That question got over a thousand replies. Some of the requests where reasonable, some where stupid, some I just didn't understand. But I read each and every one and tried my best to consider them.
I think this kind of vendor/partner interaction was completely unique at the time.  Now, blogs fill some of that need, but I still would assert that the level of access that Lotus customers and partners had and have direct into the engineering team is unparalleled in the industry.
My time at Iris during R6 was the best I've ever had in my professional career, almost every day was a new challenge and game to play. But ultimately IBM killed Iris. In late 2001 IBM folded Iris Associates and converted all the employees to IBM. All the energy the Iris building had was seeping out. Really it was more pushed out by the BS that IBM management kept trying to sell us, like that nothing important would change and you'd still be able to work unfettered. Yeah right, just ask anyone who still works there how much time they spend battling bureaucracy vs. writing code. But such is big corporate management, they don't even realize how much they're damaging things, they are essentially politicians not engineers. It's sad, but I'm grateful for the time I had at Iris anyway, if it was still there I'd go back in a second.
Lest anyone think I was going to shy away from this...
Damien is right.  The culture is changed.  Like a lot of companies, the mid-to-late '90s were a completely different kettle of fish from a work environment perspective than today.  I know some ex-colleagues at Microsoft often tell me "it feels like Lotus used to feel".  Well, that's the way it was, and just because Lotus has changed doesn't mean that some or all of that change was evil.  

One of the things I liked about adding some "bureaucracy" to the Notes/Domino environment was that it put an end to rogue projects.  When I moved to Lotus product marketing in 1999, we launched an effort codenamed "Bluejay", which was a collection of interesting rogue projects -- including Domino Network File Store and Domino Collaboration Objects.  Neither of these features were really ready for primetime; DNFS was implemented on an outdated protocol (Netbios), ran on one server platform, and had a hardcoded connection limit; DCO had no long-term commitment behind it, just a couple of enthusiastic product managers.  Today, new features or projects have to have a long-term plan before they can ever leave the door -- meaning a customer won't get stuck holding the bag if they start using it.  

One of the things I didn't and don't like about adding some "bureaucracy" is that sometimes, decisions are forced upon one product for the greater IBM good.  IBM says we will ship all software products in a consistent number of international languages.  That means Notes, too, even if the market for Notes in outer Mongolia (a hypothetical!) is minimal.  But as an IBM shareholder, I completely get the reason for this decision -- the customer in outer Mongolia shouldn't get stuck if they buy into the IBM strategy, with a lot of "oh, but we're not translating that product" kinds of answers.

Anyway, thanks to Damien for a fascinating look at how the Notes formula language came to be rewritten for Notes/Domino 6.  Still today, a lot of people tip their hat in his direction for an incredible effort.  I think it demonstrates so many things that are just core about Notes and Domino, and why the product continues to be #1 in the market after fifteen years.
Link: Damien Katz: Formula engine rewrite >
(Update: Check out the comments on Damien's site -- the original version's author has commented ;))

Ed Brill | Highland Park, IL USA Category:  Lotus  Add/View Comments [28]

Post a Comment

  1. 1  mugadafino  | 1/4/2005 9:36:10 PM

    great story.. a Lotus unsung hero...too bad IRIS got IBMized...

  1. 2  Ben Langhinrichs http://www.GeniiSoft.com/showcase.nsf/GeniiBlog | 1/5/2005 12:51:52 AM

    What has been really fun recently is watching Damien watch us learn just how powerful the new formula language is. Rocky posted about the use of @Transform, and I posted some uses for that and the new custom sorting in @Sort, and here is what Damien had to say in the comments on my blog:

    This is so cool! Finally, years after I created all that stuff I'm seeing examples of what it can accomplish. I've never actually had occasion to use anything of the new stuff, so this is the closest I've come to seeing it in action.

    I know what it is like to work hard on code and not really see it in action, so I am glad that Damien gets a chance to watch us enjoy his creation. You did a great job, Damien!

  1. 3  Ben Rose http://blog.jaffacake.net | 1/5/2005 6:15:01 AM

    A good story and a great opportunity to give a big thanks to Damien and all the other hard working developers on the Notes/Domino team.

    I had some exposure to the Iris guys back when I was working in Lotus Development UK, we used to kick the developers' asses in Halflife Deathmatch running across the WAN. Great fun and we actually got some work done too. I remember one of the guys, I think it was Bruce Kahn, agreeing to fix an R5 bug for me if I got more frags than him!

    Cheers Damien...I too would go back to the Lotus I remember and love...even it they yellow has a blue tint these days.

  1. 4  Bob Brodsky  | 1/5/2005 8:39:05 AM

    Ed,

    Domino Network File Store is still a great idea and we still need such functionality in the product for specific requirements. There is still no other way to accomplish Storage Independant Links! Kudos to the couple of enthusiastic product managers and all the programmers that created it and got it "in the Box."

    The limitations that you mention are NOT a requirement, however.

    Bob

  1. 5  John Voigts  | 1/5/2005 8:45:47 AM

    Fascinating post. I went to Damien's site and read through his whole story. Thanks for sharing this, Ed and Damien. It takes me back to when I first got my hands on Notes (late 1994), and began to see the potential and not having a clue how to code much of anything at that time. I had been a salesman for 10 years who had simply begun tinkering with a Filemaker Pro database to keep track of my own customers. My crude, fledgling CRM database came to the attention of the boss and now I've spent 10 years developing what is now a fairly robust CRM system in Notes. Like Damien, sheer determination and the willingness to take risks and learn is the only thing that got me through the first few years. Of course I only developed in production in those days. At least a hundred times, I would just by the skin of my teeth at 5 or 6 PM get back out of the deep woods I had gotten into that morning with some new functionality that was going to really be cool. The early days were more fun. If we had gone some other route than Notes, that requires a true programmer from the beginning, I would probably still be a salesman today. Becoming a Notes developer has been a fun ride.

  1. 6  Darren http://www.dadams.co.uk | 1/5/2005 8:55:56 AM

    Interesting comment about IBM killing off Iris and it's culture. I can't say any more without shooting my mouth off so I'll leave it there... except one thought... remember the recent rumours about IBM acquiring Apple? Day 1: the Apple marketing team are pulled in to a room whereupon an IBM marketing executive tells them "okay you lot, this is how marketing is done..."

    Iris used to have FREE vending machines - a guy I was with pillaged several bags of Swedish Fish.

  1. 7  Ben Rose http://blog.jaffacake.net | 1/5/2005 9:14:22 AM

    @4 Bob - I liked it (DNFS) in testing. Does it still work under R6?

  1. 8  Alan Lepofsky  | 1/5/2005 9:32:06 AM

    @6, I as well don't want to get into a long "Iris vs. Lotus vs. IBM" debate. Simply put, each has(d) their good points, each has(d) their bad points. I'll just share the best comment I've ever heard about this topic, it comes from "Iris" developer John Paganetti. At Lotusphere a few years ago in the "heat of the IBM takeover" someone in the crowd of the Meet the Developers session tried to be an instigator of how "bad" IBM is... and John strongly and passionately responded with something along the lines of "We would not be here today if it were not for IBM". That's it, let's leave it at that. We can argue and complain about lost candy bars, and how big companies have lots of politics (this is not unique to IBM), but the fact is everyone in the "Domino community" has IBM to thank for their jobs. Would I like things to be even better, see IBM even more behind Domino, and see market share go 90/10 for Domino instead of 50/50... sure I would. But at the same time, I am thrilled Notes did not fade into oblivion around 1996.

  1. 9  Ben Rose http://blog.jaffacake.net | 1/5/2005 9:43:23 AM

    Alan - I agree with the sentiments of your post, but 'everyone in the "Domino community" has IBM to thank for their jobs' has a touch of irony for me as it was that very same IBM that issued my redundancy notification when they closed Lotus departments all over Europe and moved support services to Dublin. I was one of the lucky ones who found a good job soon afterwards, but I haven't seen many of the others in the community since.

  1. 10  Damien Katz http://www.damienkatz.net | 1/5/2005 10:05:14 AM

    Thanks Ed.

    I remember working with you on the Welcome Page or Porthole as I would sometimes call it (Portal ... Porthole... it's all good), and I remember you as being easy to work with. But it seemed like you got promoted to something else pretty quick, and then your replacement did also, and then there was this never ending shuffle of PMs bouncing around all over the place. Maybe I was too hard to work with. I am a pretty demanding guy when it comes to work.

    I don't remember quite how the Welcome Page got its name (it might have been me, but I think I'd remember if it was me, I'm really not sure), but I do remember that marketing approved the final name without fuss, so don't blame me!

  1. 11  Ed Brill www.edbrill.com | 1/5/2005 10:21:35 AM

    I didn't say you were too hard to work with. I just was agreeing with your self-characterization. ;)

    As for the Welcome Page name...that wasn't my point. It was the user experience that changed from beta to FCS. I'll blog about it at some point -- it's not an individual critique, though.

  1. 12  Darren http://www.dadams.co.uk | 1/5/2005 10:32:50 AM

    @8 - Alan, you're certainly right. If it weren't for IBM, I doubt that I'd be loving using Notes 7 right now. Should have made a New Year resolution to look on the bright side.

  1. 13  Alan Lepofsky  | 1/5/2005 10:51:03 AM

    @9 Ben, please understand I was not trying to be insensitive to situations such as the one you outline. I was trying to be more general about the growth of Notes/Domino.

  1. 14  Ben Rose http://www.jaffacake.net/bensblog.nsf | 1/5/2005 1:00:50 PM

    @13 Alan - I know, and what you're saying is true. It's a bit like when we say they would be talking German in Paris if it wasn't for the British in WWII. It's true, but it doesn't mean the French like it very much...

  1. 15  A coworker  | 1/5/2005 1:10:36 PM

    Let's not forget one important point though (the one which makes Damien's story so fascinating):

    In today's IBM culture, Damien's success story would have been virtually impossible. He would of been pre-empted right away from the get go due to all kinds of business controls and what not, so his manager would not have been able to give him the freedom and trust he so desperately needed to pull it off, because the bean-counters would of been all over him before he could lay one finger on the keyboard.

    However, and this is for you, Damien:

    "It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat."

    { Link }

  1. 16  Ed Brill www.edbrill.com | 1/5/2005 2:23:45 PM

    I'm not sure I agree with that assessment, Co-worker.

    It's not a coding example, but take a look at the creation of the IBM Workplace Forum. They were able to do things that one would normally consider impossible in IBM, like turn around a customer mailing in 24 hours. There are still plenty of ways to be creative and innovative at IBM.

  1. 17  Ed Fisher  | 1/5/2005 3:16:10 PM

    The core formula language being coded by one guy that didn't know C++. This is what I am paying $100k a year in maintenance fees for!? :)

  1. 18  Sean Burgess http://www.phigsaidwhat.com | 1/5/2005 3:24:39 PM

    I actually have DNFS listed on my resume. I implemented it 3 companies ago to handle something important, I just can't remember what it was.

    In my mind, you really have to have 2 classes of IT developers. The ones who are super creative and work better just coming up with solutions on their own and the ones who need structure and guidance so that they don't royally screw things up. Too many companies don't take the time to recoginize the difference between the two and nuture them in different ways.

    Sean---

  1. 19  Christopher Byrne http://www.controlscaddy.com/ | 1/5/2005 3:31:05 PM

    Four memories about my Lotus ==>> IBM Transition:

    1. My first IBM Conference Call about an internal project we were doing for ITS started with "Your billing code for this conference call is..."

    2. EA2000 - A rogue application for travel expenses that incredibly difficult to use, that was created without any IBM approvals, had the internal auditors pulling their hair out, and still was mandated for use.

    3. The fact that I found out I could have bought a new GM (or was it Ford) car at far below retail as an IBM Employee *AFTER* I had just bought a new *used* car.

    4. Vision insurance was no longer free.

    Then there was the coworker from Florida who was stuck in NY on 9/11 while on a project. All of his clothes and his laptop were in a hotel that was damaged in the attacks. So he and another Floridian with no place to stay and no clothes rented a car and drove home. The expense was submitted and denied by the bureaucracy because of "Excessive mileage" (true story that was eventually worked out I believe).

    Alas, but life moves on and bureaucracy is everywhere:-).

  1. 20  Henning Heinz  | 1/5/2005 4:33:23 PM

    I think Ned Batchelder got it right:

    It's a good story of how a major feature got built against the odds by one determined engineer.

    And only a short part is about Iris and IBM. This is not an IBM issue anyway (from my opinion). A lot of cool shops have just disappeared (I made the transition while working for Computer 2000 and the company completely changed without the help of IBM). One of the reasons might be that a company just cannot stay "cool" if your colleague has to leave and less people buy your thrilling products/developments. As an outsider I never understood why IBM did that but they make a lot of money so it cannot be all wrong. They are the most admired company in Germany too (at least if you take the yearly polls about peoples preferred employer). But I like Alans comment in No(8), especially the "Would I like" statement.

  1. 21  Gary  | 1/5/2005 7:53:09 PM

    So that's what happened to my DCO toolkit.

    Good thing I ship the source code with it.

    What about Domino on Linux? That would never happen today.

  1. 22  Ed Brill www.edbrill.com | 1/5/2005 9:35:06 PM

    Right, I can see where the IBM of today would have zero ability to produce a project on LINUX.

  1. 23  Alan Lepofsky  | 1/5/2005 11:05:02 PM

    Gary, how about some MS products on Linux? Hmmm....

  1. 24  Axel  | 1/6/2005 4:28:05 AM

    Is IBM like one, big, coherent thing?

    Or is it that in time in one branch there is a good clima for creating good useful products and in others, well they are glueing together some stuff, don't care about performance, need a 2cpu server to make it run...

  1. 25  mugadafino  | 1/6/2005 9:53:33 AM

    so.. how many skunkworks "we should do this regardless of what the focus groups say" are still around at IBM? Innovation happens without planning in most cases. it seems to be the right thing to do by the programmer of the day ! - which in this case is Damien!

  1. 26  Axel  | 1/6/2005 12:07:28 PM

    @mugadafino: so I am a programmer and the results of my job are pure random?

    Shall the god of innovation hear my prayer and release his wisdom to my brain. bzzz bzzz bzzz

    Please stop programmer heroe stuff, read some good books, learn some good apis/frameworks/design patterns and work hard, even if its often boring. Very boring.

  1. 27  The Troll  | 1/7/2005 10:09:09 PM

    Damien is a lightweight. Big deal. He rewrote the formula engine. So What. It was 22 years old and maybe it should have been improved 33 times faster. In any event, it is an obsolete bunch of crap. Damien is the epitome of the Iris pompous loser who thinks his shit doesn't stink. Notes was invented by that sell out loser Ray Ozzie, not you, so stop acting like you had anything to do with it.

  1. 28  Ben Poole http://www.benpoole.com | 1/8/2005 6:06:13 AM

    Another brave anonymous comment - presumably from the same moron over at Damien's site.

    Bravo! You astound us all with your depth of insight.