January 4 2005
Damien, buddy, I hope you feel better after this 5,000 word entry....
While I was on the template team I developed the original Notes Welcome page, I did work on e-mail and a lot of work on some obscure administrative applications, among other things. At the time I quit I was working on new features for the R5 point releases. But I hated it. I hated pushing around pixels all day, negotiating with the Quickplace and Sametime teams who wanted tie-ins and functionality on the Welcome page. I frequently worked 80 hours a week, was stressed and was seriously burning out. Why did I work so hard at something I hated? I don't know, I think because I wanted desperately to get away from it. So I thought if I just worked harder, someone would be impressed enough to give me a shot working on something I wanted to work on. Something more geeky and computer sciencey, instead of the touchy-feely UI crap where marketing types argue with me all day about how they think it should work and look.Disclosure: When I moved to Lotus product management in May, 1998, my initial responsibility was for e-mail migration tools. When I completed the licensing agreement with Binary Tree for their DUS tools, my second product management assignment was to be responsible for a Notes client feature we then called "Headlines", which became the Notes R5 Welcome Page. I worked with these young, aggressive programmer types (well, one in particular, wink wink) who thought they knew how the new feature should work and look. [Not to put too fine a point on it, but the fact that the feature was spec'd as "Headlines" and came out as "Welcome Page" was not a marketing decision, and with hindsight probably not the right decision. The Workplace welcome page in Notes 6.5 and 7 comes pretty close to the original "headlines" vision. But I guess that's another blog entry ;)]
Anyway, back to Damien's story:
No one on the language services team, or even at Iris, understood the runtime engine internals completely, also known as "Compute" (the compile/decompiler was called "Compile", great names eh?). There was no documentation, the code was barely commented, many variables had one and two character names, wild pointer arithmetic was everywhere and not even the stored byte code format the compiler produced was documented. Bugs were piling up and there was no one to fix them. On top of that the engine was becoming a bottleneck for server performance, particularly for building view indexes. The engine needed to be rewritten from the ground up, it needed to be made maintainable, it needed to be faster, and it needed to be completely backwards compatible.Damien mentions this several times in his essay -- the importance of backwards compatibility being designed in from the outset. Damien's overall effort shows that backwards compatibility while moving the ball forward is absolutely achievable -- why some vendors always throw things out and start over is beyond me.
Now you might think that I produced a bunch of design documents and specifications and presented them to the various senior engineers and architects, but I didn't. I remember being surprised by this myself. Even Wai Ki didn't have much to say about my design or how it should be implemented. The philosophy was that if I did those things, everyone would meddle with the design and nothing would get done. It's truly easier to ask forgiveness than to ask permission, not to mention things get done a lot faster if you just do them. It sounds weird, but that was big part of the culture at Iris. Just get it done and do it right, do whatever it takes. So that's the way I worked, I was just going to go headfirst into it, damn the torpedoes.Despite Damien's pronouncement later in his essay, I would assert some of this culture still survives in the Notes/Domino team. Just the other day, I was offered the opportunity to code up a multi-level undo feature....oh how I wish I still had the chops to do it. Someone will for a future release, I just know it...
Damien mentions his then-manager Wai Ki Yip several times in the essay. Wai Ki is presenting a session on NSF-DB2 at Lotusphere 2005. I had a chance to review Wai Ki's LS2005 presentation earlier today -- I'm very impressed. That's one you'll want to see.
That's when I started adding all the new stuff that I wanted to add to the engine. I remember I posted a question to a forum dedicated for professionals who worked with Notes and Domino (The Business Partner Technical Forum), and I asked what sorts of features they would want to see in the new engine. That question got over a thousand replies. Some of the requests where reasonable, some where stupid, some I just didn't understand. But I read each and every one and tried my best to consider them.I think this kind of vendor/partner interaction was completely unique at the time. Now, blogs fill some of that need, but I still would assert that the level of access that Lotus customers and partners had and have direct into the engineering team is unparalleled in the industry.
My time at Iris during R6 was the best I've ever had in my professional career, almost every day was a new challenge and game to play. But ultimately IBM killed Iris. In late 2001 IBM folded Iris Associates and converted all the employees to IBM. All the energy the Iris building had was seeping out. Really it was more pushed out by the BS that IBM management kept trying to sell us, like that nothing important would change and you'd still be able to work unfettered. Yeah right, just ask anyone who still works there how much time they spend battling bureaucracy vs. writing code. But such is big corporate management, they don't even realize how much they're damaging things, they are essentially politicians not engineers. It's sad, but I'm grateful for the time I had at Iris anyway, if it was still there I'd go back in a second.Lest anyone think I was going to shy away from this...
Damien is right. The culture is changed. Like a lot of companies, the mid-to-late '90s were a completely different kettle of fish from a work environment perspective than today. I know some ex-colleagues at Microsoft often tell me "it feels like Lotus used to feel". Well, that's the way it was, and just because Lotus has changed doesn't mean that some or all of that change was evil.
One of the things I liked about adding some "bureaucracy" to the Notes/Domino environment was that it put an end to rogue projects. When I moved to Lotus product marketing in 1999, we launched an effort codenamed "Bluejay", which was a collection of interesting rogue projects -- including Domino Network File Store and Domino Collaboration Objects. Neither of these features were really ready for primetime; DNFS was implemented on an outdated protocol (Netbios), ran on one server platform, and had a hardcoded connection limit; DCO had no long-term commitment behind it, just a couple of enthusiastic product managers. Today, new features or projects have to have a long-term plan before they can ever leave the door -- meaning a customer won't get stuck holding the bag if they start using it.
One of the things I didn't and don't like about adding some "bureaucracy" is that sometimes, decisions are forced upon one product for the greater IBM good. IBM says we will ship all software products in a consistent number of international languages. That means Notes, too, even if the market for Notes in outer Mongolia (a hypothetical!) is minimal. But as an IBM shareholder, I completely get the reason for this decision -- the customer in outer Mongolia shouldn't get stuck if they buy into the IBM strategy, with a lot of "oh, but we're not translating that product" kinds of answers.
Anyway, thanks to Damien for a fascinating look at how the Notes formula language came to be rewritten for Notes/Domino 6. Still today, a lot of people tip their hat in his direction for an incredible effort. I think it demonstrates so many things that are just core about Notes and Domino, and why the product continues to be #1 in the market after fifteen years.
Link: Damien Katz: Formula engine rewrite >
(Update: Check out the comments on Damien's site -- the original version's author has commented ;))