Information Week: Congress Probes Case Of The Missing White House E-Mails

February 27 2008

Even the US Congress thinks so...

California Rep. Henry Waxman, the chairman of the House Oversight and Government Reform Committee, briefly outlined the ongoing saga of the White House e-mails in his opening statement, noting that the Bush administration in 2002 effectively dismantled the Automatic Records Management (ARM) System instituted by the Clinton administration in 1994.

The White House "intentionally dismantled an effective system and replaced it with a primitive alternative that just didn't work," said Waxman. ...

Theresa Payton, CIO for the White House Office of Administration, said in her prepared statement that the incoming Bush administration transitioned from Lotus Notes to Microsoft Exchange over a two-year period from 2002-2004, and that the ARM System established by the previous administration did not work properly with Exchange.

"ARMS was a custom-designed application," Payton testified, "and I understand that it was discovered that it just could not be effectively integrated with Microsoft Exchange."

As a result, thousands or even millions of e-mail messages to and from White House officials -- many of them generated during the crucial period in the run-up to the invasion of Iraq, in March 2003 -- have proven to be irretrievable.
Also of interest is David Gewirtz's update over at OutlookPower.

Link: Information Week: Congress Probes Case Of The Missing White House E-Mails > (Thanks, Richard)

Ed Brill | Frankfurt am Main, Deutschland Category:  In the news  Add/View Comments [53]

Post a Comment

  1. 1  Lawrence Micallef http://lcem.com | 2/27/2008 2:55:34 PM

    Just another example of the blind jump to Exchange without bothering to check the consequences... wonderful to see it happens at the highest of levels too.

    Let's hope this helps add ammunition to the arsenal of Notes/Domino Evangelists on why jumping into the Exchange arena is going to hurt.

  1. 2  Gregg Eldred http://www.ns-tech.com/blog/geldred.nsf | 2/27/2008 3:41:05 PM

    Nice job of pulling the appropriate quote, Ed. I simply linked to the articles (as they were yesterday), without going into too much detail.

    @Lawrence: Yes, the migration was probably poorly handled, but it was the TIMING of the migration. Just at the start of the ramp-up to the Iraq invasion. Plus, we have one very high, senior member of the White House staff that seems to have a hard time hanging on to his Blackberry. He has lost several, but no one ever said "Enough is enough."

    I find no humor in any of this.

  1. 3  Drew Ingersoll  | 2/27/2008 3:58:34 PM

    Ed

    The Waxman quote you have was in context of the new backup systems put in place - not the e-mail application itself being primitive. The way it is parsed here might give readers the wrong impression of the Notes vs. Exchange switch over. The point about rethinking your backup strategies/methods if you are going to switch is very valid however.

    It is my understanding that they are "finding" some of the old mail - still not all of it yet. Hopefully it helps us understand what was going on in the WH regarding war. As an IT person I wish we could see the emails and discussions on why the switch to Exchange - and maybe holding someones feet to the fire for the obvious mess they made of things with our tax dollars.

  1. 4  Lars Olufsen http://www.olufsphere.com | 2/27/2008 4:17:59 PM

    It is simply beyond me how something like this can happen.

    Did they not run a Proof Of Concept?

    Was there no Pilot Implementation?

    No Test environment set up to run a migration on safe data?

    Risk assesment, anyone??

    This is the administration of the person leading the government of the most powerful nation in the world?

    Doesn't even matter to me if Notes or Exchange is hero or culprit, or if it's the backup routines that failed.

    It's amateurish at best. I don't even want to think about what it is at worst!

  1. 5  Ed Brill http://www.edbrill.com | 2/27/2008 4:23:56 PM

    @3 I am not sure whether Representative Waxman was making that distinction or not. Like most non-IT end-users, he was likely thinking about the system in totality. I quoted it that way because in those cases where a customer has moved their e-mail from Notes to Exchange, we often see that they don't put in place equivalent applications -- such as compliance/records management, as in this case -- as they had before. I have no idea why that wasn't done at the EOOP, but it seems like it would have been an important part of the proposal to migrate.... at least if it wasn't a "political decision" ;)

  1. 6  mark hughes  | 2/27/2008 6:30:24 PM

    Maybe if Waxman had given Exchange some Super Human growth hormon from his investigations, it would have worked.

  1. 7  mark hughes  | 2/27/2008 6:41:01 PM

    Maybe if Waxman had given Exchange some Super Human growth hormon from his investigations, it would have worked.

  1. 8  Keith Brooks http://lotustech.blogspot.com | 2/27/2008 8:41:16 PM

    "that the ARM System established by the previous administration did not work properly with Exchange"

    Well gee, do you think a system meant to run with Domino would ever work with Exchange?

    @4, you realize of course that when a new President is elected, the old staff "disappears" overnight usually. So the It peopel who knew and managed it were tossed, in came Bush's people, who knew nothing about Domino and proceeded to bring in Exchange because that is what they know.

    Ok, a bit simplistic, but you get the idea.

    Another notch in my "companies/organizations which faltered after moving away from Domino".

  1. 9  John Smart http://www.greyduck.com | 2/27/2008 9:56:39 PM

    Here's a link to an April 11, 2004 USA Today article about a white house spokesman defending white house use of RNC accounts... which have also been lost.

    { Link }

    But I know I'm getting political here, and outside of Ed's main point re migration woes and pains of mail management that Notes can likely do better than Exchange, so I'll circle back and end my comments with this...

    My guess is that the RNC uses Exchange, too.

    >8^D

  1. 10  Dave Madison  | 2/28/2008 12:39:16 AM

    Seems the "custom designed ARMS system" and the lack of any integration testing before implementation is more the story here, and not Exchange. I'd venture to guess they would have had the same issues had they gone with Oracle, Zimbra, or any other mail system for which ARMS was not custom designed.

  1. 11  Dave Madison  | 2/28/2008 12:47:02 AM

    Was this the same ARMS system running, as you suggest, against a Domino server that managed to lose emails during the Clinton Administration?

    { Link }

    { Link }

  1. 12  Lars Olufsen http://www.olufsphere.com | 2/28/2008 2:11:27 AM

    @8 ... are you kidding?

    Are you meaning to say, that in the White House, all IT-systems are potentially replaced every four years because of the backgrounds and previous experiences of the staff?

    I'd venture a guess that, in most EVERY other "company" or "administration", you hire people to work with the tools already at hand, instead of changing tools every time your staff moves, leaves, rotates etc.

    But if every new administration at the White House has to replace all the IT-systems that doesn't fit their knowledge profile, then I can easily understand if the administration acts a little dubious now and then, especially in the 3 years of transition. ;-)

  1. 13  Mark Hughes  | 2/28/2008 7:13:20 AM

    @12 You must remember when Bush and his staff arrived all file cabinets were empty, stuff was torn up, and "w" was missing from all the keyboards. Not exciusing the IT switch but seems like everything was ripped out before they got there.

  1. 14  Keith Brooks http://lotustech.blogspot.com | 2/28/2008 8:25:14 AM

    Lars, not every system, but every system administrator or at least many of them.

    Politics is not run like a business, it is more run like a rip and replace migration.

  1. 15  Wayne http://www.portalpartnership.com | 2/28/2008 8:51:08 AM

    Off on a tangent perhaps but today I received and HTML email from virgin atlantic (which rendered quite perfectly in my 8.0.1 Notes client :-)) when I noticed the following line:

    If you would like to view an enhanced version of this email in your web browser, complete with illustrations, or you are viewing this email in Lotus Notes, please go to:

    :-( most depressing!!

    Perception is reality so for people flying Virgin Lotus Notes is a poor choice it seems.

  1. 16  Jaime Bisgrove http://augurgroup.com/b/ | 2/28/2008 9:33:36 AM

    IMHO it would be a better tact to work towards better portability. By that I mean make it easier to migrate from and to Exchange. A lot of the Lotus Notes fears (however unfounded) are that once in Notes, its hard to get back out.

  1. 17  Darren http://www.dadams.co.uk | 2/28/2008 9:43:05 AM

    I know of a couple of companies who moved to Exchange and discovered it was a BAD mistake... trouble is you'll never get anyone to be a reference for that.

    So, nice to see a case study of ineptitude in the public domain. Mind you, it looks like a mixture of technical failings and carbon-based life-form error. I don't think Exchange is 100% to blame, but it's a good cautionary tale for any organisation looking to make a swap without a solid reason / business case.

  1. 18  Kevin Mort  | 2/28/2008 10:07:08 AM

    @8/14 - Yep exactly. Especially when the incoming administration is of the opposite party. Things get all pear shaped then for sure!

    Of course since we're talking Microsoft here, rip & replace is SOP. ; )

    @15 - Yes that is indeed sad. What is worse is that they're sometimes developing those emails with Outlook in mind - a bad idea for interoperability with internet communications.

  1. 19  Danny Lawrence  | 2/28/2008 12:11:55 PM

    @8 -- I'm pretty sure the front line IT folks in the White house are Civil servants, and as such are pretty hard to get rid of. If GWB's management team wanted the change, they could impose it, but it is not like they walked into the White House on 1/20/2001, and had to hire all new Systems people.

  1. 20  Ed Brill http://www.edbrill.com | 2/28/2008 2:11:26 PM

    @10 which sort of goes to the question of, on what basis did they justify the change? Also, I would have expected you to at least finally acknowledge the switch to Exchange -- you spent months trying to put doubt on whether they were actually using Exchange when the story first broke. You kept saying that it was a switch to Outlook, how did we know it was a switch to Exchange.

    Perhaps you knew the embarrassment that was coming.

  1. 21  John Smart http://www.greyduck.com | 2/28/2008 2:13:38 PM

    @13: Yeah, that's what I had heard too, but it turned out to overblown.

    CNN: { Link }

    NYTimes: { Link }

  1. 22  Rock http://www.lotusgeek.com | 2/28/2008 2:25:58 PM

    Ed, you know I blogged about this yesterday, right? I gave my take on it there:

    { Link }

    I had a problem earlier today in that the post mysteriously disappeared, but I have recovered it, and I am in the process of recovering the responses as well.

  1. 23  Ed Brill http://www.edbrill.com | 2/28/2008 2:48:59 PM

    @22 this post is dated yesterday, too :-)

  1. 24  Rock http://www.lotusgeek.com | 2/28/2008 7:45:07 PM

    @23 (Ed) - Ahh. Sweet ;) I checked yesterday before I posted mine to see if you had blogged about this, and you hadn't; maybe you did it offline and I just missed it, or you posted around the exact same time I did. I posted mine about 2:40PM EST. That also explains why you didn't comment over there ;)

    Did you get a chance to read my post? I thought you'd like the story about Sen. Kennedy's office...

    --Rock

  1. 25  Ben Poole http://benpoole.com | 2/28/2008 7:52:09 PM

    @10, 11, you're so rabidly pro-MS, why don't you fly your colours fully? A GMail address doesn't cut it, surely?

    @20 shame on you Ed, you know MS don't get embarrassed... ever!

  1. 26  Ed Brill http://www.edbrill.com | 2/28/2008 8:09:53 PM

    @25 Ben, Mr. Madison works for Microsoft. Just look him up on linkedin. He used to do competitive work for them, left for Foldera, and now is back in their mobile team.

    His vague lawyerly "reasonable doubt" approach gets old, but at least he's consistent.

  1. 27  Dave Madison  | 2/29/2008 1:38:58 AM

    @25. I'm not sure how that response answers my question. Was ARMS the same system the Clinton administration was using when it too "lost" mail stored on their Domino servers. Yes or no?

    @26 Same question applies. Yes or no? I know you like this to be about Exchange, but given ARMS history, seems it's not the mail server that is the problem. Unless you can prove ARMS never lost mail before the WH moved to Exchange.

    Not exactly sure how where I'm employed is relevant to the question about the functionality of ARMS.

    @20. Yes, the WH is reportedly running Exchange. But again, given there is a history or ARMS losing mail regardless of the back end I don't why that matters.

    Nice attempt at avoiding the question by poisoning the well. But it doesn't work. The question still remains.

  1. 28  Ed Brill http://www.edbrill.com | 2/29/2008 7:55:54 AM

    I don't think either of us are qualified to comment on ARMS and what it did or did not do. Unless another credential on your resume is having worked at the White House?

  1. 29  Timothy Briley  | 2/29/2008 8:37:33 AM

    @28 -

    If anyone outside of the federal government is qualified to talk about this topic, it's David Gewirtz at dominopower/outlookpower. Everyone should really take five or ten minutes and read his latest post. It's sad and scary at the same time.

    { Link }

  1. 30  Charles Robinson http://cubert-codepoet.blogspot.com | 2/29/2008 8:46:47 AM

    @27 - It's a custom system originally implemented in 1994 during the Clinton administration, in response to a federal finding that they must capture electronic communications. I don't think anyone reading this blog knows for sure what form it takes today or what changes have happened through the past 14 years. If they do it's probably covered by a Secret seal so they can't tell you, anyway.

    Let's assume ARMS today is identical to the original implementation. The system was obviously poorly developed, and the implementation was even more poorly managed. Those are the issues to me. Not the technology used, but the lack of action to fix the problems since it's apparently never worked.

    It does beg the question why has this same scenario been repeated for every administration since ARMS was implemented. The Presidents (Clinton and now Bush) must know that it's broken, so why don't they put some pressure on people to fix it? It would seem they like neither are a Harry Truman ("the buck stops here").

  1. 31  Ian Scott  | 2/29/2008 8:57:29 AM

    @29 - The 'wagon wheel' quote is covered here:

    { Link }

    Some serious correction is required. IBM should be very concerned if Lotus Notes is being likened to Betamax and the assertion is made that Lotus Notes is no longer supported.

    This cannot be allowed to stand. Congress needs to be told.

  1. 32  Ed Brill http://www.edbrill.com | 2/29/2008 9:06:20 AM

    @31 I'm working on it.

  1. 33  Charles Robinson http://cubert-codepoet.blogspot.com | 2/29/2008 9:07:10 AM

    @29 - "I wouldn't want to do business with somebody ... still using wooden wagon wheels."

    Now that's plain silly. There is ample evidence of the government doing business with people still using wooden wagon wheels. Laura Bush took a mule-drawn carriage ride (which uses wooden wagon wheels) while she was in Charleston a few years ago. In fact, the Reagan's, the Clinton's, and both sets of Bush's have, as well as many members of Congress. :-)

  1. 34  Drew Ingersoll  | 2/29/2008 9:52:11 AM

    Wow! Nice understanding of Technology there Congressman Issa

    Mr. ISSA. The Clinton Administration used Lotus Notes, right?

    Ms. PAYTON. Yes.

    Mr. ISSA. Lotus Notes no longer exists, right? It is no longer supported.

    Ms. PAYTON. It is no longer supported. Some groups may still use it, but it is no longer supported.

    Mr. ISSA. I wouldn't want to do business with somebody still using Lotus Notes or still using wooden wagon wheels. If I understand correctly, though, certainly I checked with the House of Representatives, we can no longer support it for members who want to stay on it.

    Mr. ISSA. Okay. So here we have a situation where the Clinton Administration is on a platform that has to be phased out. Simply, they lost the war of who is going to supply emails. A period of time goes on in which Yes, we are dealing, to Dr. Weinstein's concern, with getting good archives, but we are also dealing with the fact that I can't play my Betamax tapes any more, either, and I can't seem to find anybody who has a Betamax player any more.

  1. 35  Ed Brill http://www.edbrill.com | 2/29/2008 9:56:19 AM

    @34 this is the specific quote that I am working on right now.

  1. 36  Drew Ingersoll  | 2/29/2008 10:08:33 AM

    Ed

    Invite him to do the keynote at Lotusphere next year.

  1. 37  Ian Scott  | 2/29/2008 10:21:57 AM

    @32/35 - I really do wish you all the best. Perhaps they mean it's not supported by Congress but even so the record needs to make that clear. If that's not what they mean I must meanwhile resist the temptation to speculate as to the originating source of it.

  1. 38  Ian Scott  | 2/29/2008 11:37:30 AM

    Crikey. I only just absrbed the bit about not wanting 'to do business with somebody still using Lotus Notes...'.

    At face value this means Ms. Payton does not want to do business with IBM or Coca Cola or the British Government or the British Royal Navy or.....

    There are wide reaching and serious implications of making such a statement (eg nuclear weapons). It sounds like Ms. Payton has wandered past her pay grade.

  1. 39  Ed Brill http://www.edbrill.com | 2/29/2008 11:47:17 AM

    @38, Ian, Ms. Payton didn't make the statement -- A US Congressman did. THAT is the frightening part.

  1. 40  Drew Ingersoll  | 2/29/2008 11:54:26 AM

    @38 - that was Congressman Issa(R) of San Diego making that comment. Ms Payton is the clueless flunky in the White House IT department - when you read the testimony it is apparent that Mr. Issa is using the offending statements here as a cover up for the incompetent IT department put in place by the Republican WH(Alan R. Swendiman=Brownie from Katrina).

    If you want to read the testimony unvarnished it is at { Link }

    If you read it you will see that there is some controversy in early parts of the testimony as whether to allow the previous CIO to make a statement about their incompetence - R's line up on on side and D's on the other - it continues through the entire hearing. Mr. Issa is trying to throw a lifeline that Notes is dead and that is why it HAD to be done this way.

    Sadly, Not only did the ARMS system not work with Exchange(duh?) but they went to Booz Allen to build another system to be Exchange compatible and it was scrapped as well.

    Finally Ms Payton realized that "hey maybe other dept. in the government have this issue" and decides to now implement a Documentum solution that DoD uses. They are still working on it.

    Not much collaboration going on in our government.

  1. 41  Ian Scott  | 2/29/2008 12:15:17 PM

    @Ed - OMG! I stand corrected (despite my best efforts I do miss things on this BlackBerry). Thank you. That really is terrifying.

  1. 42  Ian Scott  | 2/29/2008 12:18:44 PM

    @40 - many thanks for the link

  1. 43  Charles Robinson http://cubert-codepoet.blogspot.com | 2/29/2008 2:15:39 PM

    @40 - To be clear, McDevitt was not the CIO prior to Payton. He reported to Payton for a while, then to a Deputy Directory. In the reports I find he is simply called a "computer technician".

  1. 44  Steven  | 2/29/2008 2:44:33 PM

    @40: There is a lot of collaboration going on, but not all areas have the mandates that the White House has. Keep in mind who made up these email retention requirements and why that had to be implemented. Many agency have retention mandates for various type of records, electronic or otherwise, but some of these specifically state that the electronic mail system will not be used to maintain these records. Again, this would not apply to the White House

    This may or may not do positive things to your image of the federal govt, but check out NARA Code of Federal Regulations - 36 CFR Subchapter B, Part 1234 -- Electronic Records Management

    Granted this is taken out of context...

    (i) Agencies must not use an electronic mail system to store the recordkeeping copy of electronic mail messages identified as Federal records unless that system has all of the features specified in paragraph (b)(1) of this section.

    (ii) If the electronic mail system is not designed to be a recordkeeping system, agencies must instruct staff on how to copy Federal records from the electronic mail system to a recordkeeping system.

    (c) Agencies that maintain their electronic mail records electronically shall move or copy them to a separate electronic recordkeeping system unless their system has the features specified in paragraph (b)(1). Because they do not have the features specified in paragraph (b)(1) of this section, backup tapes should not be used for recordkeeping purposes. Agencies may retain records from electronic mail systems in an off-line electronic storage format (such as optical disk or magnetic tape) that meets the requirements described at 36 CFR 1234.30(a). Agencies that retain permanent electronic mail records scheduled for transfer to the National Archives shall either store them in a format and on a medium that conforms to the requirements concerning transfer at 36 CFR 1228.270 or shall maintain the ability to convert the records to the required format and medium at the time transfer is scheduled.

    (d) Agencies that maintain paper files as their recordkeeping systems shall print their electronic mail records and the related transmission and receipt data specified by the agency.

    { Link }

  1. 45  Drew Ingersoll  | 2/29/2008 3:00:09 PM

    @44 - So if there is a lot of collaboration going on why did the White House IT waste so much money on a)trying to get AMS system to work with exchange b)hire Booz Allen to build something new (EIMRCS or whatever it was called and scrapped) - and finally c) use what other departments/agencies use with exchange (Documentum) -- taking all of 6 years to get there? 6 years? Why didn't someone just ask how other departments did it to start when decision to migrate to exchange was made.

    I love (d) BTW - lets print them out!

    Where is paragraph (b)(1)? what are specified features?

  1. 46  Steven  | 2/29/2008 3:25:37 PM

    See the link at the bottom of @44 for "(b)(1)"

    Hey, don't knock "(d)", that is the one email admins LOVE!

    Considering that the WH has to keep EVERY email (as it seems), that is multitudes of levels higher than the NARA requirements that many other agencies operate under. You need to understand was is deemed a "record" and apply that to email for other agencies. Heck, some records only have to be kept for 180 days. Compare that with what the WH is expected to do.

    But one could only make assumptions as to why the White House choose to do their own thing for their unique set of circumstances, assuming that is indeed what they did.

    Sorry if I'm being a bit vague, but there are some ethical boundaries I have to not step over.

  1. 47  Nathan T. Freeman http://nathan.lotus911.com | 3/1/2008 6:49:37 AM

    I know you can't do this, Ed, but I have to say, in the current political climate, I would have to play the political card to the hilt...

    Things the Bush Administration can't find:

    Osama bin Laden

    WMDs in Iraq

    Federal government records in an email system

    The moon with a telescope.

    Or perhaps...

    Lies the Bush Administration believed:

    Saddam had weapons of mass destruction

    It was okay to spy on Americans

    Notes is dead

    Or even...

    Thinking of migrating from Domino to Exchange? Join such illustrious company as Home Depot, CompUSA, and the Bush White House! Great move!

    I mean, it's not like they can get back into the White House again. Even the Republican candidate considers the current administration to be like the Three Stooges.

    And MS clearly knows it, because they're trying to distance themselves from the implementation and lay the blame on the prior system implementation, aren't they?

  1. 48  Pedro Quaresma  | 3/1/2008 7:39:57 AM

    "Thinking of migrating from Domino to Exchange? Join such illustrious company as Home Depot, CompUSA, and the Bush White House!"

    Brilliant Nathan, this could be IBM's next advertising campaign! ;-)

  1. 49  Dave Madison  | 3/4/2008 12:00:52 AM

    @30). Exactly the point, Charles. This is more an issue of the functionality and effectiveness of ARMS and less about what mail server it is running against. To try to make this an Exchange issue is really a stretch. This really has much more to do with the competence of the people running the system, since it's happened both on Notes and on Exchange.

  1. 50  Dave Harris http://www.wavysworld.com | 3/5/2008 4:04:36 AM

    Did they manage to keep records for 29/02/2004?

  1. 51  Charles Robinson http://cubert-codepoet.blogspot.com | 3/5/2008 7:41:31 AM

    @49 - The issues with ARMS while the White House was on Notes only applied to the Office of the Vice President, and were a result of it not being implemented properly. It's not the "people running the system", it's the people in charge of the implementation. To be more explicit, it's the CIO and the higher-ups, not the staff who have to support it. As we have seen in the case of Steven McDervitt, the people responsible for the administration of the systems raised objections and were ignored.

    The fact is the White House did the migration to Exchange with no archiving solution in place. ARMS was a custom-built archiving solution for Domino and couldn't possibly be reworked for Exchange. It would be akin to running Exchange on an iSeries. It's just not possible. I do think people should be questioning the decision making process that led to the Exchange migration, particularly in light of recent, inaccurate, testimony.

  1. 52  Ed Brill http://www.edbrill.com | 3/6/2008 12:33:04 AM

    @all regarding the testimony from last week, some very interesting updates today. Hope to post something very soon.