BBC: Benefits computer failure chaos

November 26 2004

Engineers are trying to fix a massive computer failure at the Department for Work and Pensions (DWP) that has caused chaos in the benefits system.  More than three-quarters of its staff could not access payment processing systems on Monday after an update to desk-top computers went wrong. ...
About 80% of the DWP's network crashed on Monday and technical experts from Microsoft and the computer firm EDS have been working around the clock to find the fault.
Meanwhile staff were forced to communicate by fax because the e-mail system broke down.
Oops.
Link: BBC: Benefits computer failure chaos > (Thanks to Choddo and Two Fish)

Ed Brill | Highland Park, IL USA Category:  In the news  Add/View Comments [10]

Post a Comment

  1. 1  Ben Rose http://blog.jaffacake.net | 11/26/2004 9:06:35 AM

    "Since then, representatives of computer contractors EDS and Microsoft have been working around the clock to find the fault which has left the staff having to communicate by fax because the e-mail system has broken down. "

    Hmmm, Microsoft email system failing? Surely not...

  1. 2  Philip Storry  | 11/26/2004 9:40:44 AM

    Actually, to be fair, I understand that this isn't server-side. This article is better: { Link }

    It looks like they shafted the vast majority of client PCs. So their email servers are probably still happily running, without any faults whatsoever. But nobody can boot their copy of Windows to even start Outlook and read mail...

  1. 3  Alan Bell http://www.dominux.co.uk | 11/26/2004 9:49:53 AM

    Thanks for pointing out that article Philip, most of the reports on radio are useless, they provide zero information about what went wrong presumably because they don't think anyone can understand technical language. On the Today program they didn't once mention Microsoft, the spokeswoman kind of said it was a client side problem with an upgrade. How do Microsoft avoid bad press when their software is involved in a huge failure? I don't for a second believe that Microsoft is at fault here, some idiot did a stupid thing, but for anyone else this would be a mainstream press disaster and their share price would dive. So far Microsoft have kept their name out of the mainstream and only technical press like the Register are telling the whole truth.

  1. 4  Philip Storry  | 11/26/2004 10:31:16 AM

    Alan [3],

    No problem. El Reg may be biased, but at least their technical facts are usually correct. ;-)

    I've been thinking of how you could avoid this, by the way - after all, it would affect any mail client - not just our favourite one.

    Here's my suggestion - a PXE boot system to a server on site. That server contains some kind of ghost image which boots a tiny DOS-like kernel (can be done very quickly!), checks a flag, and then either continues into a maintenance mode OR transfers booting to the machine's boot sector.

    That way, you could boot straight into a recovery system and just re-image PCs without problems. In this case, they could just have shipped an image out oevrnight to those boot/ghost servers and then set the flag to go into some kind of automatic file comparison - just replace any files that differ with the known good ones from the backup image. Hey presto, all the PCs can come back up pretty quickly...

    Wow. I have some pretty cool ideas. Now all I need is a venture capitalist to fund me, and I can get rich with my new company - safe-boot.com!

    Oh, wait - that economic model died. Oh well - I'll just have to wait for IBM or someone else to implement this kind of feature. Providing they have the same idea, that is! *nudges Ed*

  1. 5  Alan Bell http://www.dominux.co.uk | 11/26/2004 10:32:54 AM

    Do you think they were using SMS or Tivoli? Rumours seem to be that they pushed out an XP patch in the form of some specific DLLs to all their W2K boxes. Overwriting W2K system DLLs with the XP ones can't be healthy.

  1. 6  Ed Brill www.edbrill.com | 11/26/2004 10:35:38 AM

    Whois Server Version 1.3

    Domain names in the .com and .net domains can now be registered

    with many different competing registrars. Go to { Link }

    for detailed information.

    No match for domain "SAFE-BOOT.COM".

  1. 7  Ben Rose http://blog.jaffacake.net | 11/26/2004 11:51:26 AM

    @5 - Indeed, how the patch was deployed to the wrong OS is beyond me. It would be like adding a Notes 6 patch to a Notes 5 client..not really possible.

    I'm guessing there's a printer doing a P45 somewhere in that organisation right now!

  1. 8  Paul Mooney http://www.pmooney.net | 11/26/2004 12:11:53 PM

    Might be wrong here but SMS can check the OS version before deploying an update.

  1. 9  Ben Rose http://blog.jaffacake.net | 11/27/2004 4:49:58 AM

    At the end of the day, this just looks like human error. Probably easier to fix than an exchange crash.

    (Sent from my Nokia 9500)

  1. 10  Randall Shimizu  | 11/29/2004 8:37:56 PM

    They have the worst of both worlds EDS and Microsoft. The NMCI project in San Diego has just about wiped out EDS. Some people have described NMCI project as a make or break project for EDS.

    I am hearing a lot of stories recently about about SP2 taking down PC's and networks. I am quite amazed that Microsoft is not doing more to help IT people understand the ramifications and internals of deploying SP2.