In November last year a 'test' email was sent to the entire list of NHS email addresses (850,000) by an IT consultant working in the service. The test email was sent to what was thought to be a local group of email addresses created for the region. What actually happened was the email went to the entire NHS email list. This in itself is a face-palm moment for any IT worker but the real insult to injury came when around 80 or so annoyed NHS staff hit reply to ask to be taken off the list, that's 80x reply-all to 850,000 mailboxes.
Whilst this isn't a serious issue of security or patient safety it lead to emails being backed up for a few hours in the system, not ideal, but not a major incident. It should be noted that the system didn't fall over due to the huge volume of emails, which is actually testament to the underlying infrastructure.
The fault here is being laid at the door of IT contractor Accenture, they were specifically asked to ensure the system had failsafes to ensure this couldn't happen. It looks like the mistake was simply selecting a box titled "only in my organisation", which the consultant thought was only in her local group (a reasonable assumption), but the organisation in this rule was actually the entire NHS, a config fail of epic proportions.
The failsafe functionality is apparently still to be put in place by Accenture, I'd imagine there has been a stern email, or two (or 500 million) about this.
For the record, we all make mistakes like this in IT, it's just sometimes nice to enjoy the moments when it's somebody else :)
Full story can be seen at https://www.theregister.co.uk/2017/01/31/nhs_reply_all_email_fail_half_billion_messages/
Article Category Tags
Click one to see related articles hand-written by Cowshed WorksBusiness