Wednesday, May 06, 2009

Only NSW Health Could Mess Up Like This!

I know it seems to be a little unfair to make critical comments two days in a row but this really is a ripper.

Power failure lasting 36 hours cripples hospital care

  • Kate Benson Medical Reporter
  • May 6, 2009

DOCTORS at more than 100 hospitals in the state could not access patient records or vital test results for up to 36 hours last weekend after a power failure crippled NSW Health's computerised database.

Some records were lost, X-ray and pathology results could not be accessed and staff were forced to use whiteboards to keep track of emergency patients after the main server shut down at 9am on Saturday because of a faulty circuit-breaker.

Back-up power from the Cumberland Data Centre, which provides computer access to the Greater Western, Greater Southern and Sydney West area health services also failed, plunging some of the busiest hospitals in the state into chaos.

Thousands of patients were affected, with doctors and nurses forced to take notes on paper and go to other parts of the hospital to collect hard copies of results, extending treatment times and adding to the confusion.

Some staff, who did not want to be named, said the weekend was chaotic and a shambles. One surgeon said it was fortunate no lives were lost.

.....

A spokesman for Mr Della Bosca said workers doing routine maintenance at the data centre had triggered the outage. No patients had reported problems connected to the blackout but a full investigation would be launched. "If necessary changes will be implemented to prevent a recurrence," he said.

with Louise Hall

BLACKED OUT

Hospitals at Westmead, Auburn, Blacktown, Nepean, Lithgow, Mount Druitt, Cumberland, Blue Mountains, Dubbo, Bathurst, Orange, Mudgee, Parkes, Bourke, Albury, Queanbeyan and Goulburn were affected.

Full article here:

http://www.smh.com.au/national/power-failure-lasting-36-hours-cripples-hospital-care-20090505-au1s.html

Further coverage is here:

Patients weren't at risk during hospital power failure: Della Bosca

LiveNews | Francis Keany and Richard Maxton

“There are no reports of adverse patient outcomes related to the temporary power outage.”

NSW Health Minister John Della Bosca

The health system is in damage control again after revelations wide-sweeping blackouts meant doctors at more than 100 hospitals couldn't access patients records or test results for 36 hours.

The power failure last weekend crippled the health database impacting on more than 100 hospitals.

Some records were lost and doctors were unable to access x-ray and pathology results.

.....

However surgeons and the NSW Opposition claim the power loss indicated a serious failure.

NSW Shadow Health Minister Jillian Skinner has told 2GB's Alan Jones the failure of the back up system is extremely concerning.

"The Minister's response is almost, 'Oh well, don't worry about it, no one's lives were at risk'. The same with the Health administrators that spoke out yesterday.

"I joined doctors in saying this is a real worry. We've got to get guarantees this will never happen."

More here:

http://www.livenews.com.au/news/patients-werent-at-risk-during-hospital-power-failure-della-bosca/2009/5/6/205376

There are a few things to be said here:

First the Health Minister would not have a clue and has no way of knowing if any patient was harmed or put at risk. The oppositions comments are more than reasonable I believe.

Second the fact that problems lasted for more than an hour or two suggests total incompetence at both a planning and operational level. Everyone recognises that there can be single point failures that can take a little while to sort out – but 36 hours! Just who was the clueless person who designed the fail safe systems that it took this long to become operational again. Management of the shared services environment should be held accountable and it would be more than reasonable to conduct a short formal PUBLIC enquiry to properly apportion blame and put appropriate remedies in place. I bet that does not happen!

Third, this enquiry should also examine how the manual back-up system worked so the whole Health Sector can learn any lessons that may be available. From the reported comments of staff it does not seem the manual systems coped all that well.

Fourth, I wonder why, if this happened at the weekend, why it has taken until Wednesday for the issue to be reported? Could it be that staff are a bit nervous about speaking out – Mr Garlig SC did suggest there was a little problem with bullying etc in NSW Health?

Fifth incidents like this need to be seen to be effectively handled so we don’t create undue anxiety regarding EHR initiatives. Just hiding the issue under the carpet does not help!

On a positive note this incident should prompt a risk review of all contingency and continuity plans state wide – and testing of them fully to ensure they actually work as intended. 36 hours is just way to long as further automation occurs. I bet that if done, the resulting document would never make it into the public domain as it should!

David.

3 comments:

  1. David
    This is surely a joke - a single circuit breaker failure equals a 36 hour outage! Who is responsible for the Data Centre DRP and how often is it tested? The distribution authority (Transgrid NSW) sets itself 99% uptime targets and achieves them. Hospitals and health facilities get priority restoration of service (quite rightly) as being an essential service. If a major commercial outsourcer (eg Fujitsu or IBM) had allowed something like this to occur they would face substantial financial penalties.
    If this is the best that NSW Health can do, any EHR initiative is simply a dream

    ReplyDelete
  2. You're right - anyone who actually thought this through would have checked the facts about a 36 hour outage. The journalist certainly should have...

    ReplyDelete
  3. How NOT to win practitioners over to an EHR:
    http://www.streetcorner.com.au/news/showPost.cfm?bid=10391
    Staff at Nepean Hospital have reverted to paper record keeping because the new medical records system is too cumbersome and prone to failure.

    ReplyDelete