This useful article appeared a few days ago.
Learning from the Qld Health payroll fiasco
The torturous implementation of Queensland’s Health Payroll system will almost certainly be remembered as one of the most disastrous IT projects in our country’s history.
What began as a $6.19 million contract between the State of Queensland and IBM Australia to replace QLD Health’s aging payroll system eventually led to over 35,000 payroll anomalies and will ultimately cost taxpayers $1.25 billion.
While the final autopsy of the project is yet to be concluded, the project has been subjected to numerous reviews and a state level inquiry that saw a number of key IBM and QLD State Government personnel take the stand.
While we will have to wait until the end of July to see the report from the Commission of Inquiry, the public hearings have so far provided sufficient glimpses of how and why this project went so far off the rails.
Troubled beginnings
There were early signs of inadequate internal governance and mismanagement well before the QLD Health payroll project was awarded to IBM in December of 2007. The Queensland Treasury had spent the preceding two years "burning through the budget" by attempting to implement a standardised SAP-based HR and finance system across the whole of government operations - the very same platforms that were picked for the Queensland Health payroll project.
To aid in the software development and implementation, Queensland Treasury hired a large number of contractors from a mix of external consulting companies including Accenture, IBM, Logica and SAP.
Former managing director of Accenture, Marcus Salouk, described the state government progress between 2005 and 2007 as “burning through their budget without getting commensurate outcomes”.
“The Queensland Treasury were not experienced system implementers and in my observation, not professional project managers. I was concerned that going through that process, they would end up with insufficient budget to actually get where they needed to get to with implementing the new system”, Mr Salouk told the inquiry.
After gaining little headway, the state government decided to relinquish its role as a systems implementation lead to a single prime contractor.
The role was awarded to IBM in December 2007 with CorpTech, the specialised business unit of the Treasury, entrusted with the responsibility of managing the prime contractor, IBM.
The delays suffered during the wider state rollout of the SAP-based system had already highlighted that the departments within the state ,including Queensland Health, were not ready for a standardised system headed up by an external prime contractor, as it would require the government to detail its specifications with a great amount of accuracy.
Former program director of CorpTech, Darrin Bond, said in a witness statement that just prior to IBM being awarded the prime contractor role, the departments within the state “were still debating and arguing about what they would or would not get and what they would and would not accept”.
In the absence of the internal requirements being agreed upon for this government-wide system, the scene was set for a prime contractor to come in under a continually varying scope environment and ultimately leading to a project blowout of both time and cost.
The writing was already on the wall.
….. (Lots more of the saga here)
Lessons learnt?
Questionable procurement practices aside, delivering an ICT project of this complexity and scale is not a joke. And the task was made harder by basing implementation on loosely defined business requirements within an impossible timeframe. A project blowout was always on the cards.
The system failed critical user acceptance testing (UAT) processes but instead of addressing the issues, the bar for testing was simply lowered and less stringent guidelines adopted in an effort to get the system out the door to meet a time imperative.
There is little doubt that the state government should have acted to address the underlying issues instead of ignoring the risks and letting a flawed system go into production. A system that left thousands of Queensland Health employees underpaid, overpaid or not paid at all.
The Queensland government obviously isn’t alone when it comes to government run ICT project blowouts. New South Wales had the failed Tcard project, while the Victorian government botched up the deployment of the Myki smartcard systems.
These were all flawed systems that were rolled out prematurely and resulted in significant blowouts at the expense of taxpayers. They also highlight the systemic deficiencies in internal governance, contract management, strong controls over budgets and thorough testing and implementation regimes.
While the recriminations over the $1.25 billion bungle in Queensland will continue long after the Commission of Inquiry hands its final report to the state premier, the real question is if any lessons have been learnt from the big-ticket failure. The taxpayers would certainly hope so lest they end up holding another costly clean-up bill in the near future.
Read much more here:
As a reminder of just how hard even apparently simple large scale IT can be we also had appear this week this one:
Another Government IT Debacle: Ultranet facing scrap heap
Date June 19, 2013
Jewel Topsfield
The disastrous $180 million Victorian school intranet could be scrapped at the end of the month prompting fears that months of student work and reports would be lost.
The four-year contract with NEC to run the troubled network has not been renewed days before it expires on June 30, with a decision yet to be reached on its future.
Victorian Education Minister Martin Dixon said the government was committed to protecting the Victorian education system from the "Ultranet debacle", which he said the Auditor-General had confirmed was "botched from conception to implementation by the former Labor government".
Mr Dixon said the Ultranet had already cost Victorian taxpayers at least $180 million – three times its original budget – despite being used by only 4 per cent of the intended 1.5 million teachers, parents and students.
Lots more here:
What seems to come from all this is that just no-one seems to be able to learn the lessons from failed projects despite the fact that what you need to do for a successful project has been well known for years and are well understood by most competent project managers.
I have to say the difference between what is understood as needed to be done and what seems actually to be done in many projects is a conundrum for me. The difference between conception and execution is a major problem and - as in the Qld Health example - what seems to happen is that things start with a clear sensible plan and are then disrupted by externalities that are not under the control of either the project team or project leader.
One really wonders why we seem to have to learn the same lessons time after time!
David.
Why does the government continue to use large consultancies like IBM and Accenture to do this work?
ReplyDeleteWhere is the obligation to the community as a whole when such huge slices of taxpayers money result in such a debacle.
Can you imagine one of the large engineering companies building a power station or a bridge that fails and then just walking away with a shrug of the shoulders blaming government incompetence and weak or moving requirements as the nuclear disaster rolls out? They would be crucified in the media.
Of course a payroll system is not as frightening as a bridge or power plant failure, but the loss of $1.25billion in revenue does just as much harm when such large amounts of money fail to get used for the needy in society and go into the coffers of large multinationals.
There needs to be a sense of accountability here and discipline on both sides.
@anomymous01:00pm - agree entirely. It continues to happen and there is no accountability from these large players - and not just in Australia.
ReplyDeletePerhaps time to allow some local players a go to see if they can do better. I also think that the software vendors are to blame here as well - their systems are just too complex and archaic. Look at HealthSmart, the UK NPfIT etc, and you see common denominators - old tech vendors and bid SI companies.
Governments have problems, vendors sell solutions.
ReplyDeleteVendors assume that governments have worked out what their problems are and can define accurately what solutions they need.
Alternatively, vendors assume that they can work out what the problem is at the same time as they are delivering the solution.
In my experience both these assumptions are wrong.
The difficulties are magnified by the business case/RFT process where big decisions are made before a full understanding of the problem has been achieved.
The only alternative to the big bang, business case/RFT process is a slower, step by step approach.
Unfortunately, any vendor working on the early analysis/strategy phases (referred to as "above the line") make themselves ineligible for later phases. And to compound the difficulties, government insists on full project time and cost estimates at the start.
IMHO government's only have themselves to blame. They keep doing the same bad things over and over again and expecting different outcomes - which is one definition of insanity. It is the government's responsibility to make sure the solution they buy matches the problem they have. The vendor cannot know this.
The insanity will continue until there is a major change in the way governments make decisions in the light of limited knowledge and of great uncertainties.
It is my understanding the system went live on DAY 1 without any parallel running; once, twice or thrice. Surely that is an essential role and responsibility of the prime contractor and project manager. Therefore, if there was no parallel running the question to be asked is why not; and if there was even one attempt what were the results?
ReplyDeleteNormal practice is to conduct acceptance testing before any form of live production is attempted, parallel or not.
ReplyDeleteAcceptance testing means that the system owner (the government in this case) is happy that the system performs as required - re functional and performance.
This is why requirements are rather important. If you don't know what the system is required to do, you can't tell that it won't do it.
Decisions re acceptance testing and strategies like parallel running should be owned by the steering committee, not the project manager. The steering committee is the government's representative on the project.
There is a difference between responsibility for deciding to do something and for doing it.
The prime contractor would be responsible for doing parallel running, but the steering committee should be responsible for specifying that it be done.
So, in the case of Queensland Health, the fundamental questions are:
ReplyDelete1. Was acceptance testing conducted and signed off by the customer?
2. Did live production commence before acceptance testing was signed off?
3. Were any parallel runs implemented in parallel with ‘live production’ before the Health Department turned off the ‘old’ system and became fully dependent on the ‘new’ system?