This appeared last week:
Evaluation: what’s the use?
17 Jan 2018
Description
Concerns about non-use of evaluations have plagued the profession since it emerged in the 1960s to guide government decision-making about social policies and programs. While there is a substantial body of empirical and theoretical literature about evaluation use, this literature does not identify the factors that are considered most important to facilitating evaluation use or the pathways to evaluation use. Additionally, much of the literature is from North America and Europe and there has been no large-scale study of evaluation use in Australia. This study aimed to identify Australasian Evaluation Society (AES) members’ perceptions of the levels of use of evaluation and the factors associated with use, as well as how evaluators overcome barriers to use. It used a questionnaire of AES members and in-depth interviews with evaluators. The AES members who responded perceive both demand-side factors, particularly leadership commitment and individual receptiveness to evaluation, and supply-side factors, particularly involvement of stakeholders in identifying the evaluation purpose and effective communication of findings, as important to evaluation use. Evaluators employ a range of utilisation-focused strategies and have some success in negotiating the barriers they encounter to use. Evaluators’ experience reflects that the factors that are most important to use and the pathways to use differ by context, suggesting that existing theoretical models of evaluation use could be strengthened through recognition of context-based pathways.
Publication Details
Identifiers: DOI: 10.1177/1035719X1701700404
Volume: 17 Issue: 4
Pagination: 25–38
Copyright: Copyright © 2017, © SAGE Publications
Language: English
License Type: All Rights Reserved
Here is the link to the paper (Free)
Essentially what the paper points out is that the key barriers to evaluation are (what a surprise) politics, resourcing and leadership. Despite the barriers I would have to suggest for the vast majority of the time evaluation is important and necessary if we are not to just blunder uselessly around!
I rather suspect the author has it right on the bassiers! How to fix things? Another story I fear.
David.
I have quite strong views on evaluation in health care having witnessed very perverse outcomes. I think evaluating outcomes is great as long as people are not aware of the targets and ideally evaluation is done randomly.
ReplyDeleteI remember a hospital being given money to reduce waiting times and the effectiveness was going to be evaluated. They simply employed people to send a letter to everyone on the waiting list and if they failed to respond in 7 days they were taken off the list. I am sure the evaluation revealed great bang for the buck!
MyEHR was being evaluated on number of enrollments so they employed scalpers to sell tickets in hospitals and medicare offices to make the numbers look good and that has really made a difference!
Evaluating outcomes if great as long as the targets are unknown, and the people being evaluated are unaware that its happening. The point of eHealth is to collect good data so that everyone knows whats happening based on good information. That would enable real evaluation of outcomes to occur. I don't see any good data around, apart from lab data that was setup at the turn of the last century. This was done without government help, when pathologists were owned by doctors. Since then we have seen no return on the $2B eHealth investment despite help from HealthConnect, NEHTA 1 & 2 and now ADHA.
Perhaps government does not know what its doing? That is my evaluation from the sideline. What is working apart from what was put in place before they stepped in to help?
Andrew although not my area of expertise I do find that patterns pertaining to current monitoring and evaluation frameworks seem founded (purposely or inadvertently ) on predefined outcomes and based on assumptions of causal, linear, and attributable connections between sets of activities ( and in the case of eHealth –political or commercial drivers) and changes at its outcome and impact levels.
ReplyDeleteHealth and as a result health IT, these assumptions tend not to reflect the reality of the complex systems inherent in healthcare, such as markets or communities.. The AHHA in their recent blueprint calls out for an organisation at arms length of its founders and political influence, through that perhaps an opportunity to discuss possible ways to design monitoring systems, and provides guidance on project design that is informed by scientific research in the field of complex health systems.
I appreciate a need for projects to deliver tangible outputs, (shines widgets) but it seems evident this is being done at the cost of learning. We no longer seem to be able or willing to embrace dialogue around complex domains and embrace alternatives where it means having to say, well we learned, that option based on emergent evidence no longer has merit and let’s move forward.
We do need an ability to say "We were wrong" and "Lets try a different approach" but it seems that the illusion of success is enough today. Can't end well and applies in many areas. Success often starts with failure.
ReplyDeleteTotally agree, we are led to believe it is all about Agile and innovation, we’ll fail fast fail safe, explore, engineer review, all sound like ‘we discovered a few flaws in that approach’ are permissible and valued. Not sure why we keep hanging on hoping for something wonderful to happen
ReplyDelete"Not sure why we keep hanging on hoping for ...."
ReplyDeleteBecause you are stupid and because you don't know what else to do. This is a circular argument because you are stupid and .... .
4:46 PM you don’t run the ADHA Capability section do you? You sound just like her
ReplyDelete@4:46 not sure that adds any value. Why continue holding out for something wonderful to happen is a fair question. IMHO the longer thing go on the more locked in the nation becomes to a IT solution, one that constrains rather than extends. The world has changed from when the MyHR was conceived. I guess the challenge is how you actually turn it off and what happens to all those blobs of random information
ReplyDelete