World BankIndependent Evaluation Group.pptVIP

World BankIndependent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger Session outline Defining and measuring evaluation utilization Reasons why many evaluations are under-utilized Examples of evaluation utilization: the World Bank “Influential Evaluations” study Ways to strengthen utilization If time permits .. Further discussion on “presenting the message” 1. Defining and measuring evaluation utilization Defining evaluation outcomes and impacts Use How evaluation findings are utilized by policymakers, managers and others Influence How the evaluation influenced decisions and actions Consequences How the process of conducting the evaluation, the findings and the recommendations affected the agencies involved and the target populations.. Consequences can be: Positive or negative Expected or unanticipated Measuring evaluation outcomes and impacts Changes in individual: Knowledge Attitudes Behavior Changes in organizational behavior Changes in program design or implementation Changes in policies and planning Decisions on project continuation, expansion and funding. Measurement issues Time horizon Intensity Reporting bias Many agencies do not acknowledge they have been influenced Attribution: How do we know the observed changes were due to the evaluation and not to other unrelated factors? Attribution analysis How do we know if observed changes were due to the evaluation and not to other influences? Stakeholder interviews Surveys [pretest/posttest or posttest only] Analysis of planning and policy documents for evidence of evaluation influence Analysis of mass media Key informants Examples of attribution methodology used in “Influential Evaluations” See: “Influential evaluations: Detailed case studies” pp. 69-72 Attribution analysis framework  A. Comparison of 2 user surveys + Stakeholder opinion survey  Comparison of sample surveys of service users (in 1993 and 1999) found reported improvement of services [potential im