Meta-evaluation of Project and Programme Evaluations in 2014-2015
Metaevaluation of Programme and Project Evaluations in 2014-2015 (Opens New Window) (PDF, 224 pages; 1,47 MB)
The meta-evaluation assessed the quality of evaluation and appraisal reports and their corresponding ToR and ITT. Assessments were based on the requirements expanded upon in various MFA manuals. A second phase analysed what the reports revealed about Finnish Development cooperation, using an innovative mixed deductive-inductive approach.
The quality of evaluation reports and ToR was not rated very highly except for certain, and non-critical, characteristics. Both presented lack of evidence, insufficient answers to the evaluations questions, absence of analysis on topics including aid effectiveness, cross-cutting issues and Finland’s Human Rights Based Approach policy. ToR were not sufficiently specific in their direction and did not include instructions on the need to report on a number of policy issues. Appraisal reports were rated very low, particularly because they were treated as ex-ante evaluations in a context where the proposed intervention was not sufficiently defined to undertake that kind of analysis.
The analysis of Finnish development cooperation was based on a smaller sample of evaluation reports that had met a minimum level of quality. Appraisals and inductive analysis insights were added to a rigorous and rated deductive analysis in order to conclude.
Finnish development cooperation was found to be very relevant in that it is aligned to both Finnish and beneficiary organisation policies and strategies. It also responds generally to the needs of the targeted beneficiaries. Reports do not provide much information on the extent to which initiatives are relevant to specific Finnish development cooperation policies, however. It was only moderately effective in meeting its higher-level objectives. Efficiency was awarded a low score, and many of the reports did not report on it. The sustainability rating was rather low because the reports did not show why they believed that interventions would be sustainable. The very low rating given to impact reflects the fact that very few reports were able to indicate what the impact would be. An across-the-board absence of information systems to gather required data on impact accounts for a large part of the low score.
Recommendations deal with strategic level and operational level conclusions, as well as suggested avenues for the further development of capabilities of MFA and the abilities of MFA staff.