Meta-evaluation of project and programme evaluations in 2015-2017
The Ministry of Foreign Affairs of Finland (MFA) regularly commissions meta-evaluations. This evaluation was commissioned to Particip GmbH. In this assignment, 51 decentralised evaluation reports of bi-, multi-, and multi-bilateral interventions conducted between September 2015 and August 2017 and 45 ToRs were subject to methodological quality and summative content assessment. The two-stage multi-method analysis followed a participatory approach. It built on comprehensive standardised assessment tools and thematic coding. Summary statistics and qualitative content analysis were applied.
We find that overall quality is satisfactory for 60% of the assessed ToR. Nevertheless, numerous weaknesses were identified which reveal capacity gaps within MFA. Given that on average higher ToR quality is associated with higher report quality, we highly recommend to improve the evaluation manual, to enhance knowledge of evaluation methodologies and practices and to consider improving existing structures.
Regarding the quality of the reports, we observe that findings are often based on weak methodologies but appear to be somewhat reliable. Two thirds of the reports feature some, one third substantial quality flaws. Thus, we recommend to enhance quality assurance and to ensure methodological expertise when recruiting evaluators.
According to the evaluation reports, the overall quality is assessed as of moderate quality or better for 70% of the interventions. Relevance is considered as strength and sustainability as greatest challenge.
However, given the lack of information on the whole population, we cannot assess the representativeness of this sample of reports for this part of Finnish development cooperation. We therefore recommend to run an inventory of interventions to enable stratified sampling in future.
Meta-evaluation report 2018 (in English, PDF file, 196 pages, 2 MB)