Joint Programme Evaluation final evaluation report august 2016 DanChurchAid (dca) Jhamsikhel, Lalitpur Nepal Dr. Prakash Bhattarai and Dr. Dhruba Gautam

1.1 Objectives of the Evaluation

In a broader picture, the evaluation was focused on the assessment of the effectiveness and outcome of DCA’s thematic programmes in South Asia, particularly in Nepal and Bangladesh, with a specific focus on their contribution to the reduction of inequalities and poverty for women and men. Specifically, this evaluation will assess the level of progress being made towards the achievement of project impact, outcomes, and outputs and identify areas to continue, improve, and design new similar programme in the years to come. The results of the evaluation are also intended to help to design country specific programmes/projects in Nepal and Bangladesh and also identify the need for any further technical assistance to support the strengthening of resilience in above countries. Furthermore, the evaluation will also help to understand the relative merits of different implementation options and thematic approaches.

1.3 Methods and Methodology of Evaluation

In order to carry out this assignment, the key evaluation Framework of Analysis developed by the consultants is mainly focused on meeting the evaluation objectives. In a broader picture, this evaluation has assessed the OECD/DAC criteria i.e. Relevance, Effectiveness, Efficiency, Impact, and Sustainability aspects of the project. Likewise, the evaluators have applied the results-based management approach together with the “most significant change” method¹ in assessing the programme’s overall achievements at the levels of outcome and impact.

1.3.1 Data Collection

In order to collect relevant data and information, this evaluation combined both strategic desk study and qualitative field research methods. The evaluation consultants obtained the relevant data through the desk review of relevant project documents and reports, situational observations during field visits, in-depth interviews with local, national, and regional level partners, stakeholders, and DCA staffs in Nepal and Bangladesh.
At the local level, consultants convened in-depth interviews with project beneficiaries, local staffs, local government officials, political and civil society leaders and local project partners. At the national level of both countries, consultants had interviewed with selected project partners, relevant project staffs, and other individuals who were familiar with DCA’s programme interventions. Selection of geographical locations and selection of respondents in both countries for FGDs and in-depth interview was based on maintaining the caste, gender, ethnic, and geographical diversity. This was also intended to receive varied responses to capture the real picture of the project. A total of 13 FGDs and 27 in-depth interviews were commissioned and captures the outcome and impact level information. Likewise, 3 consultation meetings with partners, 3 meetings with current and former DCA staffs, 9 meetings with stakeholders were conducted in Kathmandu. Consequently, 11 Skype interviews were conducted with around 25 individuals from Bangladesh which includes DCA staffs, NGO partners, and experts in various thematic issues. Along with FGDs, in-depth interviews, consultation meetings, and stakeholders’ meetings, evaluators also observed the small scale mitigation works and livelihood schemes in the field. Participant observation tool was also used to gather people’s inner feelings and degree of satisfaction on the programme. This was also intended to receive varied responses to capture the real picture of the project.
To the adequate coverage for an impact evaluation of the projects, at least half of the total project location was covered during project evaluation period in Nepal. This could not be possible in the case of Bangladesh due to restriction in the consultants’ entry into Bangladesh.
For the primary data collection, semi-structured questionnaire were drafted (see the annexes below) by the evaluation team. The questionnaires are in line with evaluation objectives and also meeting the evaluation criteria specified in the ToR.
The sample of respondents was drawn on the basis of gender, caste, and ethnicity, cultural, linguistic and geographical diversity within the selected project locations. For this purpose, along with evaluation team’s own assessment, the evaluation team took the help of DCA staffs in identifying adequate respondents. The sampling method selected was purposive, as this evaluation was carried out for a definite purpose, and interviewing the right people was paramount to receiving the ideal response.

1.3.2 Data Analysis and Report Preparation

Information gathered through this evaluation study has been analysed through a comparative content analysis as well as interpretative approach. This information has been utilised to draw appropriate conclusions, strategies and evidence-based recommendations based on Framework of Analysis and meeting the evaluation criteria specified by DCA on the ToR for evaluation consultants.
Once data collection was over, data was triangulated and analysed using largely qualitative data analysis techniques. The qualitative data was entered into data entry formats, coded for common themes, triangulated and finally analysed to summarise the findings. These findings were then used in formulating the conclusion and recommendations. In order to ensure that no important information is missed from the field, Daily Interpretive Analysis of each interview (either fully or partially) was prepared.
Once there was a broad agreement between the evaluation consultants on the findings of the evaluation, the final report was prepared.

1.3.3. Limitations

This evaluation also has a number of limitations. Some key limitations are presented below;

• 
  The evaluation coincided with the agricultural season. Due to this fact many people were away in the field and could not be interviewed for the longer period of time.
  

• 
  Rains and bad roads increased the time needed to reach communities and reduced the time that could actually be spent in the communities.
  

• 
  It was not possible to visit Bangladesh due to unjust and forceful deportation of evaluators by Immigration Authority at Dhaka airport. Hence the evaluators only relied on the information received from Skype interview.
  

• 
  Due to limited time as well as lack of accessibility and availability during the information gathering process, the voice of regional and India based NGO partners, and some important staffs from the DCA headquarter could not include in this evaluation report. However, one senior level staff from DCA headquarter were interviewed in the later phase of this evaluation.