YAD and Fambul Tik Publish Evaluation Report On 10 Years Development Cooperation With the Government and People of The Federal Republic of Germany.

The headquarters of Youth in Action for Development (YAD) in Kenema is now overwhelmed  with intellectual euphoria as their leading German partner, Fambul Tik e.V., broke the news of the final evaluation report on their development cooperation over the years with the German government. YAD has collaborated with German NGOs and the Federal Ministry of Economic Cooperation and Development (BMZ) since the year 2012. 

An ex-post evaluation covering seven (7) different projects implemented in ten (10) years period, from 2012 to 2022, was carried out by movimentar GmbH, a professional development consulting firm based in Bremen, Germany. The German Federal Ministry of Economic Cooperation and Development (BMZ) and Fambul Tik e.V. had hired movimentar GmbH to conduct a comprehensive ex-post evaluation on their co-funded projects implemented by YAD in Sierra Leone, with three overarching objectives: 1) to assess how projects implemented by Fambul Tik and YAD with BMZ co-funding advanced the attainment of sustainable development in Sierra Leone; 2) conduct a firsthand assessment of ongoing projects to determine their prospect of success and replicability, and adapt evaluation reports as guidance for their successful implementation; 3) assess negative/positive outcomes/impacts of such projects and give tangible recommendations that can serve as guidance for ongoing and future projects implemented by Fambul Tik and YAD in Sierra Leone.

Though those projects were systematically evaluated formatively during implementations focusing mainly on achievements of tangible outputs and quick impacts, none of them had been evaluated in a summative term. Therefore, it was in the interest of the implementing partners, the government of Sierra Leone as well as project beneficiaries to conduct a thorough summative/ex-post evaluation of those projects to determine how sustainable, replicable, and scalable they have been and how much they had contributed to the attainment of sustainable development in post conflict Sierra Leone.

Evaluation Methodology

Movimentar conducted the evaluation along with local research assistants who aided their international research team throughout the process. This enabled them to translate/interpret questionnaires into local dialects such as Mende and Krio, which are widely spoken in the targeted communities. The team used a very sophisticated scientific algorithm based on the Organization for Economic Co-operation and Development (OECD) evaluation criteria, which focused on the six aspects of development assessment, namely: Relevance, Coherence, Effectiveness, Efficiency, Impact and Sustainability. Amongst the key informants interviewed were internal and external institutional stakeholders in government institutions, UN agencies, embassies, implementing partners, city and district councils, paramount chiefs, youth leaders, women`s leaders, students, teachers, principals and ordinary project beneficiaries. The team used both quantitative and qualitative assessment methods to provoke responses to key informant interviews (KII) using online questionnaires, focus group discussions and one-to-one interviews across the project communities.

The team formulated their questionnaires based on the OECD criteria, grading effects of the projects from A=5 (very good), B=4 (good), C=3 (regular/acceptable), D=2 (poor), E=1 (very poor).

Summary of Evaluation Findings

Relevance and design (A = 4.6 – ‘very good’): The projects have very high relevance considering the needs of their target groups. The main needs reported by beneficiaries were hunger, water-related diseases, and poor access to education. This finding is in line with the results of the sentiment analysis of the qualitative data obtained from the individual and group interviews. When asked about the relevance of the support considering the needs and priorities of the beneficiaries, 75% of the key words used by the respondents were classified as positive by the sentiment-analysis algorithm. The logical frameworks of the evaluated projects are clear and show a coherent design and a clear indication of the expected results to be achieved. The intervention logics explains the rationale for achieving the desired results, and the activities proposed are appropriate, practical, and consistent with the envisaged outputs and outcomes. There is evidence that the design and implementation of all projects have been informed by an analysis of the needs and priorities of vulnerable groups such as women, children, youths, adolescents, and elderly. The project proposals, field observations, and beneficiary responses show that the implementation of the projects followed a human rights-based approach and the principle of ‘leaving no-one behind’. Although none of the projects cites a specific gender action plan, there is evidence that gender has been integrated into the engagement model of YAD and Fambul Tik (e.g., the strategy of equalizing the number of male and female active agents in the projects, taking the example of the CHMs). Developing a gender-mainstreaming plan that is culturally sensitive and adapted to the context could help improve the contribution to gender equality, with a focus on promoting more women in leadership positions, for example. The issue of eradicating female genital mutilation could also be explored in future projects since it is still widely present in Sierra Leone’s rural areas, which remains one of the 28 African countries that still partake in practice.

Coherence (B =4.4 – ‘good’): The extent of synergies between the projects and government actions, institutional strategies, and the policy scenario was very high. The projects addressed several priority areas of Sierra Leone’s national development plans. The project activities made relevant contributions to national policies and strategies. The inclusion of local government authorities in the project implementation, such as in the district council of Kenema, has resulted in concrete contributions to the implementation of national and institutional policies: the development plans Poverty Reduction Strategy Papers (PRSPs) and the Agenda for Prosperity (A4P, 2013-2018). There is evidence of synergy with the ministries of health and of education for the COD and the MCPS and NCSS projects, respectively. Results demonstrate good alignment of projects with SDGs 1, 2, 4, and 61. The planned impact on child literacy rates, reduction of water-related diseases, hunger reduction, and vocational training of youths and adolescents directly fulfil the objectives of the mentioned SDGs. YAD has representatives assigned to WASH coordination meetings and the Agriculture and Food Security coordination meetings. The same goes for the Technical and Vocational (TECVOC) Council, with someone appointed to the general District Development Coordination Committee (DDCC).

Effectiveness (A= 4.6 – ‘very good’): Planned output and outcome targets were achieved to a great extent. The majority of outcome and output indicators were achieved. However, the Covid-19 pandemic has negatively affected some of the project activities and delayed some indicators. The results suggest that the projects made good use of the resources made available by the donor and adapted well to the consequences caused not only by Covid-19 but also by Ebola, the rainy seasons, and the inflationary crisis. Participation of key stakeholders during planning and implementation phases was high. Collaboration with implementing organizations, relevant ministries, and local government authorities took place. Beneficiaries participated in the project activities and took over important responsibilities. There is strong evidence that the projects have achieved their objectives, considering the evidence from primary data collection. For example, in the online survey the majority of participants (82%, 10 participants) positively rated the extent to which the objectives had been achieved. Responses to the KIIs/FGDs with beneficiaries made it clear that the support received was helpful in their lives.

Efficiency (B = 3.7 – ‘good’): The overall efficiency of the projects was good. The activities can be considered cost-efficient. Several external factors such as the Covid-19 pandemic and the inflation crisis made financial management difficult. However, activities involving fundraising and volunteering effectively supported self-sustainability of projects and activities. These factors included the Covid-19 outbreak, an inflationary crisis, and social disorder disrupting economic activities. Management quality and efficiency of the projects (including work planning, procurement, financial resource management, budget allocation, and timely outputs) was good. High transparency and good communication as reported by stakeholders have contributed to this. The project-management system (including technical expertise as well as monitoring, planning, and reporting systems) was moderately functional, sufficient, and goal-oriented. Key stakeholders reported positively on the project-management system. Especially the feedback system implemented by YAD and the close work with Fambul Tik can be highlighted here. Transparency and accountability as well as communication have been satisfactory.

Impact (B = 4.4 – ‘good’): The projects’ impacts were very high. They contributed to the development of Sierra Leone – to a reduction in waterborne diseases, improved education, and the economy of the villages – and were aligned with national strategies. The COD project improved the health and hygiene situation of various villages and thereby reduced water-related diseases and contamination. By separating the toiles by gender, physical condition, and age, vulnerable groups were given more comfort, safety, and dignity. The impacts from the different projects reinforced each other. For example, by reducing diseases, more children were able to visit the schools. According to key informants, the schools help to literate the communities and thereby contribute to economic growth of the families. This is supported by the educational training given in the MYRC for youths, who now have a better reputation in their communities. One of the country’s major problems, hunger, was tackled by the NIHP project, which enabled beneficiaries to improve their farming techniques and thus increase food production. While there were several indirect positive impacts, no negative ones were reported. Some of the unplanned impacts were, for example, increased recognition and growth of YAD as well as a sense of unity created within the villages. Based on the positive impacts that the projects had, several key stakeholders stated that the projects should be implemented in other parts of the country, too.

Sustainability (A = 5.0 – ‘very good’): The sustainability of the projects is very high due to a high level of ownership, a long-term orientation of the activities, and a very high contribution to the institutional and management capacity of local partners. The majority of key stakeholders were convinced that the projects would continue to benefit the beneficiaries after the end of the support and even will benefit future generations. The fact that at the time of the assessment all established project facilities and activities were still active supports this. Mechanisms were established to sustain all project components such as community findings and committees responsible for monitoring and/or maintenance and repairs. The high level of ownership by the beneficiaries leads to a high sustainability of the projects. For example, the constructed facilities such as the sanitation facilities, the MYRC, and the market are regarded as the community’s responsibility and have benefits for all. Thus, the target groups are willing to put efforts into their maintenance. This level of ownership was achieved by multiple training sessions and good involvement of the beneficiaries in planning and implementation. Institutional and management capacity of local partners were very high despite the short duration of most projects. In particular, capacity-building activities for various leaders and counsellors play important roles for the enforcement of laws at community level.

Conclusion

The seven projects were highly relevant to the national context and the needs of the target groups. They have contributed to the sustainable development of Sierra Leone in line with government goals. In their intervention logics (including planned objectives and outcomes), the projects prioritized vulnerable groups, that is, women, children, youths, elderly people, and persons with disabilities. This is exemplified by the engagement model that YAD and Fambul Tik implemented for the equal participation of women and men in activities, the inclusion of youth organizations in support of YAD’s actions in Kenema City, and the adaptation of project structures and tools for the use of older people and persons with disabilities. The activities carried out in MYRC phases I and II, for example, were relevant in providing the youth from Kenema City and surrounding villages a safe place for their meetings, where they also received workshops on conflict resolution issues and career counselling. Key informants confirm that youths are satisfied with the services provided, which is reinforced by the survey results, according to which 91% of the respondents rated the project relevance ‘very high’ (73%) and ‘high’ (18%). When asked about their opinions on the work done by Fambul Tik and YAD, 89% of the keywords used by the respondents in key-informant interviews were positive (in a sentiment analysis; see more about this in section 8.1). Regarding the relevance of the support considering the beneficiaries’ needs and priorities, 75% of the keywords used were positive.

The relevance of the work is high considering the needs of the beneficiaries of the other projects, such as poverty eradication (KCM), hunger eradication (NIHP), hygiene improvement (COD), and improving education for children (MCPS and NCSS). These projects all address the requests made by the beneficiaries themselves during planning, as YAD and Fambul Tik applied a participatory approach for the design of all implemented projects. Despite that, there seems to be room for improvement in terms of data collection and management. The evaluation team did not receive any baseline study or survey reports among the documentary evidence.

The results of the KIIs and the documentary evidence suggest that the objectives and outcomes were well achieved considering the challenges of Covid-19. The results indicate, however, that the support received during Covid-19 in terms of food-for-work is requested to be continued for the implementation of the still ongoing NIHP. The only limitations mentioned by the respondents are the halting of project activities in the years 2019 and 2020 in view of the restrictions related to Covid-19 as well as those during the Ebola crisis from 2015. However, according to online survey findings regarding the question: “To what extent does the design respond to the needs related to the Covid-19 pandemic?”, only 45% responded positively (9% ‘very good’ and 36% ‘good’), which was expected considering that the projects had to adapt to the Covid-19 context.

The projects’ impact in the areas of education, health, and economy was high. Building primary and secondary schools in the villages of Matakan and Gandorhun contributed to the education of children and their safety, reducing the need to travel far to study. Additional analysis could be done in assessing impacts in terms of a reduction of child labour, considering the improved access to education. Regarding hygiene, the primary and documentary evidence suggest that COD has contributed to reducing the cases of waterborne diseases, thereby increasing the quality of life for vulnerable groups and their ability to invest their energy and time in activities that contribute to their own development and that of their families and communities. The KCM project has promoted income generation for female traders in the Konjo community, as women are now able to sell the produce harvested on the fields in a more organised and regular manner. Finally, the impact on hunger eradication was demonstrated by beneficiaries of the NIHP project, whose application of the farming techniques learnt in the project resulted in higher outcomes of their harvests. Assessing the impacts at household level would require an additional household survey to explore the benefits in terms of income, employment, education, and nutrition, for example. Further analysis on standard indicators of well-being at household level, such as the Life Satisfaction Index, Food Consumption Score, Coping Strategy Index, and Household Dietary Diversity Score, could help assess impacts in a more detailed way. Despite that, there is no indication of any major indirect and unexpected negative impacts, and the positive results reported in the online survey and KIIs suggest that the projects addressed the main needs highlighted by the beneficiaries.

The collected evidence suggests a positive evaluation of all aspects related to the overall management, quality of results, transparency, and communication. Data about the timely provision of inputs, however, suggest the need for improvement. Similar to the documentary evidence and key-informant interview results, about 27% of the survey respondents rated this as ‘regular’. This may be due to under-budgeting and the required time for project changes, considering requests for changes in the budget caused mainly by the inflationary crisis in Sierra Leone, which seems to have contributed to the delay of some activities.

The projects show a very good alignment with international goals, particularly the SDGs 1, 2, 4, and 6. The results indicate that the areas addressed in the education, health/hygiene, and economic sectors were highly coherent with existing national policies and strategies. The contribution to the sustainable development of Sierra Leone was confirmed by the key informants.

There is evidence of partnerships between project implementers with Kenema City government authorities, government ministries, and youth organisations. Similarly, the results suggest an active involvement and participation of beneficiaries in the planning and implementation of each project. Although none of the projects cite a specific gender action plan, there is evidence that gender has been integrated into the engagement model (e.g., strategy of equalizing the number of male and female active agents in the projects, taking the example of the CHMs). Developing a gender mainstreaming plan that is culturally sensitive and adapted to the context could help improve the contribution to gender equality, with a focus on promoting more women in leadership positions, for example. Eradicating f emale genital mutilation could be explored in future projects, since it is still common in the rural areas of Sierra Leone, which remains one of the 28 African countries that still partake in the practice.

The findings suggest the need to improve the way project indicators are monitored and presented in YAD internal reports. Household surveys could help projects to become more evidence-oriented, while contributing to inform policies and other actions by the government and development partners. Training on robust monitoring tools and systems could be included more frequently in the training of YAD members. Commercial cloud-based management-information systems that help improve project management, such as Teamwork Projects, Trello, Asana, or Basecamp, could help improve both the implementation and monitoring of tasks on a routine basis.

The sustainability aspect was the most highly rated by the key informants. Again, the participatory approach that YAD and Fambul Tik implemented contributed to creating a sense of ownership and responsibility among the beneficiary communities. This, together with the partnership with local and national authorities, helps to maintain the delivered structures in the long term. Strategies created by stakeholders, such as the implementation of local laws punishing those who degrade project structures, monthly payment of a fee to the local village bank account in cases of repairs, and delegation of monitoring responsibility to village residents in reporting to YAD members are examples of best practice supporting sustainability.

Lessons learnt and best practices

Lesson learnt 1: The introduction of the beneficiaries’ participatory approach in the implementation of the projects is key to the successful sustainability of the project’s outcomes.

Best practice 1.1: The open and regular local meetings between the organizations implementer and all the local authorities and villagers provides a democratic space where everyone has the right to speak and share their needs, which are agreed upon as a group.

This allows for finding effective solutions to their own problems, based on their own experience.

Best practice 1.2: The inclusion of the beneficiaries in the achievement of the project’s objectives, such as the use of their labor force in the construction of the schools, youth

centers, markets, and toilets, for example, was not only an economic way of financing the projects but also created a feeling of unity and ownership in the communities, as mentioned in most of the answers in the assessment.

Best practice 1.3: The establishment of representatives who monitor the project facilities, such as the Konjo Youth Development Association (KYDA) for the Community Market and Social Mobilization Centre and the community health monitors (CHM) in the villages covered by the COD project promoted collaboration with local authorities to ensure the sustainability of the projects. KYDA leaders and the CHM received theoretical and practical training so that they can continue to manage the project after implementation. This practice can be further expanded in future projects in different fields.

Lesson learnt 2: The introduction of a charging platform for cellphones in KCM sped up and eased the communication between beneficiaries and YAD staff and improved business in the market.

Best practice 2.1: The source of energy through solar panels in all structures implemented by the projects contributes to their sustainability.

Lesson learnt 3: Feedback obtained through the KIIs/FGDs suggests a strong and frequent communication and monitoring network implemented by YAD members with the ambassadors (representatives) of each village. The selection of representatives from each village to work as the main connecting bridge between their residents and YAD members helped to coordinate the tasks and avoid misunderstandings in the implementation of activities. Despite that, the results emphasized the importance of increasing capacity building in ongoing and future projects.

Best practice 3.1: The selection of representatives of the beneficiary groups for the implementation of each project is an example of a best practice, which catalyzes the work of the implementing organization with the decentralization of responsibilities and assurance of community ownership.

To read the actual report including annexes, please follow this link: YAD and Fambul Tik Publish Evaluation Report On 10 Years Development Cooperation With the Government and People of The Federal Republic of Germany.