By: Arno Maatman, Project Director 2SCALE
… not everything that counts can be counted !! and not everything that can be counted really counts … While I fully agree with this statement, 2SCALE cannot avoid measuring its results!!!
Last year, the Ministry of Foreign Affairs, contracted SEO Amsterdam (Dutch name: “SEO Economisch Onderzoek”) to carry out an in-depth evaluation of the 2SCALE program. The evaluation will be carried out in three phases, and the final report is expected in May 2018.
- A study of the partnerships, and results obtained so far, through interviews/ field visits.
- A comparison between the 2SCALE and other more or less similar private sector development programs, and
- An appraisal of the results of the external impact evaluations – carried out by Research Solutions Africa (RSA) and the American Institutes of Research (AIR).
The first two phases of the evaluation have been carried out and the results, so far, extremely positive. Some statements from the SEO reports:
- “In terms of outputs and short-term outcomes, 2SCALE has overall performed well relative to its targets, especially when taking into account that it might be too early for all outcomes to have materialized and that there have generally been too many targets.”
- “2SCALE has been successful at attracting private sector investments.” (although largely driven by some multinationals with large contributions)
- “2SCALE has been successful in strengthening female-led SMEs in a number of countries, especially in countries such as Mali and Ethiopia where the gender gap is large.”
- “Based on our case studies, we have identified many cases where 2SCALE seems likely to have contributed to improving food and nutrition security.”
- “Our case studies also provide evidence of cases where 2SCALE has contributed to ecologically sustainable food systems.”
- “The development additionality of 2SCALE projects generally appears strong, particularly with respect to the strengthening of the position of (women) suppliers and to the introduction of new products aimed at BoP consumers.”
- “2SCALE has implemented measures that help to ensure sustainability”.
- “There are already cases where the success of 2SCALE is being replicated.”
The impact evaluation, by RSA and AIR, is also just behind us. This evaluation, advised by 2SCALE’s project advisory committee, was designed according to random control trial (RCT) principles. A baseline survey was conducted in 2015 (September – October) for control and target groups of farm households; a 2nd wave of surveys was carried out in 2017 (same period, same households). Differences in trends between control and target groups can, in principle (i.e., when target and control groups are more or less similar, apart from being “beneficiary” of the 2SCALE program, or not) be attributed to 2SCALE.
Among the data collected was information on cropping systems, commercialization of target crop, membership and relations with farmer organization, household food consumption and incomes. This time, the results were much less convincing. What happened? First, a blanket methodology for partnerships that are so different from each other, i.e., that focus on different value chains and markets, implement completely different activities, and follow different dynamics, does not work. Moreover, the whole concept of control and target groups is difficult to implement in the 2SCALE context, without in-depth understanding of the partnership. Second, impact on household incomes and food consumption cannot be achieved in just two years. 2SCALE focuses on trust and connectedness, on co-innovation, on alignment of activities and investments; it takes a while before value chain relations and efficiencies translate into higher household incomes. Finally, while some variables at intermediate outcome level were used, in the statistical analyses, they were not making much sense; including for instance, the occurrence of weeds, and the number of different crops grown by the farmer… What can I say? If you use fertilizer, your weeds may grow more as well; and if you already have a diversified farming system, as the sorghum farmers in Kenya, 2SCALE has indeed not put any attention to further diversification.
For the time being I prefer – not just for opportunistic reasons! – to believe the evaluators of SEO, as they went into the fields, visited the partnerships, and held interviews with various different stakeholders. However, our experiences do raise some critical questions on how to monitor and evaluate performance of a complex program, based on a “rich” portfolio of partnerships, as 2SCALE. While I do not have an easy way out, I strongly believe in the following design principles for a functional M&E system:
- The focus should be, as much as possible, on the individual partnerships; performance monitoring systems should be designed for each partnership separately, and the indicators should be collected, not by 2SCALE, but as much as possible by the partners themselves.
- Only a few indicators, preferably at intermediate outcome level (i.e., results that can be more or less directly related to the activities of the 2SCALE program), that can be aggregated across the different partnerships, should be used as a proxy of the overall performance of the program.
- Evaluations need to be designed carefully, and take into account the specifics (target business, intervention areas, context) of each partnership. Random control trial methodologies may be used, but should focus on jointly defined intermediate outcomes, i.e., results that are still within the “sphere of influence” of the program. The relation between intermediate outcomes and impact indicators, should in principle be established by theory and complementary research; the donor (DGIS in our case) and the program should agree on this relationship on beforehand (e.g., when relations improve and trust develops, value chains become more efficient; in the long run this also improves opportunities for farmers, and other value chain actors, to increase margins).
Finally, seeing change in the field is nice, but not (always) good enough; the same holds for good feedback from our (private) partners. We need to see change in individual and collective capacities, at the grassroots (i.e., in our agribusiness clusters) and at the target value chain level (in fact the value chain system, as it should include all feeder value chains that support “our” core value chain).
A nice proposal is coming from the Partnerships Resources Centre (PrC), who supports 2SCALE through action-research. The PrC (based on insights from different authors/ sources) proposes the following distinction in (partnership-specific) capacities: deliberative, alignment, fitting, and transformative capacities. Deliberative capacity relates to the (governance) structures of a partnership, and how these structures invite and enable all relevant stakeholders to share concerns, ideas, and to influence decision-making; aligning capacity refers to the capacity of a partnership to influence the business models of the lead partners (lead firms, or local business champions) in favor of the less powerful actors in a value chain; fitting capacity is the capacity of a partnership to embed all (generated) inclusive development processes in existing business practices and private and public networks; transformative capacity, finally, is the capacity of a partnership to create new and disrupt existing institutions. Fitting and transformative capacities are, to me, like to opposite sides of the same coin … This makes perfect sense to me: inclusive development is not easy, it requires both a capacity to “break free”, as a capacity to identify and build on what is too strong to disrupt, or (preferably) what works and deserves to grow.