Navigating New Horizons
AGRA and Mathematica’s framework to document systems change
The case study below looks at the Alliance for a Green Revolution in Africa (AGRA)’s collaboration with Mathematica to develop the 10 Dimensions of Systems Change framework, an innovative framework aimed to support AGRA to capture its contributions to systems change in African food systems.
This case study presents the rationale behind the development of the framework, its key components, and the current plans for operationalization.
Illustrations: Ivana Čobejová
Although still in its infancy, this framework has several distinctive features:
1
It was developed using a collaborative approach between personnel from across AGRA, donors and implementing partners.
2
It builds on and integrates measurement methods and tools already used in the organization.
3
It outlines a theory of how transformative change happens in food systems, that differentiates between “catalytic changes,” “systems changes” and “socioeconomic impact.”
The Centre for Public Impact has helped produce this study and we thank them for their insights and excellent collaboration. We also extend a great thanks to the staff and partners of AGRA and Mathematica for sharing their journey and learning with us. We hope you find this case study interesting and useful, and we hope that you’ll stay tuned for the next pieces in this series.
Introduction
In tackling today's complex global challenges, the global development sector must embed systemic thinking and learning mindsets into standard practices. Organizations like the Alliance for a Green Revolution in Africa (AGRA) are experimenting with innovative approaches, such as the 10 Dimensions of Systems Change framework, to steer their programme assessment, evaluation, and implementation with a systemic focus. While early in its adoption and implementation, this framework signals a shift toward rethinking conventional monitoring, learning, and evaluation methods in complex systems, in this case across Africa's diverse agricultural contexts.
Figure 2: An example of AGRA’s previous system scoring grid.
The 10 Dimensions of Systems Change framework builds on this measurement strategy to create a more holistic framework to document systems change.
The Genesis of the Framework
In spring 2021, Mathematica partnered with AGRA as an external evaluator for the PIATA programme. The evaluation was focused on assessing AGRA's progress in achieving their foundational impact objective: enabling a sustainable environment for African smallholder farmers within an agricultural revolution. The evaluation observed an exceptional performance in various aspects of programme delivery, particularly regarding contributions to policy reforms, support for state capabilities, and robust public-private partnerships. For instance, the evaluation revealed that AGRA's efforts contributed to 72 policy reforms across 11 countries, with 54% (39 reforms) currently in implementation. Moreover, significant strides were observed in enhancing state capabilities across multiple countries. The evaluation also identified areas for growth and offered recommendations on AGRA's measurement and evaluation practices. These recommendations included leveraging data and partnerships to address stakeholders' critical constraints, transforming AGRA into a more learning-focused organization, streamlining grantee reporting while conducting more rigorous farmer surveys, and continual learning-focused stakeholder engagement to help AGRA better identify strengths, areas for improvement, and develop improved action plans. Eager to act on the recommendations from the evaluation in February 2022, AGRA extended to Mathematica an invitation to support them in putting their recommendations into action. This resulted in the 10 Dimensions of Systems Change framework, built through a collaborative process that engaged AGRA personnel at different levels in the organization, donors and implementing partners.
Inside the tool: The 10 Dimensions of Systems Change Framework:
The 10 Dimensions of System Change framework organizes measures to gauge ‘real world’ changes into 10 dimensions of change, alongside a continuum that goes from “interventions”, to “catalytic changes”, to “systems change”, to “socioeconomic impact” (see Figure 3 below).
Figure 3: AGRA's 10 Dimensions of Systems Change framework.
The listed dimensions were defined participatorily and were informed by a literature review that spanned across the agriculture and systems thinking fields, and included resources such as the Water of Systems Change framework, the COM-B Model for Behavior Change, the Comprehensive African Agricultural Development Programme (CAADP), the African Seed Access Index (TASAI), the FAO Sustainable Food Systems Framework, and the Government Readiness for Agricultural and Transformation Framework. Furthermore, while the dimensions reflect the literature mentioned, they are also tied to AGRA’s corporate and business line ToCs and AGRA’s approach to systems change.
By defining a manageable set of systemic changes that AGRA is seeking to achieve, the 10 Dimensions of Systems Change framework aims to support organization to document systems change in the countries and systems where AGRA works, while enabling increased learning and accountability for its management and implementation teams.
How is the framework structured?
Stage 1: Interventions:
Interventions refer to targeted investments or collaborations within the broader system, made at different levels by AGRA. Examples of interventions include hiring consultants to spearhead policy reforms, convening public and private stakeholders to discuss policy and national strategy development, and targeted investments in public seed system infrastructure.
Stage 2: Catalytic Changes:
These are micro-noticeable changes that are essential to catalyze broader transformation in food and agricultural systems and sub-systems. These changes, which are the leading indicators of systems change, do not have to occur in any order, however changes in one or more of these dimensions must occur in order for AGRA to detect changes in the following levels (systems, SEI). AGRA expects to influence these in 1- 2 years. They comprise the following changes.
- 1. New and strengthened linkages and coordination across system actors
- 2. Increased generation and use of evidence and technology
- 3. Enhanced capacity and incentives to invest and engage other actors
- 4. Private sector investment and use of finance
- 5. Public sector leadership, investment and accountability
- 6. Engagement and behavior change of farmers, youth and women
Stage 3: Systems Change:
These dimensions are meant to capture significant systems changes that can be observed over time. These systems changes and socioeconomic impact are more medium-term changes and should occur by years 3-5.
- 7. Healthy enabling environments and supports for agricultural transformation and food systems
- 8. Inclusive market performance, including small & medium sized enterprise (SME) profitability
- 9. Resilient supply and demand of public/private products and services
Stage 4: Socio Economic Impact:
Derived from the PIATA headline objective of fostering a sustainable and supportive environment for the prosperity of smallholder farmers in Africa as part of the smallholder farming revolution, this stage focuses on AGRA’s ultimate goals, i.e.:
- 10. Economic and social benefits for farmers, youth, and women, including improved yields, income, and resilience to climate change
Changes in this dimension are tracked, with indicators and proxies measuring work opportunities, food security, productivity, nutrition, and resilience.
How will it work in practice?
AGRA anticipates that this framework will enable improved learning and accountability for the team. Currently, AGRA and Mathematica plan to monitor the 10 dimensions to detect changes in system functioning and outputs biannually. If favorable changes are detected, they will conduct a contribution analysis to determine AGRA's contributions to bring about those changes, by tracing back to their interventions. The team is taking a humble stance by noting that “contribution analysis usually reveals that multiple organizations play a role in improving systems—and each is just one player in the larger cast of characters.” It is through the learnings of what is changing in the broader system that AGRA seeks to adapt their interventions efforts to contribute to systems change.
Looking ahead, AGRA seeks to leverage the use of heatmaps to illustrate change visually While AGRA seeks to leverage quantitative and qualitative data, the heat maps will uniquely “provide an immediate visual summary of information allowing users to quickly grasp the most important or relevant data points.” They will be developed by combining AGRA‘s scorecard system with the newly developed 10 Dimensions of Systems Change framework. AGRA and
Figure 4: A mock-up example of how AGRA predicts future assessments will look using heat maps.
1 Randall Blair, “email message to,” Naja Nelson, November 15 2023 2 Techtarget (accessed 14 November 2023). https://www.techtarget.com/searchbusinessanalytics/definition/heat-map (accessed 14 November 2023). Insights and value of the 10 dimensions of systems change framework
✔️ Shifting beyond top-down solutions
One of the most striking elements of AGRA's 10 Dimensions of Systems Change framework was its emphasis on collaboration in the design process. By involving diverse stakeholders, including implementing partners and funders, the framework development team could harness insights across numerous perspectives to inform their approach. While the intended output of this collaboration was the framework, the process also contributed to build foundations for a more collaborative and inclusive environment. This is in line with best practice in empowering grantees in shaping goals and measurement methods.
Such collaborative processes have the potential to spark greater transparency and trust between practitioners and funders. Despite these two groups working in the same ecosystem, they typically do not regularly interact with one another, limiting trust-building opportunities, and ultimately hindering innovation and creativity. When trust between funder and partner is low funders are less flexible with project plans and resources and partners set goals that are easily achievable. Co-creation processes have the potential to increase mutual trust and lead to more ambitious and adaptive initiatives.
✔️ Looking Ahead: Embracing learning as the journey continues
AGRA and Mathematica are in the early stages of testing the framework's validity through practical application. This phase will allow them to adjust their methods through direct testing with programme staff and implementing partners. In November 2023, they began gathering baseline data on the 10 dimensions in priority countries. The indicators differ per country due to varying scopes of interventions and budgets. In the words of Simiyu, the AGRA team “is excited but also intrigued to test their assumptions within the framework and unpack whether this process can support the programmatic team to better identify if their interventions are instigating the catalytic and system changes” needed to improve the conditions for farmers and their communities.
Navigating Complexity: Your invitation to take a collaborative approach
Embracing systemic and adaptive thinking to navigate uncertainty and complexity
The 10 Dimensions of Systems Change Framework is in its infancy; it is too soon to comment on its value in supporting AGRA or other like-minded organizations in measuring the systemic impact of their work. Nevertheless, the collaborative approach adopted to develop the framework is something that can and should be replicated in other organizations interested in undertaking a similar effort. AGRA and Mathematica thoughtfully used an in-person co-design sessions and frequent consultations with key actors within the systems including programme leads, funders, and external evaluators to inform the creation of the framework. Bringing together those who typically hold more power such as funders and those who have less power, such as programme delivery personnel, is in itself a beneficial effort in the development sphere. Collaboration is the first step to ensure that the methodologies we create become powerful tools to understand and measure systems and ultimately create better outcomes for people. Would it be possible to go one step further and involve communities who are on the frontline? What if people at the center of the systems of interest get to tell us how to understand systems and whether and how they are changing? If you are interested in these questions, we encourage you to explore these other cases in this series:
