Gauging change in the aggregate across portfolios

UNDP’s Strategic Innovation Unit (SIU) is supporting the UNDP to move beyond single point solutions to creating the conditions for system transformation. To do this The UNDP is developing and adopting a portfolio approach to designing and managing its interventions.

Illustrations: Ivana Čobejová

The UNDP define a portfolio as a set of interconnected interventions designed & dynamically managed to generate a continuous supply of options over time to help deliver strategic development impact when facing a challenge of system transformation (ref. UNDP Portfolio Primer). The UNDP portfolio approach implies understanding issues from a system’s perspective and leveraging linkages across interventions to achieve broader goals. 

This new approach raises the question: How can we measure progress, value, and results in the aggregate (across a portfolio of interventions) towards positive change in a complex system?  

We have already explored in previous blogs how practitioners are approaching this. Including measurement and complexity, progress tracking and reporting, how to measure systems change and how to do M&E when working with complex problems. 

In this blog we share what we learned from listening to the experiences, practices, and lessons from three MEL projects/teams working in portfolios. These include:

1

Achieving Sustainable Partnerships for Innovation, Research, and Entrepreneurship (ASPIRE) a 5-year collaboration funded by USAID, between the Massachusetts Institute of Technology (MIT), the Universidad del Valle de Guatemala (UVG), and the Guatemalan Exporters Association (AGEXPORT). The ASPIRE collaboration is premised on the thesis that local innovation ecosystems play a crucial role in driving local and regional economic development. 

2

JEEViKA Special Purpose Vehicle for Agricultural Transformation (JSPVAT) is a Bill and Melinda Gates Foundation (BMFG) funded agricultural transformation initiative providing technical assistance and advisory services to support the organizational capacity of 27 Farmer Producer Companies operating under JEEViKA to leverage the resources available to them through the World Bank. JSPVAT is part of a portfolio of investments under BMGF’s Memorandum of Cooperation with the Government of Bihar to strengthen the state systems and support to poor rural women farmers.

3

Portfolio Monitoring, Evaluation and Learning (PMEL) is an Evidence and Learning intervention which aims to increase the impact of the United Kingdom’s Foreign, Commonwealth and Development Office’s (FCDO) combined development efforts in Nepal. 

The ASPIRE and JSPVAT evaluations are both being conducted by the MIT Local Innovation Group, and the PMEL evaluation is being conducted by Abt Associates UK. All three cases share a common challenge to make sense of the aggregate change their portfolios are creating/contributing to. We drew three, we believe important lessons for others to consider when developing approaches to measure progress, value, and results in the aggregate across a portfolio. These are:

✔️ Frameworks are a way finding tool for portfolio measurement.

Frameworks are a tool to structure empirical inquiry and theory development. They make explicit the key boundaries, assumptions, ideas, and beliefs driving a deliberate intervention or enquiry. Frameworks are used to succinctly capture what and how interventions are contributing to change and to meaningfully bound the assessment, learning and decision-making processes for the intervention. Frameworks are not new to MEL. In theory-driven evaluations the framework most often used is Theory of Change, evaluations more generally are guided by analytical frameworks that include the evaluation criteria, questions, and judgement tools.

When doing MEL on portfolios aiming for system change one of the key challenges is to manage the inherent complexity and huge scope of these portfolios. Portfolios aiming for system change tend to demonstrate complexity at multiple levels, including their goals which are to shift complex-adaptive systems, the diversity and number of portfolio interventions and other interventions active in the system that are not part of the portfolio. 

The MEL team implementing the JSPVAT and ASPIRE evaluations, based at the MIT Local Innovation Group, is using middle range theories to construct a meaningful and implementable methodology to gauge the Portfolio contribution to system change. Middle range theories do not theorize the effects of specific interventions. Instead, they theorize how changes to a system of interest might contribute to a set of system impacts. In the JSPVAT case the system impacts are positive changes for poor women farmers that are members of Farmer Producer Companies in Bihar. The systems of interest, in the JSPVAT case, are the agricultural value-chains that include Farmer Producer Companies and associated systems such as the government of Bihar and the engaged donor systems. The framework that the JSPVAT evaluation team are building theorizes how poor women farmers might benefit from re-engineered local agricultural value-chains. This theory then provides a canvas against which JSPVAT and other agricultural value-chain interventions can theorize their contribution.

Middle range theories provide the architecture for a system change framework against which the following changes can be tracked:

  • System impacts or improvements for key beneficiaries, in this case poor women farmers. 
  • System of interest or intermediate conditions - the general conditions, processes, and pattern of outcomes, in this case of agricultural value chains which create improvements for key beneficiaries.
  • System shifting outcomes or the intermediate outcomes triggered by the portfolio interventions which nudge/push the system of interest towards the intermediate conditions.

We have also seen frameworks based on applied theories of system transformation such as the application of the six conditions of system change, and the Four Keys. See our blog on CALI for an example of this. 

For each level in the system change framework, different methodologies are developed to investigate changes. In the case of JSPVAT, the evaluation team is running quantitative surveys to investigate whether poor women farmers are experiencing benefits from their engagement with farmer producer companies. They are also conducting case study research to test and refine their ex-ante theory of change for how the type of value chain intervention undertaken by JSPVAT might lead to improvements for poor women farmers. And they are using back casting causal analysis to investigate the link between changes at the farmer and local agricultural value-chains level back to the interventions.

What are the characteristics of frameworks that work best to measure progress, value, and resultsfor system shifting portfolios? 

  • Those that take a system centric view rather than an intervention centric view.
  • Those that are simple enough to cut through the complexity to the critical things the portfolio is wanting to discover or test in its endeavor to shift systems. 
  • Those that sufficiently bound the system shifting endeavor and portfolio to drive meaningful work, learning, and evolution of the portfolio. 
  • Those that allow zooming in and out or toggling between different levels of the system and between the system and the interventions.

✔️ After-Action Reviews, a humble yet powerful tool for working in complex-adaptive systems.

The challenge when working on complex system change portfolios like ASPIRE is that the complexity can be overwhelming. The ASPIRE team use After-Action Reviews (AAR) to manage the complexity.  

An AAR is ‘a simple method for facilitating an assessment of performance by bringing together a team to discuss a task, event, activity or project in an open and honest fashion.1’ At its essence an AAR brings together a relevant group to reflect on a project or activity using a series of questions to surface what happened, what worked/did not work, why, and what might be done differently next time. The simplicity in the AAR leads to easy adaptation of the AAR process to best suit the project, action, effort it is applied to. 

1  After action review | Better Evaluation

 

The ASPIRE team identify and support local innovation projects to create local innovation pathways and associated resources, competencies, infrastructure which ultimately strengthen and re-orient the local innovation systems towards addressing local development challenges. You can read about MIT’s multi-actor ecosystem strengthening process here. To guide the decisions and management of their portfolio of innovation and associated projects the ASPIRE team use two frameworks:

  1. A diagnostic of the capacity of the local innovation ecosystem to innovate built from MIT’s local innovation ecosystem model which includes three core system elements: actors, resources, and the enabling environment (See framework image below, courtesy of MIT). And an assessment of structural and functional innovation capacity at different levels (individual, group, and system).

2. A Theory of Change (TOC) developed by adapting a middle-range theory of how inclusive agricultural innovation processes lead to strengthened innovation ecosystems to the local context using the empirical insights from the system diagnostic. This led to a more precise ASPIRE TOC with more specific intermediate outcomes and indicators and evidenced assumptions. 

The diagnostic provides useful detail in the identification and design of local innovation projects whereas the middle-range theory provides an overall direction for the ASPIRE portfolio. The diagnostic and the Theory of Change are the ASPIRE frameworks that help to cut through the complexity and bound the system shifting portfolio. When combined with regular (6 monthly in the case of ASPIRE) AARs these produce important benefits for the portfolio, including:

  • Keep the portfolio and its interventions on course. Checking in as a whole team every 6 months allows for testing and re-orienting of the interventions. The use of a theory of change in the AAR is important in challenging the demand for pilot projects and ensuring the ASPIRE project is not diluted. It forces questions around the contribution of the pilot to increasing the production of locally developed research and innovation. 
  • Consolidate learning and evidence of progress. Throughout the AAR discussions, the team shares new insights and evidence about the local ecosystem that surfaced through the pilots. This allows for learning to connect back to the diagnostic and the ToC, updating these as needed, and for the whole ASPIRE team to have a shared and current understanding of the local ecosystem, the portfolio of pilots, and how these are contributing (or not) to strengthening the local innovation ecosystem. 
  • Forces the whole ASPIRE team to reflect on the meaning and usefulness of the collected data. For example, when the MIT evaluation team present the results against the project indicators which are mapped to the TOC the ASPIRE team reflect on whether the indicator is telling them something meaningful about the progress of ASPIRE against the TOC and therefore the ASPIRE goals. If the answer is no, then the team get to rethink whether another indicator would be more meaningful in tracking progress against a specific outcome or whether they should be focusing on a different outcome that might tell them more about their progress over a 6-month period. 

When used regularly and systematically, the AAR helps to drive learning and change.

✔️ Shifting from programmatic to portfolio management – a critical challenge.

The original idea for developing a portfolio framework at DFID Nepal was to build a portfolio level Theory of Change (TOC) that would articulate the main causal pathways through which the office contributes to transformative change in Nepal. The PMEL team started on this journey by conducting a portfolio TOC diagnostic to gauge how well the existing portfolio articulated the main causal pathways. 

The diagnostic used information from key programme documents, combined with semi-structured interviews with key office staff and implementing partners. The diagnostic investigated three areas:

  • Horizontal logic across the portfolio: to what extent are there causal pathways or linkages between major investment areas? Can it be said that these areas, together, may have a greater combined impact than they would apart? Or are they isolated, self-sufficient investment areas that could largely achieve their outcomes without the contribution of other parts of the portfolio? 
  • Vertical logic between different levels of the portfolio: to what extent are the vertical causal pathways clear in the office portfolio both: within major investment areas between major investment areas and the overall portfolio goal (i.e. the leap of faith between outcomes and goal)?
  • Organizing vs causal pathways: to what extent is there a coherent causal pathway at the portfolio level? Or to what extent is it an organizing framework for investments under a single portfolio?  

The diagnostic found that the portfolio aligned to the description of a Portfolio as an organizing framework. The PMEL team worked closely with the office to a) design a portfolio TOC, b) monitor the footprint (or reach) of the portfolio in Nepal, and c) provide portfolio level results and learning. The PMEL team worked from the programme data up to produce portfolio results and insights. These produced aggregated results on the programme activities and reach into the Nepal system. These have been useful to the office in informing internal discussion between the Ambassador, Development Director, and the Nepal Country Board about progress towards country plan goals. 

Shifting from programmatic to portfolio management is challenging when all the incentives are geared towards thinking in programmes. 

💬 Messages from evaluators: Do not lock yourself in too early! And other pearls of wisdom… 

Throughout the exploration of these cases, the practitioners we spoke to shared valuable reflections for others on or about to embark on developing frameworks to progress, value, and results in the aggregate (across a portfolio of interventions) towards positive change in a complex system? We share some of them here:

A key practice should be to ‘build in discovery and space to specify a measurement strategy once you know enough about the system to develop a meaningful measurement strategy’. System change efforts are much harder to bound than programmatic interventions and it takes time to begin to know a system. The ASPIRE and JSPVAT teams dedicated the first year to learning about the system and developing 1.0 versions of their frameworks to guide the MEL. For this to work the funder needs to be on board and understand the importance of this practice.

Build in practices that support regular documentation of the portfolio efforts, progress, and learning. AARs are helpful in enabling documenting progress for a complex intervention.The AAR outputs for ASPIRE’s 6-monthly AARs include the next 6-month plan and the AAR forms the basis for the report to the funder. The AAR can do this because it brings together the ASPIRE stakeholders to share data on project activities, progress, and lessons, and in planning the next tranche of work.