Making sense of market systems change

UNDP and the Bill and Melinda Gates Foundation are partnering in an effort intended to help us rethink how to do monitoring, learning and evaluation (MLE) when working with complex systems and in unpredictable environments. In addition to supporting efforts such as UNDP’s M&E Sandbox, this effort is enabling us to generate a series of case studies documenting emerging practices that take a systems-aligned approach to MLE. In this post we share the first of a series of case studies of efforts that seek to measure and make sense of systems change. We will publish an additional 3 cases over the coming weeks and wrap up the series with a comparative piece that draws out insights, similarities, and differences across these different cases.

Illustrations: Ivana Čobejová

The case study below looks at ACDI/VOCA’s work on Market System Diagnostics in Honduras. The case provides a useful insight into the rationale and design process for the approach, how the organization (with local partners) have evolved the approach over time, the types of data and analytical methods used, and the results and learning generated in the process.

The approach deployed by ACDI/VOCA has a number of distinguishing features (more on these in our comparative piece to be released later). 

1

The approach is intended as a learning tool to help ACDI/VOCA and local market system actors to understand the dynamics, drivers and opportunities in the Honduran economy – it is not intended as a tool to capture whether specific interventions or stakeholder have contributed to systems change. Related, the approach has evolved to become a more inclusive process with scope for Honduran partners to shape the focus of the diagnostics and make sense of the insights it generates. 

2

The analytical framework underpinning the approach has a few distinguishing features. First, it has a strong focus on causal factors shaping the dynamics and direction of the Honduran market system. These factors are identified through workshops with local stakeholders. This more ‘inductive’ approach contrasts with approaches that tend to start with a pre-defined conceptual framework for what is important to look at in a system (another of the cases we will cover has used the water of systems change framework). Second, the focus on causal factors in a system is one of several potential ways of making sense of (or measuring) a system. Other case studies in our series will cover approaches that, for instance, focus primarily on the role of relationships and relational dynamics between actors in a system.

3

The 2020 iteration of the approach draws on a mix of qualitative and quantitative data as well as statistical and story-based approaches to understanding change. While the qualitative bit is not as mature as the quantitative statistical component, the approach contrasts to other approaches for measuring systems change that tend to have either a strong quantitative or qualitative focus.

The Centre for Public Impact has helped produce these studies and we thank them for their insights and excellent collaboration. We also extend a great thanks to ACDI/VOCA and their Honduran partners for sharing their journey and learning with us. We hope you find this first case study interesting and useful, and we hope that you’ll stay tuned for the next pieces in this series.

A case study on how ACDI/VOCA is using a system diagnostics approach to learn about change in a complex market system

Within the landscape of Monitoring, Learning, and Evaluation (MLE) methodologies, it is possible to develop a bit of tunnel vision. MLE can sometimes focus too heavily on what we expect to see and not see the system for how it really is or how it is changing. This can give an incomplete understanding of the system so we never truly appreciate its potential for change. Some organizations such as ACDI/VOCA  are developing  more comprehensive MLE strategies attempting to offer a fuller and more dynamic perspective of systems change. One way ACDI/VOCA is exploring this is through the creation and implementation of a Market System Diagnostic (MSD) framework in Honduras, developed as part of their collaboration with USAID on the Transforming Market Systems Activity (TMS).

Transforming Market Systems

The international nonprofit ACDI/VOCA, headquartered in Washington, D.C., focuses globally on transforming market systems to address food security, economic growth, and inclusion for vulnerable communities.

Their work involves partnering with governments, civil society, private firms, and financial institutions to implement programs centred on locally-driven solutions.  Launched in 2018, one of their key initiatives, theTransforming Market Systems Activity (TMS), is part of the Alliance for Prosperity and Honduras 20/20 collaboration between USAID/Honduras and the Government of Honduras. It aims to address root causes of illegal migration by fostering systemic change in the Honduran market system. The primary goal is to create economic opportunities, new jobs, and increased income, especially for marginalized groups often excluded by traditional systems, ultimately reducing outward migration. TMS focuses on sectors such as Value-Added Agriculture and Tourism and Creative Industries, and looking at market shaping factors such as entrepreneurship, and the business-enabling environment, collaborating with government, civil society, and the private sector.

The market system diagnostic

The TMS’s Market Systems Diagnostic is a methodology created to analyse functional changes in market structures and enterprise behaviours with the purpose of understanding how and whether the Honduran economy is evolving to create more and better jobs, and becoming more inclusive, competitive and resilient in that process.

The framework aims to deepen the understanding of the Honduran economy on a national level by diagnosing the factors that drive or inhibit inclusive economic growth through qualitative and quantitative methods. It functions as a tool for the organization to provide more insightful information for the TMS initiatives, with the objective of monitoring whether and how the Honduran market system is becoming more competitive, resilient, and inclusive.

Finally, the diagnostic results are shared with stakeholders involved in the framework's development, fostering evidence-based decision-making across the public, private, and academic sectors, with the potential to contribute to tangible changes in Honduran society.

Who led the creation of the Market System Diagnostic?

In 2018, the TMS Activity partnered with the Council of Private Enterprises of Honduras (COHEP), the National Autonomous University of Honduras (UNAH), and the Central American Institute of Business Administration (INCAE) Business School to diagnose the determinants of inclusive economic development and job creation in Honduras from the perspective of the principal job creation agent, i.e., private enterprise.

🔎 This collaboration materialised in the creation of the 'Market System Diagnostic' (MSD), a framework designed to provide industry-level insights into key economic trends and the system attributes that influence market system performance over time. 

💬 Inside the Journey: What inspired the creation of the MSD framework?

In Honduras, the market system is heavily influenced by an informal workforce1, leading to limited availability of data for analysis. For instance, in 2018, self-employment and informal jobs accounted for 56.5% of total employment, marked by lower wages and instability. According to TMS, the lack of quality jobs is at the heart of persistent poverty and underdevelopment. Economic conditions also drive external migration, with 91% of migrants leaving in search of better economic opportunities. These factors guide TMS's efforts to create new opportunities in Honduras.

Additionally, challenges in economic regulations and worker protections exist, along with issues related to the absence of registers and limited data access due to informality. This makes it difficult to fully understand the market system's composition, operations, and changes, hindering the ability to contribute to improvement projects and interventions. For example, there is no sampling frame or accessible enterprise listing available from public archives, as many enterprises do not legally register, posing a significant challenge for change agents trying to understand and navigate the system.

This is why in 2018, the TMS team partnered with COHEP, UNAH, and the INCAE Business School to invest in diagnosing the determinants of inclusive economic development and job creation in Honduras from the perspective of the principal job creation agent, i.e. private enterprise. They created the 'Market System Diagnostic' (MSD), a framework designed to provide industry-level insights into key economic trends, behaviours and the system attributes that influence market system performance over time, thereby creating new data and insights about the system.

1 Interview with Juan Umanzor and Christian Ramos, National Autonomous University of Honduras Institute for Economic and Social. Investigation, Universidad Nacional Autónoma de Honduras (UNAH). Conducted by the Centre for Public Impact, online, 3 November 2023.

💬 Inside the Framework: How is the Market System Diagnostic tool Structured? 

Since its inception in 2018, the MSD has evolved as a methodology co-designed with key stakeholders in the Honduran market system.

🔎 2018 Structure 

In 2018, the initial diagnostic study examined 616 businesses in Honduras, representing various sizes and sectors within the value-added agricultural and tourism industries. This investigation focused on four key market segments: agroindustry, distribution companies serving as pivotal components in agricultural supply chains, lodging establishments, and tourism services2, across nine different departments in the country. The development of the 2018 MSD involved a quantitative and qualitative approach, primarily relying on a survey and supported by statistical analysis. The purpose of using both methods was to better understand, from two fronts, the numerous variables influencing the Honduran market system and how these relate to and influence each other.

Here are the key two steps in the process:

1. Survey Sampling:

  • The sample frame aimed for representation across four key enterprise functions: agroindustry, distribution entities central to agricultural supply chains, lodging establishments, and tourism services. The selection of these functions was based on their critical roles in the market system, with a focus on whether changes in behaviours or structures among respondents could indicate broader shifts in the Honduran market system.
  • The list of enterprises was drawn from various sources, including the National Commercial Directory of Buyers by the Secretary of Agriculture and Livestock (SAG), the Agribusiness Unit of the National Agri-Food Development Program (PRONAGRO), and other secondary sources. For the tourism sector, the population list was developed from the Honduran Institute of Tourism Directory of Tourism Establishments.
2. Statistical Analysis: 

  • The statistical process involved testing the strength of the relationship between each variable using Spearman's rank correlation.
  • TMS emphasised that correlation does not imply causality, but regression results can be valuable as they provide a predictive relationship and summarise the connections between multiple variables.
2 USAID, ACDI/VOCA’s TMS Activity, COHEP, UNAH, INCAE. “White Paper. Diagnosing Honduran Market Systems: The drivers of inclusive economic growth in Honduras”. ACDI/VOCA, 2019.  Available at: https://www.acdivoca.org/wp-content/uploads/2021/01/TMS-White-Paper-MSD-2018.pdf (accessed on 6 November 2023) p.9.

 

🔎 2020 Structure 

In 2020, the MSD sampling scope included agro-industry, commerce, lodging, and food services3. However, based on learning from the 2018 MSD, efforts were made to ensure that the approach and focus was shaped by a wide range of local stakeholders. This lead to broader sectoral focus, including manufacturing and other diverse industries which helped to enhance the representation and scope of analysis of this market system in Honduras.

The following process reflects its most recent iteration in 2020.

Step 1 - Stakeholder workshops to identify relevant variables:

First, the MSD team held stakeholder workshops to identify emerging themes and priorities. ACDI/VOCA achieved this by closely collaborating with local Chambers of Commerce, as these are among the few institutions in Honduras that can provide access to market data and enterprise connections. These workshops, involving 65 participants, gathered experiences and stories to create a list of variables to inform the enterprise survey. Variables included relevant issues influencing the Honduran market system based on the viewpoints of workshop participants. Key themes emerging from this more inclusive process that were not present in the 2018 MSD included economic competitiveness, inclusion, and resilience to shocks.

Step 2 - Quantify variables into indicators through enterprise survey:

Next, the MSD team transformed the identified variables from the stakeholder’s workshops into measurable indicators  for survey questions. In November 2020, ACDI/VOCA shared their enterprise survey across 7 industries, which resulted in 786 respondents across industries and enterprise sizes as highlighted below. The initial idea was to undertake these surveys with support from UNAH students (taking the role of ‘door-to-door’ surveyors) to ensure wider representativeness from enterprises. However, because of the COVID-19 pandemic, surveys were only distributed online.

Step 3 - Identify “pairwise influences” between variables:

After the community survey, the MSD team identified ‘pairwise influences’ between variables based on respondents' answers. Pairwise influences can help to understanding the relationships between variables and their impact. Generating a stronger understanding of the relationships between variables within a system is useful as it provides insights to support more effective decision-making, problem-solving, and understanding of complex phenomena. ACDI/VOCA achieved this by deploying a three-step approach: LASSO for statistical analysis, expert workshops to validate influences, and subject matter expert evaluation to rate connection strength (1-3). 

Step 4 - Run ‘systems-level analysis’ to study complex relationships:

Next, the team conducted system-level analysis, to identify how the groups of variables interact with broader trends in the market system. The MSD team used MICMAC and network analysis techniques to identify variables' influence, dependence, and betweenness centrality. This approach revealed highly interdependent variables in the Honduras market system, which then uncover nine change dynamics across the following areas: 

  • Institutions and rule of law
  • Taxes and public expenditures
  • Competition and cooperation
  • Connectivity to markets 
  • Business strategies
  • Risk and resilience
  • Growth and investment
  • Inclusive economic opportunities
  • Workforce and labour markets

The  assumption here is that this grouping enables an analysis and interpretation of the broader causal "story" behind how these groups of variables interact to influence broader trends in the market system.

Step 5 - Stakeholder discussion and interpretation of results:

Finally, teams organize and facilitate sensemaking workshops with community stakeholders composed of business leaders to discuss the preliminary results of the data analysis. Sensemaking can be done in various ways, but is a process to create space for listening, reflection and the exploration of meaning beyond the usual boundaries that people operate within. This allows groups to share and collectively explore different framings, stories ,and viewpoints. The ACDI/VOCA team conducted eight workshops with 90 participants, to present and make sense of the initial findings. 

Following the stakeholder discussion, the TMS team conducted a final analysis drawing on insights generated during the previous 4 steps. The 2020 application of the MSD, resulted in the identification and measurement of 43 variables through 112 quantitative indicators. The team ranked the indicators based on 4 factors: 

  1. Influence which measures variables that have the greatest effects on the general evolution of the Honduras market system.
  2. Dependence which measures variables which are most sensitive to changes in other variables.
  3. Centrality (betweenness) which measures the centrality of a variable based on whether it lies on the shortest path between all other variables. For the TMS, this represents critical nodes in the system because influence and dependence flows through these variables.
  4. Change over time which is a context measure added to consider whether the variable has changed between 2018 and 2020. The degree of change signals the pace of change, which might indicate the tendency to micro-level or macro-level change processes.

The ultimate purpose of this ranking is to provide guidance to policymakers, academics, or market system actors by pinpointing areas where an intervention or change could have a substantial impact on modifying other variables, thereby influencing the overall performance of the market system.

After the ranking, variables are marked as follows: 

  • Leverage point variables symbolise places to intervene in the system. Leverage points determine how the Honduran market system evolves due to their strong influence and low dependence on other variables. 
  • Critical (or relay) variables are highly influential but also highly dependent on other variables in the system which makes them both volatile and important for how the system performs.  
  • Borderline variables have moderate influence and moderate dependence on other variables in the system. These variables may function as leverage points, critical or relay variables in certain contexts meaning they may function as autonomous or dependent variables in other contexts.  
  • Autonomous variables have low influence and low dependence on other variables in the system. They are relatively disconnected and tend to be considered as context factors.
  • Dependent variables are variables which have low influence but high dependence on other variables in the system. These variables will change with changes in leverage points and critical or relay variables. 

3 Investigation, Universidad Nacional Autónoma de Honduras (UNAH). Conducted by the Centre for Public Impact, online, 3 November 2023.

Results: What did the Market System Diagnostic help the teams to achieve?
✔️ The team has leveraged what they learned to support local entrepreneurs to navigate the system more effectively. 

The MSD has empowered the team to assist local entrepreneurs in adapting to dynamic market conditions. For instance, Sube Latin America, an e-commerce startup, used MSD insights to discover that limited support for Micro, Small & Medium Enterprises (MSMEs) purchasing their services was the root cause of declining sales, prompting them to pivot and focus on supporting MSMEs more effectively. Additionally, the MSD insights helped to identify how banking regulations were hindering e-commerce transactions, leading Sube Latin America to establish the FinTech Association to address these issues. Dun Grover, Deputy Chief of Party of Market Systems and Collaborating Learning and Adapting (CLA) for TMS Activity, notes that local businesses are increasingly adopting a systemic approach based on MSD insights because: “if you are not assessing whether you are on track to being successful, it is like trying to make changes without any navigation.”

✔️ The MSD has served as a knowledge tool for local public policy and decision-makers. 

The MSD played a pivotal role in shaping the National Business Strategy for Honduras 2022-2025 led by the Honduran Council of Private Enterprise. The strategy addresses improvements across 64 aspects within eight key areas, including the rule of law, governance, tax policy, financial systems, market structures, innovation and entrepreneurship, human capital, and health. It directly integrates MSD insights to address Honduras' economic recovery and inclusive job creation challenges. In 2021, COHEP distributed the strategy to presidential candidates and political parties, inviting them to learn more about these recommendations in their policies for the upcoming administration.

Learning: What learning did the Market System Diagnostic generate for the teams?
✔️ Collaborative approaches have the potential to further elevate the position of local research entities.  

MSD's initiatives significantly raised local research entities' status, fostering collaborations, securing more funding, and driving breakthrough innovations. For instance, Gustavo Adolfo Acosta Pineda, Systems Learning and Analytics Specialist for the Transforming Market Systems Activity, highlighted how DSM boosted UNAH's position in Central America's economic and social research. These results and methodologies were shared in regional forums, expanding UNAH's networks with respected educational institutions like the University of Costa Rica. Additionally, it formed new partnerships with local institutions, enabling joint studies and information exchange. Notably, it established agreements with three Chambers of Commerce and two trade organizations, a substantial shift from the previous lack of collaboration.

✔️ Measuring systems change over time requires patience and flexibility.  

A tension that emerged for the TMS team included the difficulties associated with the time frame allocated for assessing and learning from MLE activities, both within and beyond the MSD. This challenge is also something that Laudes Foundation has been grappling with in its rubrics-based framework for measuring systems change (see more our M&E Sandbox blog). Typically, the team is tasked with the challenge of focusing on yielding results, a process that, even in the best case, may span three to five years. Dun Grover, Deputy Chief of Party of Market Systems and Collaborating, Learning, and Adapting (CLA) for the USAID/Honduras Transforming Market Systems Activity noted that changing systems might require ten years or more to actually see significant change occur. According to Grover, "it takes patience to develop the system that is the right fit for you." As such, the TMS team has been proactive in revisiting the framework and sharing results biannually, adapting the method based on multiple insights, such as the need to broaden the range and variety of stakeholders from the 2018 to the 2020 MSD. Incorporating more frequent reflection sessions to adapt and account for these changes and needs appears fundamental in addressing the tension between funding and time frame limitations. Using stories as ways to capture early change be in this connection be useful. The objective is to capture change as it happens, including adaptations to the methodology and the team's learning approach.

In sum, the MSD approach presents useful insights for those working in market systems who are interested in exploring new approaches to tackle complex challenges. This includes how to deal with access to data and representation of a wider range of stakeholders in systems analysis. The case also underlines the usefulness of qualitative methods such as storytelling to capture interim progress, enable insights from communities to be captured and valued, as well as to inform MLE teams to learn and adapt at a faster pace.

✔️ Embracing complexity: An invitation to explore storytelling.  

Looking ahead, the TMS team plans to embody a learning and adaptive approach by continuing to evolve and co-design this methodology with Honduran partners. One of the challenges that emerged was how difficult it was to conclusively evaluate Honduras’ market system on official public data alone. This is because in Honduras, a large percentage of residents participate in the informal economy. As such, this means that there are little to no data sets to represent the financial reality of many residents. Furthermore, without this available data, it also means that any initial analysis is likely skewed to mostly represent how the formal economy behaves. In the spirit of embracing complexity and experimenting with more agile MLE methods to capture real-time insights, ACDI/VOCA plans to adapt their approach by embedding storytelling to capture changes as they emerge within the system. As the MSD team prepared for MSD 2022, they hypothesized that by incorporating story-based methods more often, they will be “getting richer, qualitative narrative-based stories earlier on, not just once in midterm and interim reviews, but regularly to offer feedback and help the team to understand if there are unintended consequences, positive or negative, or second-order things that are happening”4. In the world of interconnected MLE practices, this aligns with strategies that practitioners like Poverty & Human Development Monitoring Agency (PHDMA) have adopted. In September 2023, at  a UNDP's M&E Sandbox, they shared their narrative-based approach to measure change in poverty and human development in Odisha, India - a testament to the potency of storytelling for real-time capture of complex change processes.

4 Interview with Dun Grover, Deputy Chief of Party of Market Systems and CLA for the USAID/Honduras Transforming Market Systems (TMS) Activity. Conducted by the Centre for Public Impact, online, 5 September 2023.