integrated problem solving model

The Integrated Problem-Solving Model of Crisis Intervention: Overview and Application

J. Westefeld , C. Heckman-Stone

Mar 1, 2003

Influential Citations

The Counseling Psychologist

Key Takeaway : The Integrated Problem-Solving Model (IPSM) effectively addresses crisis situations, particularly sexual assault, and can be applied to counseling psychologists' roles in crisis intervention and research.

Crisis intervention is a role that fits exceedingly well with counseling psychologists' interests and skills. This article provides an overview of a new crisis intervention model, the Integrated Problem-Solving Model (IPSM), and demonstrates its application to a specific crisis, sexual assault. It is hoped that this article will encourage counseling psychologists to become more involved in crisis intervention itself as well as in research and training in this important area.

• Search Menu
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

1. introduction, 2. evolving modes of research, 3. evolution and evaluation of cgiar research—forest, trees and agroforestry case study, 4. evolving concepts and measures of impact, 5. towards an integrated, mixed methods approach for evidencing r4d impact, 6. impact assessment strategy framework for an integrated tdr programme, 7. conclusion, acknowledgements.

• < Previous

Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations

• Article contents
• Figures & tables
• Supplementary Data

Brian M Belcher, Karl Hughes, Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations, Research Evaluation , Volume 30, Issue 2, April 2021, Pages 154–168, https://doi.org/10.1093/reseval/rvaa024

• Permissions Icon Permissions

Researchers and research organizations are under increasing pressure to demonstrate that their work contributes to positive change and helps solve pressing societal challenges. There is a simultaneous trend towards more engaged transdisciplinary research that is complexity-aware and appreciates that change happens through systems transformation, not only through technological innovation. Appropriate evaluation approaches are needed to evidence research impact and generate learning for continual improvement. This is challenging in any research field, but especially for research that crosses disciplinary boundaries and intervenes in complex systems. Moreover, evaluation challenges at the project scale are compounded at the programme scale. The Forest, Trees and Agroforestry (FTA) research programme serves as an example of this evolution in research approach and the resulting evaluation challenges. FTA research is responding to the demand for greater impact with more engaged research following multiple pathways. However, research impact assessment in the CGIAR (Consultative Group on International Agricultural Research) was developed in a technology-centric context where counterfactual approaches of causal inference (experimental and quasi-experimental) predominate. Relying solely on such approaches is inappropriate for evaluating research contributions that target policy and institutional change and systems transformation. Instead, we propose a multifaceted, multi-scale, theory-based evaluation approach. This includes nested project- and programme-scale theories of change (ToCs); research quality assessment; theory-based outcome evaluations to empirically test ToCs and assess policy, institutional, and practice influence; experimental and quasi-experimental impact of FTA-informed ‘large n’ innovations; ex ante impact assessment to estimate potential impacts at scale; and logically and plausibly linking programme-level outcomes to secondary data on development and conservation status.

Researchers and research organizations are under increasing pressure to demonstrate that their research contributes to positive change and helps to solve pressing societal challenges. Appropriate evaluation is therefore needed, not only to evidence research impact, but also to generate learning to improve research design and, ultimately, enhance impact. It is also critically important to demonstrate the contribution of research to solving development problems and leverage opportunities to attract, allocate, and optimize investments in research. This is challenging in any research field, but especially for integrated research programmes that cross disciplinary boundaries to intervene in complex systems.

The drive for increased research impact has led to a marked evolution in the way research-for-development (R4D) is understood, conceived, and implemented, with more inter-disciplinary research and transdisciplinary forms of collaboration between researchers, research users, and other stakeholders ( Nowotny, Scott and Gibbons 2001 ; Kasemir, Jaeger and Jäger 2003 ; Hirsch Hadorn et al. 2006 ). This evolution reflects epistemological assumptions that are very different from those of traditional disciplinary approaches ( Talwar, Wiek and Robinson 2011 ). There is greater appreciation of contingency and uncertainty in science ( Gibbons et al. 1994 ; Nowotny, Scott and Gibbons 2001 ). There is also recognition that scientific knowledge alone is not sufficient for action, and rather that sustainable development entails many normative considerations that link knowledge and action ( Functowicz and Ravetz 1993 ; van Kerkhoff and Lebel 2006 ). Furthermore, there is greater appreciation that the knowledge and values of stakeholders and intended users of research are relevant, valid, and important, and that each has their own motivations and biases that influence how they interact with and make use of new knowledge ( Kasemir, Jaeger and Jäger 2003 ).

This changed understanding has led to fundamental changes in the way many researchers work. New problem-oriented research approaches have evolved to engage system actors in the research process as a way to increase research effectiveness. Variations on these approaches are known as Post-Normal Science ( Functowicz and Ravetz 1993 ; Ravetz 1999 ), Mode 2 research ( Functowicz and Ravetz 1993 ; Gibbons et al. 1994 ), Problem Driven Iterative Adaption (PDIA) ( Andrews, Pritchett and Woolcock 2013 ), Transdisciplinary Research (TDR) ( Klein 2006 ; Walter et al. 2007 ; Carew and Wickson, 2010 ; Pohl et al. 2010 ; Jahn, Bergmann and Keil 2012 ; Lang et al. 2012 ; Wolf et al. 2013 ), and Sustainability Science ( Kates et al. 2001 ; Clark and Dickson 2003 ; Komiyama and Takeuchi 2006 ; Brandt et al. 2013 ; Kauffman and Arico 2014 ; Heinrichs et al. 2016 ; Kates 2017 ; Roux et al. 2017 ). There has also been a recent turn towards large, coordinated, multi-disciplinary research collaborations focused on major societal problems, such as the Grand Challenges in US universities ( Popowitz and Dorgelo 2018 ) and the Global Grand Challenges ( Bill and Melinda Gates Foundation n.d. ).

The CGIAR (formerly known as the Consultative Group on International Agricultural Research), an international consortium on agriculture and natural resource management (NRM) research, provides a good example of this transition and the corresponding learning and impact assessment challenges. An organizational reform process, which began in 2008, increased accountability for realizing social, economic, and environmental outcomes and impacts, on top of the long-established commitment to producing high-quality science. This shift was made explicit as a commitment to ‘shared responsibility’ ( ISPC 2015 : 5) for impacts in terms of reduced poverty, improved food and nutrition security, and improved natural resources and ecosystem services ( CGIAR 2016 ). A key aspect of this reform process was the creation of CGIAR Research Programs (CRPs) in 2011. These aimed, in part, to build broader and deeper partnerships, not only with other research organizations, but also with a range of policy and development actors at international and national levels, including conservation and development organizations, non-governmental organizations, policy actors, and other stakeholders. This emphasis on working through partnerships appreciates that high-quality scientific knowledge creation alone cannot address contemporary development and environmental challenges. The resulting research embodies many of the characteristics of problem-oriented TDR approaches. This is not to say that all CGIAR research is transdisciplinary; however, there is a growing proportion of CGIAR projects that apply TDR approaches, and the overall portfolio is increasingly integrated and focused on high-level challenges.

There has been good progress developing methods for assessing the societal impacts of research, especially in the health field ( Greenhalgh et al. 2016 ). However, contemporary concepts and methods of research impact assessment are not well suited for complex integrated research programmes. Impact assessment in the CGIAR has developed in conjunction with a historically technology-centric research model, leading to a substantial mismatch between the prevailing approach to research impact assessment and current needs and opportunities. Simply stated, relying solely on counterfactual impact evaluation approaches is inadequate for evaluating the full range of CGIAR research or for engaged problem-oriented research more generally. There is a need for a more complete and comprehensive set of approaches for analysing and demonstrating the impacts of integrated inter- and transdisciplinary research programmes, such as those of the CGIAR, Challenge Programs, and sustainability science. We need a broader and more nuanced conceptual framework of how research contributes to change in complex systems and how we can evaluate those contributions for both learning and accountability.

This essay explores the needs and opportunities for improved evaluation and impact assessment in an international R4D context, with a focus on lessons from the Forests, Trees and Agroforestry (FTA) CRP. The authors have many years of experience with research evaluation and impact assessment in the CGIAR and other international research and development organizations. We begin with a review of the evolution of problem-oriented research, using examples from the FTA. We then discuss the inherent challenges in evaluating this kind of research, including the limitations of conventional impact assessment approaches. We recognize that designing and selecting appropriate evaluation methods depends on expectations, so we propose a set of principles to shape such expectations and guide the development of more appropriate and realistic evaluation frameworks for multifaceted research initiatives that aim to bring about transformational systems-level change. We conclude by proposing an integrated evaluation framework to operationalize these principles.

To assess the impact of any intervention or innovation (including those informed by research), it is critically important to first understand the likely mechanism(s) through which it is expected to generate its intended effects, and to investigate the extent to which these expectations conform with reality ( Pawson 2003 ; White 2009 ). Indeed, there is a pushback against evaluations that fail to interrogate why interventions, programmes, and projects succeed and/or fail ( Harachi et al. 1999 ).

The standard and still prevailing mechanism through which R4D is expected to generate impact is the linear model of diffusion ( Godin 2006 ), also known as the pipeline model ( Hall et al. 2000 ). In this model, scientific discovery leads to technological innovation that is piloted, refined, and then disseminated to and adopted by intended users at scale, resulting in efficiencies, improvements, and benefits to society.

van Kerkhoff and Lebel (2006) suggest two versions of this conventional model that illustrate the link between knowledge and action. The trickle-down model holds that good research will be taken up by users based on its inherent value. The researcher’s job is to do good quality, innovative science and share the resulting knowledge through normal scientific communications, such as peer-reviewed articles and conference presentations. The second version, the transfer and translate model, emerged out of research utilization studies in the 1970s which recognized that trickle-down approaches had largely failed to influence social policy. This version of the model acknowledges the need for additional effort to communicate science, but it is still framed as a one-way process of translating, transferring, and mobilizing science-based knowledge to users. Examples include agricultural extension services, where specialized intermediary agents transfer research findings to users, or evidence-based healthcare, which consolidates scientific knowledge through evaluations and syntheses of existing scientific literature and translates the knowledge, ostensibly for application in clinical practice and public policy. The model assumes that there is an objective truth that can be discovered by science and that the main barrier to improved outcomes is lack of access to that scientific knowledge by intended users.

There is a growing consensus among practitioners, policy-makers, and the research community that the scaling of technological innovations alone cannot solve contemporary social and economic challenges ( Howaldt 2019 ). van Kerkhoff and Lebel (2006) list the key critiques: science is socially and institutionally embedded and cannot be entirely objective in its definition and execution; scientific knowledge is socially constructed, in that observations are subject to interpretation, so knowledge is always uncertain; the boundary between science and the rest of society is artificial, created by social and political processes and therefore changeable and contestable; power and special interests shape the linkages between research-based knowledge and action; and science reflects cultural biases and inequalities.

Indeed, there are high and increasing expectations that science should serve society and the benefits should be demonstrable ( Stokes 1997 ; Sarewitz 2016 ). Functowicz and Ravetz (1993) published their seminal work on ‘Science for the Post-Normal Age’, recognizing that in many contemporary societal problems, facts are uncertain and values play a major role in decision-making. As noted by Ravetz (1999) , ’[c]ontrary to the impression conveyed by textbooks, most problems in practice have more than one plausible answer, and many have no answer at all’ (649). The ideas and concepts of post-normal science call for new problem-solving strategies in which the role of science is appreciated within complex and uncertain natural and social systems.

Increased focus on problem-solving and social engagement has led scholars and researchers to develop TDR approaches that integrate across disciplines and beyond expert knowledge to embrace non-expert and public knowledge and enable social learning ( Randolph 2004 ; Hirsch Hadorn et al. 2008 ; Pahl-Wostl, Mostert and Tabara 2008 ; Robinson 2008 ; Lang et al. 2012 ). Lang et al. (2012) outline the key steps in TDR as: (1) joint framing of the problem and building a research team composed of different kinds of scientists and societal stakeholders; (2) co-producing solution-oriented and applicable knowledge through collaborative research; and (3) (re)integrating and applying the knowledge that has been produced in both scientific and societal practice. Thus, TDR promotes sustainable change with methods that give non-academic actors a role in the research process and which integrate and expand their knowledge and capabilities, leading to improved action.

Figure 1 illustrates, in a simple schematic diagram, some of the kinds of value that can be created at different stages in the research process through productive interactions.

TDR Contributions to Impact.

Sustainability Science likewise takes a TDR approach. Kates et al. (2001) define the three core objectives of Sustainability Science as: (1) understanding the fundamental interactions between nature and society; (2) guiding these interactions along sustainable trajectories; and (3) promoting social learning necessary to navigate the transition to sustainability. A key characteristic of Sustainability Science is that research is defined by the problems it addresses rather than the discipline(s) it employs ( Kates et al. 2001 ; Clark and Dickson 2003 ; Clark 2007 ; Bettencourt and Kaur 2011 ). Transitioning to sustainability requires socio-technical change in the rules, practices, and norms that guide the development and use of technologies, as well as the social and institutional structures for continual learning and adaptation ( Smith, Stirling and Berkhout 2005 ; Miller et al. 2014 ).

This shift to more engaged, solution-oriented research, is evident at all levels, from heightened interest in TDR in graduate student research ( Willetts and Mitchell 2016 ), university-led grand challenge programmes ( Popowitz and Dorgelo 2018 ), and the United Nations Sustainable Development Goals (SDGs), which recognize the importance of synergies and trade-offs and the need to link physical, social, and natural capital. The key characteristics of use-inspired research are that research is problem-driven (as opposed to science-driven) and complexity-aware (as opposed to reductionist), and that it incorporates multiple sources of knowledge and knowledge co-production processes, supports decision-making (i.e. it does not just provide a technological solution), and pays attention to interactions between society and environment. In this mode, scientists not only generate new knowledge but also act as knowledge brokers and change agents ( Miller, Muñoz-Erickson and Redman 2011 ). Interactive models acknowledge that societal actors other than scientists are important in creating science’s societal impact and the mechanisms and pathways to impact increase manifold, changing the nature of the research questions, data, analyses, interpretations, and outputs. Spaapen and van Drooge (2011) use the concept of productive interactions as ‘exchanges between researchers and stakeholders in which knowledge is produced and valued that is both scientifically robust and socially relevant’ (212). In practical terms, productive interactions in research may mean capacity-building, co-learning, relationship-building, coalition-building, and multiplication of outreach beyond what would be possible in a classic disciplinary research project. In other words, the research process itself may generate benefits as much or even more than its final products . Moreover, the resulting impact on society may manifest at multiple levels simultaneously (e.g. from farmers’ fields and local institutions through to sub-national and national policy formulation and implementation). The key point for the current discussion is that there are different modes of research, and in order for impact assessment to be reliable and useful, we need to be clear about the mode in which we are operating.

We now turn to the example of the FTA CRP for an illustration of how integrated, problem-oriented research has developed in practice. The example will also help consider specific needs, opportunities, challenges, and advances in research evaluation.

3.1 Technology roots

The CGIAR provides a good example of both an evolution (still incomplete) towards a more engaged, transdisciplinary sustainability-science approach, and the attendant evaluation challenges.

CGIAR research began in the early 1970s with a strong technology focus and an emphasis on crop-improvement ( McCalla 2014 ). Major scientific advances in plant physiology, genetics, agronomy, and chemistry, as well as advances in plant breeding technologies, were packaged and delivered to users as improved seed and technology packages, resulting in remarkable increases in food production. However, it was quickly realized that technologies developed through research were often imperfectly suited to the priorities and circumstances of smallholder farmers. Despite overall increased yields, there were substantial gaps between research station potentials and realized yields in farmers’ fields. Field practitioners recognized the need to better understand the constraints faced by smallholder farmers and their decision-making processes as a way to bridge the yield gap. ‘Farming systems research’ developed as a collection of methods for researchers to understand farm households and their decision-making beginning in the 1970s ( Collinson 2000 ; Byerlee, Harrington and Winkelmann 2003 ).

However, even as CGIAR research expanded to a broader range of issues, the early successes of breeding programmes on crop production (especially semi-dwarf rice and wheat) and the impact of the green evolution on food security in India skewed donor interest towards crop breeding. As McCalla (2014) observes, promising systems research programmes were abandoned and converted to commodity-focused centres, and new ecology-oriented centres increasingly adopted commodity mandates as well.

After a relatively slow period of expansion, in the mid-90s, the CGIAR incorporated centres focused on NRM, including forestry [Center for International Forestry Research (CIFOR)], agroforestry [World Agroforestry (ICRAF)], water management (International Water Management Institute), and fisheries (World Fish), and a stronger eco-regional focus was implemented. Still, NRM research in the CGIAR has always had a strong emphasis on maintaining or increasing agricultural productivity and complementing CGIAR genetic improvement research. As Gregerson and Kelley (2007) observed, NRM research in the CGIAR ‘is typically focused on producing knowledge that results in technology options, information and methods/processes that enhance […] the productivity and stability of ecosystem resources’ (13).

3.2 An evolving research model

External reviews of CGIAR social science ( Barrett et al. 2009 ) and NRM research ( ISPC 2012 ) advocated for more Sustainability Science and transdisciplinary approaches, and endorsed exploiting a wider range of interventions and impact pathways, with more emphasis on designing for impact with explicit theories of change (ToCs). Barrett et al. (2009) also explicitly recommended to ‘[f]ocus on impact but end the impact measurement obsession’ (4). They recommended going beyond purely science-based partnerships to engage with government, civil society, stakeholders, and other relevant actors to help ensure that: research questions are relevant to development needs; values and concerns of intended users are represented in the research process; and pathways to impact are actively developed and supported.

An organizational reform process started in 2008 substantially accelerated the move towards engaged, solution-oriented research approaches. The shift was catalysed by an explicit commitment to ‘shared responsibility’ ( ISPC 2015 : 5) for impacts, defined as reduced poverty, improved food and nutrition security, and improved natural resources and ecosystem services ( CGIAR 2016 ). This increased the accountability of research centres and individual researchers to realize social, economic, and environmental outcomes and impacts, on top of the long-established commitment to producing high-quality science.

A key aspect of the reform process was the creation of CRPs. These were intended to facilitate broader and deeper partnerships with other research organizations and with a range of policy and development actors. The focus on outcomes and emphasis on working through partnerships acknowledges that high-quality scientific knowledge creation alone cannot adequately address contemporary sustainable development challenges. The 2017 CGIAR Quality of Research for Development framework supports this approach by shifting from a traditional academic notion of science quality evaluation to a broader concept of research quality (discussed below) that is assessed on its potential and actual contributions to development processes ( ISPC 2017 ).

3.3 Forests, trees and agroforestry consortium research programme

The FTA CRP is one of 15 CRPs within the CGIAR. It is led by CIFOR in partnership with ICRAF, Bioversity, and four non-CGIAR research organizations: Tropical Resources Institute (CATIE), the French Agricultural Research Centre for International Development (CIRAD), International Network for Bamboo and Rattan (INBAR), and Tropenbos ( FTA n.d. ). A large and increasing share of research performed by FTA (and within the CGIAR more generally) uses transdisciplinary approaches, engaging a range of scientists and societal stakeholders to frame the problem, co-produce applicable knowledge, and actively promote integration and application of research-based knowledge in complex systems.

FTA has developed a ToC for the entire programme, composed of five flagship ‘projects’ (FPs) on: (1) tree genetic resources; (2) forests and trees in livelihoods; (3) sustainable forest value chains; landscape dynamics, productivity, and resilience; and (5) climate change adaptation and mitigation (FTA n.d.). Each FP has its own ToC, and many individual projects within the FPs have explicit ToCs documented. At the programme level, key outputs are characterized as knowledge, tools, guidelines, models, and policy recommendations. Some FTA outputs are packaged as technological innovations, but few are truly discrete, stand-alone technologies analogous to an improved crop variety. Rather, most FPs model their work in a systems context and recognize that their efforts interact with multiple external actors and processes. The FPs are expected to work through targeted engagement with various actors in the system, contributing to capacity development of both researchers and research users. Many FTA research projects involve stakeholder engagement in one way or another. Co-generation of knowledge with various partners is intentional and considered a critical component of the research to impact pathway.

This integrated approach is exemplified in the R4D approach developed in FTA’s FP2 on ‘Enhancing how trees and forests contribute to smallholder livelihoods’. Coe, Sinclair and Barrios (2014) argue that sustainable increases in agricultural production and the maintenance of environmental services cannot be achieved simply by developing and promoting specific technologies. Rather, interventions need to adapt to fine-scale variation in social, economic, and ecological contexts. Furthermore, achieving benefits at the farm level will require appropriate service delivery mechanisms, markets, and appropriate institutions, along with technological innovation; all these aspects need to be addressed by research. Finally, the approach explicitly recognizes that scaling will require substantial effort beyond the capacity and reach of any research organization working alone. Coe, Sinclair and Barrios (2014) therefore recommend engaging actively and directly with development and private sector actors, what they call the ‘development praxis’ (73), to enable co-learning and scaling.

As an example, at the project scale, ‘Support to the Development of Agroforestry Concessions in Peru’ (SUCCESS) used stakeholder engagement and multiple impact pathways. SUCCESS aimed to support the implementation of a new tenure mechanism that offers agroforestry concessions (AFCs) to households as a way to realize positive ecological and socio-economic impacts. The project’s multi-actor engagement approach aimed to develop smallholder knowledge and government capacities for AFCs, and build coalitions with key stakeholders to influence the political agenda. As illustrated in the SUCCESS ToC ( Figure 2 ), the project worked to create space for dialogue, develop capacity and co-generate knowledge with small-holder farmers and government agents, and build coalitions among various actors in the system, in combination with more typical research collaborations and research-based knowledge outputs. The main institutional innovation was the AFC, but it was not developed by the research project. Rather, the project set out to help the AFC mechanism work more effectively.

Simplified SUCCESS Project ToC.

The ‘Global Comparative Study on Reducing Emissions from Deforestation and Forest Degradation’ (GCS-REDD+) project focused research on identifying challenges and providing solutions to support the design and implementation of effective, efficient, and equitable policies and projects. The research involved more than 60 research partner organizations in 15 countries. Four main modules aimed to: (1) document and analyse relevant strategies, policies, and measures; (2) assess and learn lessons from sub-national REDD+ implementation (i.e. pilot projects); (3) analyse approaches to setting monitoring and reference levels as a contribution to the design of measurement, reporting, and verification standards; and (4) investigate potential synergies between REDD+ and climate change adaptation approaches. The programme was intended to contribute to improved policy and practice in sub-national REDD+ project implementation and at national and international policy levels. Each module involved a high degree of engagement with a range of partners. Contributions of the project were framed as changes in policy processes and practice, which resulted from both new knowledge and actions taken by various partners and stakeholders. The full ToC and a more elaborated explanation of the project outputs, engagement, and outcomes are presented in Young and Bird (2015) . That evaluation recognized that, while overall REDD+ progress was limited by the international policy environment, there was evidence that the project positively influenced capacity and contributed to the discourse and development of improved systems for implementing REDD+ at international and national scales. Outcomes were achieved through: (1) the production and dissemination of high-quality independent research; (2) the development of approaches and tools that were applied by others; (3) provision of expert support at the international and national levels; (4) the hosting of international events and training; and (5) collaboration with and capacity development of national partners ( Young and Bird 2015 ).

In projects of this kind, the research process itself generates value through partnerships and networking, identifying and defining the problem, methodological development, enhancing capacity, and otherwise influencing the research agenda. Each element can make valuable contributions independently or in combination with the data, analysis, and primary scientific knowledge generation process. As noted in a CGIAR review of NRM research ( ISPC 2012 ), impact can result through: (1) new ways of thinking about land management and production systems change the paradigm of production; (2) decision-making and visualization platforms; (3) partnerships with innovators and entrepreneurs who are best placed to convert research outputs to practical application; and (4) coordinated and widely accessible metadata. Impact pathways involve changing the context and informing and influencing transformative change processes.

3.4 Challenges for learning and impact assessment

This brief overview of the FTA research programme reveals several characteristics that confound learning and impact assessment in this and other integrated sustainability science and TDR programmes.

Spatial scale

The programme works at multiple spatial and temporal scales. Research is being conducted at the scale of genes, farm- and forest-scale management, and sub-national, national, and international policy and trade.

Time scale and time lags

Some work focuses on immediate problem-solving and some deals with long time scales. In many cases, processes initiated and/or supported by FTA research take time to mature and often require changes in the context in order to be fully realized.

Multiple interventions, multiple impact pathways

The research is being done within complex systems, with many other actors and processes operating simultaneously. As discussed above, the research aims to contribute through capacity-building and empowerment of various systems actors, relationship-building, methods development, problem definition, and agenda-setting, as well as science-based knowledge creation. It would be unrealistic to try to isolate pathways, which would miss important contributions.

Co-generated knowledge

Related to the multiple impact pathways, FTA research increasingly engages with, supports, and empowers other actors to do their work.

We have discussed how research has evolved to be more engaged, transdisciplinary, and change-oriented, and likewise how the CGIAR and its research have evolved to be more effective and impactful. But how do we assess its impact? Clearly, if the nature of research changes, so must the way we conceptualize and measure the impact of research.

The term ‘impact’ itself is poorly defined and ambiguously used ( Belcher and Palenberg 2018 ). Most definitions of impact related to research implicitly assume a linear model of impact but focus on different parts of the model. ‘Academic impact’ or ‘research impact’ normally refers to the intellectual contribution made to a field of study within academia. There has also been a strong push to recognize and evidence the ‘societal impact’ of research. Many research funders ask for some indication of potential societal impact in grant applications. Societal impact typically refers to ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ ( REF 2011 : 48). CGIAR donors tend to think of impact in terms of concrete realized benefits in improved human welfare or environmental conditions. When they ask for demonstrations of impact, they are often asking for tangible realized benefits: evidence that the world has been improved in some measurable way. They expect research impact assessment to demonstrate significant contributions to desired development goals and validate large-scale effects at the mission level in terms of reduced poverty, improved food security, and/or improved natural resource conditions ( Raitzer and Winkel 2005 ).

Ex post impact assessment has been presented as critical to satisfy the accountability imperative for a publicly funded institution and essential for continued support by investors to the CGIAR ( Kelley, Ryan and Gregersen 2008 ). There is a perennial promise that, if researchers can prove their impact, funders will provide more unrestricted support. As McCalla (2014) discusses, this promise has never been realized. However, as Raitzer and Winkel (2005) discuss, impact assessment of agricultural research has made little reference to actual demands from any audience group, and there has not been any subsequent systematic assessment of donor needs and expectations regarding impact assessment (Stevenson, personal communication).

Raitzer and Winkel’s (2005) survey reported that donors were primarily interested in demonstrating the contribution of research to development goals and validating large-scale effects at the mission level to justify and defend funding decisions to higher decision-making bodies. Somewhat surprisingly, respondents were not especially interested in trying to attribute credit among collaborating institutions and rather felt it appropriate that relevant contributions and investments should be considered in concert. Indeed, Raitzer and Winkel (2005) report that some respondents felt non-CGIAR influences were insufficiently credited in CGIAR impact studies. Respondents expressed interest both in large-scale estimates of adoption and productivity effects, as well as smaller-scale analysis of detailed effects at the household level.

As discussed above, since 2005, there have been many changes in the CGIAR and in the nature of the research performed by the CGIAR. There has also been a greater learning focus in the evaluation field ( Patton 2008 ). Indeed, considering the complexity and multiple impact pathways of R4D and TDR more generally, it is clear that there is a pressing need to learn what works (and what does not work) and how to facilitate continuous improvement.

4.1 Impact assessment in the CGIAR

The current predominant approach to impact assessment in the CGIAR uses a counterfactual framework with experimental or quasi-experimental methods ( Stevenson, Macours and Gollin 2018b ). In the former, units (e.g. individuals, households, villages) are randomly assigned to different ‘treatment conditions’ and the resulting treatment groups are compared statistically, typically against pre-specified outcome or impact indicators. If random assignment is not feasible or appropriate, quasi-experimental approaches use various alternative strategies (e.g. difference-in-differences, regression discontinuity, and propensity score matching) to control for selection bias in the comparison of treated and untreated units ( Khandker, Koolwal and Samad 2010 ). Both approaches aim to estimate statistically aggregated effects of a hypothesized cause (e.g. an intervention or improved crop variety). There has been a long and storied debate about the pros and cons of randomized control trials (RCTs), as well as quasi-experimental and other quantitative impact assessment approaches ( Donaldson 2009 ; Frieden 2017 ; Deaton and Cartwright 2018 ). We do not intend to enter that debate. We fully appreciate that, used appropriately, experimental and quasi-experimental impact assessment methods can and have contributed substantially to empirical impact assessment work. However, these methods are insufficient on their own, and often inappropriate in the context of problem-oriented TDR.

First, as already discussed, TDR research is emergent and not discrete. Experimental and quasi-experimental approaches are only appropriate when there is a well-specified and discrete treatment. If the treatment itself is multi-pronged, evolving (e.g. if the research programme is co-generating knowledge, enabling and supporting various stakeholders and influencing the policy discourse), and/or under-specified (e.g. if the full range of influences is emergent and not fully known), there will be uncertainty as to which variation of the emergent research-informed innovation is responsible and for which specific effects. This limits scope for learning and for determining what should be scaled out further ( Veerman and van Yperen 2007 ). Each TDR case is unique, and we need nuanced understanding of context to be able to understand processes of change.

Second, and more fundamental, experimental and quasi-experimental approaches are only appropriate for ‘large n’ interventions; that is, interventions that target sufficiently large numbers of units where it is possible—at least theoretically—to assign units (ideally at random) to varying treatment conditions ( White 2010 ). Outside of laboratory or research station settings, the number of units that need to be assigned to such conditions is typically large. This is due to the inevitable increase in ‘statistical noise’ resulting from both the greater heterogeneity among such units and an inability to isolate them from extraneous factors that additionally influence the status of the outcome and impact indicators of interest. As discussed above, while policy-oriented research programmes may aim to generate benefits for many units (e.g. farming households), they typically work through a small number of units (e.g. a national climate policy; a regional land use planning framework; or an agri-food system) as a way of inducing such ‘large n’ change. They are ‘small n’ interventions, where generating statistically aggregated ‘treatment effect estimates’ is implausible and inappropriate. As we elaborate below, a more appropriate approach is needed to analyse and evidence the extent to which the ‘small n’ unit in question has changed and the likely factors responsible for that change.

Moreover, interactions are expected between targeted ‘small n’ outcomes (e.g. the mitigation of policy constraints) and ‘large n’ impacts (e.g. smallholder farmer income), thereby potentially violating what economists call the stable unit treatment value assumption (SUTVA) ( Rubin 2005 ). SUTVA is violated and, by extension, the ability to precisely estimate the counterfactual when there are spill-over effects between treated and untreated groups or equilibrium effects (e.g. water pollution) affecting both.

Finally, an inherent feature of TDR is the co-generation of new knowledge, learning, and/or improved capacities together with intended research users and other stakeholders. It follows that any resulting benefits for society and/or the environment can neither be solely attributed to the researcher, nor the research investment in question.

These same limitations may also apply to discrete research-informed innovations, such as improved crop varieties. Here too, greater emphasis is being placed on stakeholder engagement in research and technology development processes, such as farmer participatory varietal selection (see Joshi and Witcombe 1996 ). In this kind of research, technological innovation itself may be just one of many factors responsible for any realized benefits. This is nicely illustrated by Bulte et al. (2014) who conducted both traditional and double-blind RCTs of an intervention promoting improved cowpea varieties in Tanzania. The traditional RCT estimated a 27% average gain in yields over traditional varieties. However, the double-blind RCT estimated that two-thirds of the average increase was due to a placebo effect. That is, the farmers in the traditional RCT were told that they were receiving improved chickpea varieties and consequently invested relatively more in the management and care of the ‘improved’ crops.

Indeed, there is concern that the current bias towards quantitative approaches, most notably RCTs, is steering the evaluation agenda—and the research agenda—away from potentially impactful interventions that cannot easily be randomized ( Deaton 2010 ). Again, the CGIAR provides a good example. In NRM research, there are difficult challenges in isolating lines of causality, attributing impacts to particular interventions, estimating meaningful counterfactuals, and establishing quantitative measures ( Kelley, Ryan and Gregersen 2008 ). As a result, several observers have lamented the lack of evidence of NRM impact. Renkow and Byerlee (2010) found that NRM research had not shown the same returns on investment as crop genetic improvement research. They further suggested that investments in NRM research in the CGIAR should be reduced relative to crop genetic improvement. Renkow and Byerlee (2010) reviewed the impact of policy-oriented research in the CGIAR and found that analyses tended to be confined to documenting impact pathways as opposed to measuring specific impacts. They acknowledge ‘NRM work typically deals with systems, rather than components to a greater degree than other types of CGIAR research’ (397), which they suggest increases the location specificity of NRM research and probably limits the international public good dimensions. In fact, NRM research conducted to understand processes and interactions in systems and extrapolation domains has the same potential to yield global public goods as crop improvement research. When NRM processes are understood, extrapolation is possible.

A World Bank meta-evaluation of the CGIAR ( Lesser 2003 ) also found that NRM research was under-evaluated and another review highlighted the lack of evidence of contributions of NRM research to impact at scale ( ISPC 2012 ). The ISPC (2012) review recommended that it is ‘necessary and legitimate to pursue research […] to develop new methods for impact assessment that recognize the contributions of NRM research’ (6). A series of studies on NRM research outcomes between 2013 and 2016 noted low rates of adoption of NRM technologies and practices and a general recognition of the lack of a clear and compelling vision for NRM research as a whole ( Stevenson and Vlek 2018 ).

It is notable that many of these previous efforts followed a technology adoption model. Renkow (2010) posits a simplistic two-phase impact pathway. In the first phase, research outputs combine with political inputs to produce policy outcomes such as new laws, regulations, and institutions (with the implicit assumption that simply making new knowledge available will trigger policy change); in the second phase, those policy outcomes produce welfare changes (i.e. impacts). Moreover, the NRM research examples reviewed in the Standing Panel on Impact Assessment (SPIA) case studies were all technology-based interventions ( Stevenson and Vlek 2018 ). Stevenson, Macours and Gollin’s (2018b ) report, ‘The Rigor Revolution in Impact Assessment: Implications for CGIAR’, also focuses predominantly on assessing technological innovation. The report acknowledges that there are multiple impact pathways for CGIAR research, and it offers brief discussion of the need for methodological pluralism. However, the second main point in the conclusion identifies the need for work in this area:

‘impact evaluation and efficacy studies need to focus on causal relationships for which we have the greatest uncertainty and for which information would have the highest value. This suggests a greater focus on theory—away from searching for ‘what works’ in the abstract and towards finding out why certain things work and others do not in particular contexts […] It is less obvious how to make methodological breakthroughs on tracing policy influence or measuring the outcomes from capacity-building efforts, though the principle of independent theory-based evaluation should be prominent’ ( Stevenson, Macours and Gollin 2018b : 29).

In another discussion on estimating impacts of R4D, the authors state: ‘[e]ven if one starts from the viewpoint that contributions to science and to capacity from the research process itself will be excluded, the challenges associated with estimating benefits from a research-based technology or other innovation are enormous’ ( Stevenson, Macours and Gollin 2018a : 3). They go on to point out that it is important to recognize these other kinds of contributions, but do not explore this important question further.

5.1 Principles for evaluating multi-faceted TDR initiatives intervening in complex systems

How then do we conceptualize and assess the impact of emergent, stakeholder-informed, and systems-focused research? We assume that few would disagree that problem-oriented TDR has potential to significantly contribute to solving some of the world’s most pressing challenges, and, in turn, positively impact society and the environment. We also assume general agreement that research funders and the public more generally have a right to reliable feedback on what this contribution is and how it can be strengthened. Therefore, unless we want to restrict research to narrowly focus on developing and improving ‘large n’ technologies and move away from seeking to influence change in complex systems, expectations for what counts as credible evidence needs to be considerably broadened. The inevitable complexity and limitations also need to be explicitly recognized. We propose a set of evaluation principles to guide the evaluation of TDR programmes.

Use a portfolio of methods. A multi-scale, transdisciplinary research programme will generate multiple kinds of outputs and pursue multiple impact pathways. It is important to assess key elements using the most suitable methods. A portfolio of multi-method and multi-level evaluative work is needed for most TDR programmes. Theory-based qualitative approaches (discussed below) can be used to assess outcomes of interactive work with stakeholders seeking to improve policy and/or practice. Some research-based innovations will be suitable for testing using conventional ‘large n’ impact assessment approaches. A policy implementation assessment may be useful to understand the extent to which the co-generated policy recommendations were implemented. Ex ante modelling approaches can further help estimate the range of likely impacts associated with such implementation ( Kelley, Ryan and Gregersen 2008 ). A programme should aim to build a broad portfolio of evaluative evidence over time.

Focus on central aspects (key ‘nodes’) of the overarching theory of change. The portfolio of evaluative inquiry should cover the primary components or ‘nodes’ of the overall (project or programme) ToC. The ToC nodes may relate to specific kinds of outcomes (e.g. changes in stakeholder capacity or aspirations) and actual impacts on the ground (e.g. reduced deforestation, improved household income). They also may encompass change at different scales and/or be structured around defined research areas. Given that these different components of the research programme are expected to work together to contribute to systems-level change, evaluative efforts should include a focus on interactions.

Aim for representativeness. The overall set of assessments should aim to identify and select sets of representative cases and contexts as a basis for learning from the range of experience and for extrapolation of results.

Prioritize quality over quantity. It is better to have fewer high-quality evaluations than numerous evaluations of dubious quality. As argued by White and Phillips (2012) , qualitative ‘small n’ evaluations need to be implemented following rigorous qualitative research protocols to be able to successfully address issues of cause and effect. While there is debate on the relevance of independence in evaluation ( Picciotto 2013 ), it is critical that the evaluation team be as impartial as possible, mitigate potential sources of bias in data collection, and ensure the veracity of key findings (e.g. via triangulation and/or the identification of ‘signatures’). Attention will also be needed to improve the application and specification of theoretical assumptions in research project design and within evaluations. This will help frame sets of evaluations (i.e. series of case studies) to improve their explanatory power. Careful and transparent evaluation design, data collection, analysis, and reporting, along with improved peer-review and other quality control efforts, can help bolster credibility.

Build evaluability and evaluation into research projects where possible. Research projects can be designed to facilitate evaluation and generate evidence for impact assessment. For example, explicitly documenting a project/programme ToC at inception supports strategic planning and, not incidentally, provides a framework for monitoring, data collection, and outcome evaluation. Experimentally testing the impact of policy or technology options at the project scale, as part of the research process, can generate evidence to estimate impact when such options are implemented at scale, bearing in mind the inherent external validity considerations. Participatory policy option development with stakeholders can help explicate underlying assumptions, which can be used in ex ante impact assessment. The more creatively research can fulfil its own objectives, while generating evidence of its own impact, the better.

Look for unintended consequences. Intervening in complex systems is likely to generate unforeseen outcomes and impacts, both positive and negative. Support provided to a regional government, for example, to strengthen its land use planning and management system may negatively impact a particular user group in ways unforeseen in the research engagement’s earlier phases. Evaluations of specific aspects of the research programme and those examining interconnections should deliberately seek to identify unintended consequences and investigate them when they are found.

Facilitate and document learning for enhanced research effectiveness. As indicated above, evaluating a complex TDR programme is unlikely to be a one-off exercise. Rather, several sources of evaluative evidence, together with relevant impact-related research evidence, will be combined to paint an overall picture. Each of these evaluation and relevant research pieces should include (ideally co-generated) recommendations for strengthening research impact. These recommendations should then be taken on board, formally documented, and ideally highlighted in one or more overarching evaluative pieces. Research programmes that learn from failure and continuously seek to strengthen the modalities through which they generate societal impact are intrinsically valuable and should be attractive from an accountability perspective as well.

Acknowledge (and embrace) the inherent limitations . Outcomes and impacts may be realized at multiple levels, likely with varying time-lags and uncertain impact trajectories. Many potential benefits may be difficult to anticipate a priori , and some may not be traceable at all. High-level outcomes and impacts in complex systems are beyond the control or even influence of most research programmes. Change processes can be modelled and monitored, with evidence of contributions to the process, but high-level impacts can rarely be attributed directly to the research. Therefore, expectations for precise and comprehensive impact measurement should be tempered.

Figure 3 illustrates a stylized set of TDR impact pathways. A research project/programme aims to create new knowledge and/or innovations to address development problems, but the process can (and in TDR, is designed to) also make substantial contributions in problem framing, methodology, collection and sharing of data, etc. This work, up to and including outputs, is within the sphere control of the project ( Hearn 2010 ). Each of these processes and outputs is expected to support, encourage, or otherwise influence other actors by changing knowledge, attitudes, skills, and/or relationships, resulting in changes in policy and/or practice (sphere of influence) that ultimately contribute to impacts in terms of human and/or environmental social condition. Here we set out the elements of an impact assessment framework for a TDR programme.

Research-to-Impact Pathways.

6.1 Challenge framing

Integrated TDR programmes, like FTA, are particularly challenging to evaluate. Not only does the programme embody the characteristics of problem-oriented TDR described above, it comprises five distinct research areas through its FPs. Each FP includes multiple projects, most of which are funded bilaterally, with donor priorities inevitably constraining strategic alignment. FTA is effectively an umbrella for several relatively distinct pathways, rather than a single initiative. The same is true of Grand Challenge Programs and other large inter- and transdisciplinary programmes. Nevertheless, these programmes are expected to ultimately contribute to mission-level development impacts. In FTA’s case, these are specified as global targets in terms of reduced deforestation and forest degradation; reduced rural poverty and inequality; reduced loss of biodiversity and ecosystem services; and improved land-use governance and management. Evidence is needed to show that the programme has contributed to these targets.

6.2 Nested ToCs

A fundamental assumption of an integrated programme is that its individual components are strategic and coordinated in a way that will contribute collectively and significantly to high-level outcomes and impacts. Planning and evaluation can be greatly facilitated by developing and documenting the ToC for the overall programme to model the key actors, activities, outputs, outcomes and intended impacts. A ToC at the programme scale needs to encompass the full range of these different elements, so detail is necessarily limited. It is, therefore, useful to have more detailed ToCs at sub-programme scale (e.g. sets of closely-linked projects) and project scale. At the project scale, it is possible to precisely specify outcomes to guide planning and evaluation (discussed below). The aim is to develop a systematic, integrated, and nested ToC and evaluation framework as a base for other evaluative work.

6.3 Research quality appraisals

Research quality can be assessed within the sphere of control. Quality is broadly defined to include characteristics of design and implementation that will achieve outputs that are relevant, credible, legitimate, and effective ( ISPC 2017 ). Relevance refers to the appropriateness of the problem positioning, objectives, and approach to the research for intended users. Credibility pertains to rigour of the design and research process to produce dependable and defensible conclusions. Legitimacy refers to the perceived fairness and representativeness of the research process. Effectiveness refers to the utility and actionability of the research’s knowledge and social process contributions. Belcher et al. (2016) provide a set of criteria for TDR evaluation, organized within these four principles.

6.4 Ex ante impact estimation

A ToC provides a good base for ex ante impact assessment to estimate potential impacts. Ex ante impact assessment can be done at any scale, with trade-offs between precision and scope. Ultimately, information about mission-level impact for each of the main challenges is needed. Such estimates will use evidence and information from a combination of methods (discussed below) to estimate plausible ranges of FTA’s impact vis-à-vis the above targets, as well as other potential impacts, including possible negative impacts. Ex ante assessment will necessarily require application of theory and assumptions. Making the theory and assumptions transparent and therefore open to challenge and empirical testing is indeed one of the main benefits of the process. Ideally, sensitivity analysis should be done to assess the influence of key assumptions on impact estimates.

6.5 Theory-based outcome evaluation

Much of the FTA research portfolio aims to exploit multiple processes in complex systems. Here we need project-scale theory-based approaches to test whether and how research has contributed to a change process. Theory-based approaches are well suited to ‘within-case’ research and involve analysing ‘the causal links that connect independent variables and outcomes, by identifying the intervening causal processes, that is, the causal chain and causal mechanisms linking them’ ( Reilly 2010 : 734). A key aspect of this approach involves ruling out alternative explanations for an observed event or change and identifying indicators or ‘signatures’ ( Mohr 1999 : 72) that, if they occur, provide support for a hypothesized cause. Such approaches work through affirming explanations that are consistent with the facts and rejecting those that are not.

The theory can be derived inductively, working backwards from observed changes and developing explanations. This is the approach used in methods such as Realistic Evaluation ( Pawson and Tilley 1997 ; Maxwell 2004 ; George and Bennett 2007 ) and Process Tracing ( Collier 2011 ). In these approaches, the evaluator develops theories about how the project works to generate particular effects. Emphasis is placed on understanding the nature of the project and its operation. Theories about the mechanisms and circumstances by which the project or programme has contributed to effects for specific subgroups in particular contexts are then iteratively developed and tested.

The ToC can also be developed ex ante , as part of a project or programme’s design, as a set of hypotheses about what outcomes and impacts are intended or expected to result in a particular case. The hypotheses can then be tested deductively using empirical evidence from the completed project. This approach is used in methods such as Outcome Mapping ( Earl, Carden and Smutylo 2001 ), Payback Framework ( Buxton and Hanney 1996 ), and Contribution Analysis ( Mayne 2001 , 2012 ).

FTA has adopted and refined a theory-based case-study approach specifically for assessing the outcomes of research at the project scale (see Belcher, Davel and Claus 2020 ). The method uses a detailed project ToC as the analytical framework ( Weiss 1997 ; Coryn et al. 2011 ; Vogel 2012 ; Belcher, Davel and Claus 2020 ). The ToC models the change process, providing description and explanation of both how and why the project is expected to cause or contribute to a result or a set of results (i.e. outputs, outcomes, and impacts). It details the primary actors, steps, and pathways in the change process and specifies the theoretical reasons for the changes. A well-specified ToC is essentially a set of hypotheses about each step in the change process that can be tested empirically using data from document review, surveys, and interviews with key informants to assess actual outcomes against expected outcomes at each stage in the ToC. In lieu of a reliable counterfactual, it is important to consider and test competing hypotheses for how a change may have happened, and leave room to include additional elements and alternative explanations for how the project or programme may or may not have affected the outcome in question based on ex post analysis ( Rossi and Freeman 1989 ; Donaldson 2009 ; White 2009 ; Hitchcock 2018 ).

Outcome evaluations provide evidence of the scope and scale of qualitative changes and change processes in the overall effort to address programme challenges. They answer the question: who is doing what differently as a result of the research?

6.6 Experimental/quasi-experimental IA

As explained above, experimental and quasi-experimental approaches are useful for assessing the effectiveness of ‘large n’ innovations. Some research outputs will be delivered as distinct technologies or institutional innovations applied over a large number of units (e.g. smallholder farming households). In these cases, experimental and quasi-experimental impact assessment can be used.

As much as possible and where relevant and feasible, such impact assessment work should be incorporated as part of the research process itself. That is, if the research involves the use of a discrete intervention in multiple iterations, it may be possible to randomly assign the treatment and compare both before and after and with and without the treatment (experimental). If true random assignment is not feasible, quasi-experimental approaches, such as those mentioned above, can potentially be used to control for selection bias in the comparison of treated and untreated units.

For example, a research project is currently underway to experimentally compare alternative extension approaches as part of an effort to enhance the implementation of Peru’s AFC policy instrument (discussed above). While this pilot will focus on evaluating the effects of the extension approaches on the uptake of sustainable land management practices, efforts will also be made to estimate the impacts on farming household income and deforestation. In other cases, as necessary and appropriate, stand-alone quasi-experimental IA will be needed as part of the overall impact evidencing strategy. The resulting evidence can then be used to estimate impact when the innovation in question is promoted at scale, following relevant changes in policy and practice.

6.7 High-level indicators and measures

If we can demonstrate that the research has been successful at stimulating or contributing to change within the sphere of influence, it is reasonable to expect further knock-on changes. That is, if key actors act differently as a result or partially as a result of the research project, that may contribute to further changes that will help transform systems and ultimately lead to social, economic, and environmental benefits.

A key element of the Outcome Evaluation approach is the explicit identification of end-of-project outcomes, defined as outcomes that would be ambitious but reasonable to expect and to observe within the time-frame and resources of the project being evaluated ( Belcher, Davel and Claus 2020 ). End-of-project outcomes can be assessed empirically.

Higher-level (i.e. beyond end-of-project) outcomes and impacts are also modelled in the ToCs, and in ex ante impact assessments, to illustrate and explain the causal logic to mission-level impact. However, these kinds of changes are typically well outside the sphere of control and sphere of influence of a research project or programme. Changes in the sphere of interest may be indicated by United Nations SDG indicators and other secondary data on development and conservation status, but it is not possible to attribute those changes to research because there are so many other factors that affect their status.

6.8 Aggregating up

The overall strategy needs to identify and focus on the key nodes (Principle 2) and use appropriate, rigorous methods (Principles 1, 4, and 6) to evaluate representative sets of projects/contexts (Principle 3). Within the sphere of control, the focus needs to be on research quality, with quality broadly defined to include characteristics of design and implementation that will achieve outputs that are relevant, credible, legitimate, and effective ( ISPC 2017 ). Within the sphere of influence, we need to know whether and how a research programme (including knowledge creation and supporting activities) is actually encouraging, supporting, or otherwise influencing key actors in the system to make positive changes in policy and practice. In the sphere of interest, we can model impact pathways and estimate potential reach and significance, and monitor secondary data on social, economic, and environmental benefits. However, in complex systems, with multiple actors, processes, and time-lags, it is theoretically impossible to make definitive attribution claims (Principle 8). Instead, a plausible case can be made that research has contributed if we can demonstrate that: (1) there is a strong theoretical basis to expect that the research programme will contribute to high-level outcomes and impacts; (2) the research has produced good quality (relevant, credible, legitimate, and effective) outputs; and (3) expected intermediate and end-of-project outcomes (e.g. changes in policy and practice) have been realized.

We have explored the evolution of engaged TDR approaches and the inherent challenges of measuring (or estimating) the impact of TDR and R4D more broadly. Our central argument is that multi-faceted research initiatives seeking to intervene and bring about positive change in complex systems cannot be treated like discrete ‘large n’ interventions. TDR programmes are dynamic and have high potential to target multiple aspects of a problem or issue simultaneously. Relying on conventional quantitative impact assessment approaches alone, or even as the primary mode of evaluative inquiry, is therefore inappropriate. There are other methods that are suitable for interrogating cause and effect relationships in key areas of such research, most notably mechanism or explanatory approaches.

Inevitably, however, there are inherent limitations in measuring TDR impact; this needs to be recognized and accepted. Above all, we need to avoid a potentially perverse outcome where the ambitions of researchers, development practitioners, policy-makers, and others aiming to achieve transformational change are stifled by the results agenda. Contemporary and urgent global challenges are too vast and complex for piecemeal solutions.

However, we cannot use the inherent measurement challenges as an excuse to not evaluate, learn, or be held accountable. We therefore advocate for a holistic, multi-method, and integrated approach; one that is appropriate for the nature of a TDR programme, and one that does not rely on any single evaluation method or static framework. To this end, we offer eight intuitive principles to guide the development of evaluation strategies for larger multi-faceted TDR programmes and present a framework for an integrated TDR programme. We hope that these prove useful to not only researchers, research managers, and evaluators but also research investors.

Theory-based evaluation approaches are still being developed and, especially in the R4D context, there is considerable scope for further methodological improvement. As we gain experience with the methods, it will be possible to streamline and improve efficiency and lower costs. As noted above, increased emphasis on explicit ToC development and documentation as part of research project and programme design, along with improved monitoring, will reduce the costs and increase the rigour of ex post evaluations. Improved application and specification of theoretical assumptions in project design and evaluations will help frame sets of evaluations (i.e. series of case studies) to improve explanatory power. Careful and transparent evaluation design, data collection, analysis, and reporting, along with improved peer-review and other quality control efforts can help address credibility concerns. Integration of theory-based evaluation within a portfolio of methods, taking advantage of the strengths and compensating for the weaknesses of each approach, can help strengthen both accountability and learning in the research programme. This will involve: nested ToCs (project and programme scales); research quality appraisals to check that the research is addressing the ‘right’ issues in the ‘right’ ways; experimental and quasi-experimental impact assessments to both support and evidence the effectiveness of ‘large n’ innovations addressing specific dimensions of the challenge in question; project and programme level theory-based outcome evaluations to test ToCs and assess contributions to proximate outcomes; and modelling (including ex ante impact assessment) and extrapolation to estimate mission-level impacts.

This article was developed from a paper presented at a meeting of the Independent Steering Committee of the Forests, Trees and Agroforestry (FTA) CGIAR Research Program. Anne-Marie Izac, Vincent Gitz, and Florencia Montagnini provided helpful feedback on that paper. The current version was improved based on insightful comments and suggestions from Keith Child and Nancy Johnson. Rachel Davel provided editing support to the paper and manuscript. We thank the anonymous reviewers for their helpful feedback to improve the manuscript.

This work was supported by funding from the Canada Research Chairs programme and the Canadian Social Sciences and Humanities Research Council (SSHRC).

Conflict of interest statement . None declared.

Andrews M. , Pritchett L. , Woolcock M. ( 2013 ) ‘ Escaping Capability Traps through Problem Driven Iterative Adaptation (PDIA) ’, World Development , 51 , 234 – 44 .

Google Scholar

Barrett C. B. et al.  ( 2009 ) Stripe Review of Social Sciences in the CGIAR. Rome : CGIAR Science Council Secretariat . < http://barrett.dyson.cornell.edu/files/papers/SocialScienceStripeReview_Aug2009.pdf >

Google Preview

Belcher B. M. ( 2020 ). Research for Changemaking: Concepts and Lessons for Research Effectiveness. Keynote Presentation to Canadian Changemaker Education Research Forum . Toronto , Canada . < https://researcheffectiveness.ca/wp-content/uploads/sites/7/2020/01/Belcher-C-CERF-Keynote-Jan-15-2020.pptx > accessed 15 Jan 2020.

Belcher B. M. et al.  ( 2016 ) ‘ Defining and Assessing Research Quality in a Transdisciplinary Context ’, Research Evaluation , 25 , 1 – 17 .

Belcher B. M. , Davel R. , Claus R. ( 2020 ) ‘ A Refined Method for Theory-Based Evaluation of the Social Impacts of Research ’, MethodsX , 7 , 100788 .

Belcher B. , Palenberg M. ( 2018 ) ‘ Outcomes and Impacts of Development Interventions: Toward Conceptual Clarity ’, American Journal of Evaluation , 39 , 478 – 95 .

Bettencourt L. M. A. , Kaur J. ( 2011 ) ‘ Evolution and Structure of Sustainability Science ’, PNAS , 108 , 19540 – 5 .

Bill and Melinda Gates Foundation (n.d.) ‘About Grand Challenges’, < https://gcgh.grandchallenges.org/about > accessed March 2020.

Brandt P. et al.  ( 2013 ) ‘ A Review of Transdisciplinary Research in Sustainability Science ’, Ecological Economics , 92 , 1 – 15 .

Bulte E. et al.  ( 2014 ) ‘ Behavioral Responses and the Impact of New Agricultural Technologies: Evidence from a Double-Blind Field Experiment in Tanzania ’, American Journal of Agricultural Economics , 96 , 813 – 30 .

Buxton M. , Hanney S. ( 1996 ) ‘ How Can Payback from Health Services Research Be Assessed? ’, Journal of Health Service Research and Policy , 1 , 35 – 43 .

Byerlee D. , Harrington L. , Winkelmann D. L. ( 2003 ) ‘ Farming Systems Research: Issues in Research Strategy and Technology Design ’, American Journal of Agricultural Economics , 64 , 897 – 904 .

Carew A. L. , Wickson F. ( 2010 ) ‘ The TD Wheel: A Heuristic to Shape, Support and Evaluate Transdisciplinary Research ’, Futures , 42 , 1146 – 55 .

CGIAR ( 2016 ) ‘CGIAR Strategy and Results Framework 2016-2030: Overview’, < https://cgspace.cgiar.org/handle/10947/3865 > accessed March 2020.

Clark W. C. ( 2007 ) ‘ Sustainability Science: A Room of Its Own ’, PNAS , 104 , 1737 – 8 .

Clark W. C. , Dickson N. M. ( 2003 ) ‘ Sustainability Science: The Emerging Research Program ’, PNAS , 100 , 8059 – 61 .

Coe R. , Sinclair F. , Barrios E. ( 2014 ) ‘ Scaling up Agroforestry Requires Research ‘In’ Rather than ‘For’ Development ’, Current Opinion in Environmental Sustainability , 6 , 73 – 7 .

Collier D. ( 2011 ) ‘ Understanding Process Tracing ’, Political Science & Politics , 44 , 823 – 30 .

Collinson M. P. (ed.) ( 2000 ) A History of Farming Systems Research . Oxford : CABI Publishing .

Coryn C. L. S. et al.  ( 2011 ) ‘ A Systematic Review of Theory-Driven Evaluation Practice from 1990 to 2009 ’, American Journal of Evaluation , 32 , 199 – 226 .

Deaton A. ( 2010 ) ‘ Instruments, Randomization, and Learning about Development ’, Journal of Economic Literature , 48 , 424 – 55 .

Deaton A. , Cartwright N. ( 2018 ) ‘ Understanding and Misunderstanding Randomized Controlled Trials ’, Social Science & Medicine , 210 , 2 – 21 .

Donaldson S. I. ( 2009 ) ‘In Search of the Blueprint for an Evidence-Based Global Society’, in Donaldson S. I. , Christie C. A. , Mark M. M. (eds.) What Counts as Credible Evidence in Applied Research and Evaluation Practice? pp. 1 – 18 . London : SAGE Publications .

Earl S. , Carden F. , Smutylo T. ( 2001 ) Outcome Mapping: Building Learning and Reflection into Development Programs . Ottawa : International Development Research Centre .

Frieden T. R. ( 2017 ) ‘ Evidence for Health Decision Making—beyond Randomized, Controlled Trials ’, New England Journal of Medicine , 377 , 465 – 75 .

FTA (n.d.) ‘What is FTA?’, < http://www.foreststreesagroforestry.org/what-is-fta/ >

Functowicz S. O. , Ravetz J. R. ( 1993 ) ‘ Science for the Post-Normal Age ’, Futures , 25 , 739 – 55 .

George A. L. , Bennett A. ( 2007 ) Case Studies and Theory Development in the Social Sciences . Cambridge, MA : MIT Press .

Gibbons M. et al.  ( 1994 ) The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies . London : SAGE Publications .

Godin B. ( 2006 ) ‘ The Linear Model of Innovation: The Historical Construction of an Analytical Framework ’, Science, Technology, & Human Values , 31 , 639 – 67 .

Greenhalgh T. et al.  ( 2016 ) ‘ Research Impact: A Narrative Review ’, BMC Medicine , 14 , 78 .

Gregerson H. , Kelley T. ( 2007 ) ‘The History of Natural Resource Management Research in the Cgiar’, in Waibel H. and Zilberman D. (eds.) International Research on Natural Resource Management: Advances in Impact Assessment , pp. 12 – 20 . Oxford : CABI Publishing .

Hall A. et al.  ( 2000 ) ‘ New Agendas for Agricultural Research in Developing Countries: Policy Analysis and Institutional Implications ’, Knowledge, Technology & Policy , 13 , 70 – 91 .

Harachi T. W. et al.  ( 1999 ) ‘ Opening the Black Box: Using Process Evaluation Measures to Assess Implementation and Theory Building ’, American Journal of Community Psychology , 27 , 711 – 31 .

Hearn S. ( 2010 ) ‘Outcome Mapping: Planning, Monitoring and Evaluation. Outcome Mapping Learning Community’, < https://www.slideshare.net/sihearn/introduction-to-outcome-mapping > accessed March 2020.

Heinrichs H. et al.  (eds.) ( 2016 ) Sustainability Science: An Introduction . Dordrecht : Springer .

Hirsch Hadorn G. et al.  ( 2006 ) ‘ Implications of Transdisciplinarity for Sustainability Research ’, Ecological Economics , 60 , 119 – 28 .

Hirsch Hadorn G. , Hoffmann-Riem H. , Biber-Klemm S. , Grossenbacher-Mansuy W. , Joye D. , Pohl C. , Wiesmann U. , Zemp E. (eds.) ( 2008 ) Handbook of Transdisciplinary Research . New York : Springer .

Hitchcock C. ( 2018 ) ‘Probabilistic Causation’, in Zalta E. (ed.) The Stanford Encyclopedia of Philosophy . < https://plato.stanford.edu/entries/causation-probabilistic/ >

Howaldt J. ( 2019 ) ‘ New Pathways to Social Change–Creating Impact through Social Innovation Research ’, Journal for Research and Technology Policy Evaluation , 48 , 37 – 48 .

Independent Science and Partnership Council (ISPC) ( 2012 ) A Stripe Review of Natural Resources Management Research in the CGIAR . Rome : ISPC Secretariat . < https://ispc.cgiar.org/sites/default/files/ISPC_StrategyTrends_NRM_StripeReview_0.pdf >

ISPC ( 2015 ) Strategic Study of Good Practice in AR4D Partnership . Rome : ISPC . < https://ispc.cgiar.org/sites/default/files/ISPC_StrategicStudy_Partnerships.pdf >

ISPC ( 2017 ) Quality of Research for Development in the CGIAR Context , Brief N.62. Rome : ISPC . < https://ispc.cgiar.org/sites/default/files/pdf/ispc_brief_62_qord.pdf >

Jahn T. , Bergmann M. , Keil F. ( 2012 ) ‘ Transdisciplinarity: Between Mainstreaming and Marginalization ’, Ecological Economics , 79 , 1 – 10 .

Joshi A. , Witcombe J. R. ( 1996 ) ‘ Farmer Participatory Crop Improvement, II Participatory Varietal Selection, a Case Study in India ’, Experimental Agricultura , 32 , 461 – 77 .

Kasemir B. , Jaeger C. C. , Jäger J. ( 2003 ). ‘Citizen Participation in Sustainability Assessments’, in Clark W. C. , Wokaun A. (eds.) Public Participation in Sustainability Science: A Handbook , pp. 3 – 36 . Cambridge, UK : Cambridge University Press .

Kates R. ( 2017 ) ‘Sustainability Science’, in Richardson D. , Castree N. , Goodchild M. F. , Kobayashi A. , Liu W. , Marston R. A. (eds.) The International Encyclopedia of Geography: People, the Earth, Environment and Technology , pp. 1 – 4 . Boston, MA : Wiley-Blackwell .

Kates R. et al.  ( 2001 ) ‘ Sustainability Science ’, Science , 292 , 641 – 2 .

Kauffman J. , Arico S. ( 2014 ) ‘ New Directions in Sustainability Science: Promoting Integration and Cooperation ’, Sustainability Science , 9 , 413 – 8 .

Kelley T. , Ryan J. , Gregersen H. ( 2008 ) ‘ Enhancing Ex Post Impact Assessment of Agricultural Research: The CGIAR Experience ’, Research Evaluation , 17 , 201 – 12 .

Khandker S. R. , Koolwal G. B. , Samad H. A. ( 2010 ) Handbook on Impact Evaluation: Qualitative Methods and Practices . Washington, DC : The World Bank .

Klein J. T. ( 2006 ) ‘ Afterward: The Emergent Literature on Interdisciplinary and Transdisciplinary Research Evaluation ’, Research Evaluation , 15 , 75 – 80 .

Komiyama H. , Takeuchi K. ( 2006 ) ‘ Sustainability Science: Building a New Discipline ’, Sustainability Science , 1 , 1 – 6 .

Lang D. J. et al.  ( 2012 ) ‘ Transdisciplinary Research in Sustainability Science: Practice, Principles, and Challenges ’, Sustainability Science , 7 , 25 – 43 .

Lesser W. H. ( 2003 ) ‘The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research’, in Thematic Working Paper: Review of Biotechnology, Genetic Resource, and Intellectual Property Rights Programs . Washington, DC : The World Bank .

Maxwell J. A. ( 2004 ) ‘ Using Qualitative Methods for Causal Explanation ’, Field Methods , 16 , 243 – 64 .

Mayne J. ( 2001 ) ‘ Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly ’, Canadian Journal of Program Evaluation , 16 , 1 – 24 .

Mayne J. ( 2012 ) ‘ Contribution Analysis: Coming of Age? ’, Evaluation , 18 , 270 – 80 .

McCalla A. F. ( 2014 ) ‘CGIAR Reform – Why so Difficult? Review, Renewal, Restructuring, Reform Again and Then ‘the New CGIAR’—so Much Talk and so Little Basic Structural Change—Why?’, Agriculture and Resource Economics Working Paper N.14-001. Los Angeles, CA : The University of California, Davis . < https://escholarship.org/content/qt7h04960c/qt7h04960c.pdf >

Miller T. R. et al.  ( 2014 ) ‘ The Future of Sustainability Science: A Solutions-Oriented Research Agenda ’, Sustainability Science , 9 , 239 – 46 .

Miller T. R. , Muñoz-Erickson T. A. , Redman C. L. ( 2011 ) ‘ Transforming Knowledge for Sustainability: Fostering Adaptability in Academic Institutions ’, International Journal of Sustainability in Higher Education , 12 , 177 – 92 .

Mohr L. B. ( 1999 ) ‘ The Qualitative Methods of Impact Analysis ’, American Journal of Evaluation , 20 , 69 – 84 .

Nowotny H. , Scott P. , Gibbons M. ( 2001 ) Re-Thinking Science: Knowledge and the Public in an Age of Uncertainty . Cambridge, UK : Polity Press .

Pahl-Wostl C. , Mostert E. , Tabara D. ( 2008 ) ‘ The Growing Importance of Social Learning in Water Resources Management and Sustainability Science ’, Ecology and Society , 13 , 24 – 7 .

Patton M. Q. ( 2008 ) Utilization-Focused Evaluation . Thousand Oaks, CA : SAGE Publications .

Pawson R. ( 2003 ) ‘ Nothing as Practical as a Good Theory ’, Evaluation , 9 , 471 – 90 .

Pawson R. , Tilley N. ( 1997 ) Realistic Evaluation . London : SAGE Publications .

Picciotto R. ( 2013 ) ‘ Evaluation Independence in Organizations ’, Journal of MultiDisciplinary Evaluation , 9 , 18 – 32 .

Pohl C. et al.  ( 2010 ) ‘ Researchers’ Roles in Knowledge Co-Production: Experience from Sustainability Research in Kenya, Switzerland, Bolivia and Nepal ’, Science and Public Policy , 37 , 267 – 81 .

Popowitz M. , Dorgelo C. ( 2018 ) ‘Report on University-Led Grand Challenges’, < https://escholarship.org/uc/item/46f121cr > accessed March 2020.

Raitzer D. , Winkel K. ( 2005 ) Donor Demands and Uses for Evidence of Research Impact––the Case of the Consultative Group on International Agricultural Research (CGIAR) . Cali, Colombia : CGIAR .< https://cgspace.cgiar.org/handle/10568/76143 >

Randolph J. ( 2004 ) Environmental Land Use Planning and Management . Washington, DC : Island Press .

Ravetz I. R. ( 1999 ) ‘ What is Post-Normal Science ’, Futures , 31 , 647 – 54 .

Reilly R. C. ( 2010 ) ‘Process Tracing’, in Mills A. J. , Eurepos G. , Wiebc E. (eds.) Encyclopedia of Case Study Research , pp. 734 – 6 . London : SAGE Publications .

Renkow M. ( 2010 ) ‘Assessing the Environmental Impacts of CGIAR Research: Toward an Analytical Framework’, SPIA Working Paper . Rome : ISPC Secretariat . < https://ispc.cgiar.org/sites/default/files/docs/Renkow2010_0.pdf >

Renkow M. , Byerlee D. ( 2010 ) ‘ The Impacts of CGIAR Research: A Review of Recent Evidence ’, Food Policy , 35 , 391 – 402 .

Research Excellence Framework (REF) ( 2011 ) Assessment Framework and Guidance on Submissions . Bristol, UK: REF. < https://www.ref.ac.uk/2014/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf >

Robinson J. ( 2008 ) ‘ Being Undisciplined: Transgressions and Intersections in Academia and Beyond ’, Futures , 40 , 70 – 86 .

Rossi P. H. , Freeman H. E. (eds.) ( 1989 ) Evaluation: A Systematic Approach . Newbury Park, CA : SAGE Publications .

Roux D. J. et al.  ( 2017 ) ‘ Transdisciplinary Research for Systemic Change: Who to Learn with, What to Learn about and How to Learn ’, Sustainability Science , 12 , 711 – 26 .

Rubin D. B. ( 2005 ) ‘ Causal Inference Using Potential Outcomes: Design, Modeling, Decisions ’, Journal of the American Statistical Association , 100 , 322 – 31 .

Sarewitz D. ( 2016 ) ‘ Saving Science ’, The New Atlantis , 49 , 4 – 40 .

Smith A. , Stirling A. , Berkhout F. ( 2005 ) ‘ The Governance of Sustainable Socio-Technical Transitions ’, Research Policy , 34 , 1491 – 510 .

Spaapen J. , van Drooge L. ( 2011 ) ‘ Introducing ‘Productive Interactions’ in Social Impact Assessment’ , Research Evaluation , 20 , 211 – 8 .

Stevenson J. , Macours K. , Gollin D. ( 2018 a) ‘Estimating Ex Post Impacts and Rates of Return to International Agricultural Research for Development’, SPIA Technical Note N.6 . Rome: ISPC Secretariat. < https://ispc.cgiar.org/sites/default/files/pdf/ispc_technicalnote_expost_impacts_ar4d_0.pdf >

Stevenson J. , Macours K. , Gollin D. ( 2018 b) The Rigor Revolution in Impact Assessment: Implications for Cgiar . Rome : ISPC SPIA . < https://ispc.cgiar.org/sites/default/files/pdf/ispc_synthesis_study_rigor_revolution_cgiar.pdf >

Stevenson J. R. , Vlek P. ( 2018 ) Assessing the Adoption and Diffusion of Natural Resource Management Practices: Synthesis of a New Set of Empirical Studies . Rome : Independent Science and Partnership Council (ISPC ).

Stokes D. E. ( 1997 ) Pasteur’s Quadrant: Basic Science and Technological Innovation . Washington, DC : Brookings Institution Press .

Talwar S. , Wiek A. , Robinson J. ( 2011 ) ‘ User Engagement in Sustainability Research ’, Science and Public Policy , 38 , 379 – 90 .

van Kerkhoff L. , Lebel L. ( 2006 ) ‘ Linking Knowledge and Action for Sustainable Development ’, Annual Review of Environment and Resources , 31 , 445 – 77 .

Veerman J. W. , van Yperen T. A. ( 2007 ) ‘ Degrees of Freedom and Degrees of Certainty: A Developmental Model for the Establishment of Evidence-Based Youth Care ’, Evaluation and Program Planning , 30 , 212 – 21 .

Vogel I. ( 2012 ) Review of the Use of ‘Theory of Change’ in International Development, Review Report . London: DFID. < http://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/DFID_ToC_Review_VogelV7.pdf >

Walter A. I. et al.  ( 2007 ) ‘ Measuring the Societal Effects of Transdisciplinary Research Projects: Design and Application of an Evaluation Method ’, Evaluation and Program Planning , 30 , 325 – 38 .

Weiss C. H. ( 1997 ) ‘ Theory-Based Evaluation: Past, Present and Future ’, New Directions for Evaluation , 76 , 68 – 81 .

White H. ( 2009 ) ‘ Theory-Based Impact Evaluation: Principles and Practice ’, Journal of Development Effectiveness , 1 , 271 – 84 .

White H. ( 2010 ) ‘ A Contribution to Current Debates in Impact Evaluation ’, Evaluation , 16 , 153 – 64 .

White H. , Phillips D. ( 2012 ) ‘Addressing Attribution of Cause and Effect in Small n Impact Evaluations: Towards an Integrated Framework’, 3ie Working Paper 15 . < https://www.3ieimpact.org/evidence-hub/publications/working-papers/addressing-attribution-cause-and-effect-small-n-impact >

Willetts J. , Mitchell C. ( 2016 ) ‘Assessing Transdisciplinary Doctoral Research: Quality Criteria and Implications for the Examination Process’, in Fam D. , Palmer J. , Riedy C. , Mitchell C. (eds.) Transdisciplinary Research and Practice for Sustainability Outcomes , pp. 122 – 36 . Oxford : Routledge .

Wolf B. et al.  ( 2013 ) ‘ Evaluating Research beyond Scientific Impact: How to Include Criteria for Productive Interactions and Impact on Practice and Society ’, Gaia , 22 , 104 – 14 .

Young J. , Bird N. ( 2015 ) Informing REDD+ Policy: An Assessment of CIFOR’s Global Comparative Study . London: ODI. < https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/10024.pdf >

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

How to master the seven-step problem-solving process

In this episode of the McKinsey Podcast , Simon London speaks with Charles Conn, CEO of venture-capital firm Oxford Sciences Innovation, and McKinsey senior partner Hugo Sarrazin about the complexities of different problem-solving strategies.

Podcast transcript

Simon London: Hello, and welcome to this episode of the McKinsey Podcast , with me, Simon London. What’s the number-one skill you need to succeed professionally? Salesmanship, perhaps? Or a facility with statistics? Or maybe the ability to communicate crisply and clearly? Many would argue that at the very top of the list comes problem solving: that is, the ability to think through and come up with an optimal course of action to address any complex challenge—in business, in public policy, or indeed in life.

Looked at this way, it’s no surprise that McKinsey takes problem solving very seriously, testing for it during the recruiting process and then honing it, in McKinsey consultants, through immersion in a structured seven-step method. To discuss the art of problem solving, I sat down in California with McKinsey senior partner Hugo Sarrazin and also with Charles Conn. Charles is a former McKinsey partner, entrepreneur, executive, and coauthor of the book Bulletproof Problem Solving: The One Skill That Changes Everything [John Wiley & Sons, 2018].

Charles and Hugo, welcome to the podcast. Thank you for being here.

Hugo Sarrazin: Our pleasure.

Charles Conn: It’s terrific to be here.

Simon London: Problem solving is a really interesting piece of terminology. It could mean so many different things. I have a son who’s a teenage climber. They talk about solving problems. Climbing is problem solving. Charles, when you talk about problem solving, what are you talking about?

Charles Conn: For me, problem solving is the answer to the question “What should I do?” It’s interesting when there’s uncertainty and complexity, and when it’s meaningful because there are consequences. Your son’s climbing is a perfect example. There are consequences, and it’s complicated, and there’s uncertainty—can he make that grab? I think we can apply that same frame almost at any level. You can think about questions like “What town would I like to live in?” or “Should I put solar panels on my roof?”

You might think that’s a funny thing to apply problem solving to, but in my mind it’s not fundamentally different from business problem solving, which answers the question “What should my strategy be?” Or problem solving at the policy level: “How do we combat climate change?” “Should I support the local school bond?” I think these are all part and parcel of the same type of question, “What should I do?”

I’m a big fan of structured problem solving. By following steps, we can more clearly understand what problem it is we’re solving, what are the components of the problem that we’re solving, which components are the most important ones for us to pay attention to, which analytic techniques we should apply to those, and how we can synthesize what we’ve learned back into a compelling story. That’s all it is, at its heart.

I think sometimes when people think about seven steps, they assume that there’s a rigidity to this. That’s not it at all. It’s actually to give you the scope for creativity, which often doesn’t exist when your problem solving is muddled.

Simon London: You were just talking about the seven-step process. That’s what’s written down in the book, but it’s a very McKinsey process as well. Without getting too deep into the weeds, let’s go through the steps, one by one. You were just talking about problem definition as being a particularly important thing to get right first. That’s the first step. Hugo, tell us about that.

Hugo Sarrazin: It is surprising how often people jump past this step and make a bunch of assumptions. The most powerful thing is to step back and ask the basic questions—“What are we trying to solve? What are the constraints that exist? What are the dependencies?” Let’s make those explicit and really push the thinking and defining. At McKinsey, we spend an enormous amount of time in writing that little statement, and the statement, if you’re a logic purist, is great. You debate. “Is it an ‘or’? Is it an ‘and’? What’s the action verb?” Because all these specific words help you get to the heart of what matters.

Want to subscribe to The McKinsey Podcast ?

Simon London: So this is a concise problem statement.

Hugo Sarrazin: Yeah. It’s not like “Can we grow in Japan?” That’s interesting, but it is “What, specifically, are we trying to uncover in the growth of a product in Japan? Or a segment in Japan? Or a channel in Japan?” When you spend an enormous amount of time, in the first meeting of the different stakeholders, debating this and having different people put forward what they think the problem definition is, you realize that people have completely different views of why they’re here. That, to me, is the most important step.

Charles Conn: I would agree with that. For me, the problem context is critical. When we understand “What are the forces acting upon your decision maker? How quickly is the answer needed? With what precision is the answer needed? Are there areas that are off limits or areas where we would particularly like to find our solution? Is the decision maker open to exploring other areas?” then you not only become more efficient, and move toward what we call the critical path in problem solving, but you also make it so much more likely that you’re not going to waste your time or your decision maker’s time.

How often do especially bright young people run off with half of the idea about what the problem is and start collecting data and start building models—only to discover that they’ve really gone off half-cocked.

Hugo Sarrazin: Yeah.

Charles Conn: And in the wrong direction.

Simon London: OK. So step one—and there is a real art and a structure to it—is define the problem. Step two, Charles?

Charles Conn: My favorite step is step two, which is to use logic trees to disaggregate the problem. Every problem we’re solving has some complexity and some uncertainty in it. The only way that we can really get our team working on the problem is to take the problem apart into logical pieces.

What we find, of course, is that the way to disaggregate the problem often gives you an insight into the answer to the problem quite quickly. I love to do two or three different cuts at it, each one giving a bit of a different insight into what might be going wrong. By doing sensible disaggregations, using logic trees, we can figure out which parts of the problem we should be looking at, and we can assign those different parts to team members.

Simon London: What’s a good example of a logic tree on a sort of ratable problem?

Charles Conn: Maybe the easiest one is the classic profit tree. Almost in every business that I would take a look at, I would start with a profit or return-on-assets tree. In its simplest form, you have the components of revenue, which are price and quantity, and the components of cost, which are cost and quantity. Each of those can be broken out. Cost can be broken into variable cost and fixed cost. The components of price can be broken into what your pricing scheme is. That simple tree often provides insight into what’s going on in a business or what the difference is between that business and the competitors.

If we add the leg, which is “What’s the asset base or investment element?”—so profit divided by assets—then we can ask the question “Is the business using its investments sensibly?” whether that’s in stores or in manufacturing or in transportation assets. I hope we can see just how simple this is, even though we’re describing it in words.

When I went to work with Gordon Moore at the Moore Foundation, the problem that he asked us to look at was “How can we save Pacific salmon?” Now, that sounds like an impossible question, but it was amenable to precisely the same type of disaggregation and allowed us to organize what became a 15-year effort to improve the likelihood of good outcomes for Pacific salmon.

Simon London: Now, is there a danger that your logic tree can be impossibly large? This, I think, brings us onto the third step in the process, which is that you have to prioritize.

Charles Conn: Absolutely. The third step, which we also emphasize, along with good problem definition, is rigorous prioritization—we ask the questions “How important is this lever or this branch of the tree in the overall outcome that we seek to achieve? How much can I move that lever?” Obviously, we try and focus our efforts on ones that have a big impact on the problem and the ones that we have the ability to change. With salmon, ocean conditions turned out to be a big lever, but not one that we could adjust. We focused our attention on fish habitats and fish-harvesting practices, which were big levers that we could affect.

People spend a lot of time arguing about branches that are either not important or that none of us can change. We see it in the public square. When we deal with questions at the policy level—“Should you support the death penalty?” “How do we affect climate change?” “How can we uncover the causes and address homelessness?”—it’s even more important that we’re focusing on levers that are big and movable.

Would you like to learn more about our Strategy & Corporate Finance Practice ?

Simon London: Let’s move swiftly on to step four. You’ve defined your problem, you disaggregate it, you prioritize where you want to analyze—what you want to really look at hard. Then you got to the work plan. Now, what does that mean in practice?

Hugo Sarrazin: Depending on what you’ve prioritized, there are many things you could do. It could be breaking the work among the team members so that people have a clear piece of the work to do. It could be defining the specific analyses that need to get done and executed, and being clear on time lines. There’s always a level-one answer, there’s a level-two answer, there’s a level-three answer. Without being too flippant, I can solve any problem during a good dinner with wine. It won’t have a whole lot of backing.

Simon London: Not going to have a lot of depth to it.

Hugo Sarrazin: No, but it may be useful as a starting point. If the stakes are not that high, that could be OK. If it’s really high stakes, you may need level three and have the whole model validated in three different ways. You need to find a work plan that reflects the level of precision, the time frame you have, and the stakeholders you need to bring along in the exercise.

Charles Conn: I love the way you’ve described that, because, again, some people think of problem solving as a linear thing, but of course what’s critical is that it’s iterative. As you say, you can solve the problem in one day or even one hour.

Charles Conn: We encourage our teams everywhere to do that. We call it the one-day answer or the one-hour answer. In work planning, we’re always iterating. Every time you see a 50-page work plan that stretches out to three months, you know it’s wrong. It will be outmoded very quickly by that learning process that you described. Iterative problem solving is a critical part of this. Sometimes, people think work planning sounds dull, but it isn’t. It’s how we know what’s expected of us and when we need to deliver it and how we’re progressing toward the answer. It’s also the place where we can deal with biases. Bias is a feature of every human decision-making process. If we design our team interactions intelligently, we can avoid the worst sort of biases.

Simon London: Here we’re talking about cognitive biases primarily, right? It’s not that I’m biased against you because of your accent or something. These are the cognitive biases that behavioral sciences have shown we all carry around, things like anchoring, overoptimism—these kinds of things.

Both: Yeah.

Charles Conn: Availability bias is the one that I’m always alert to. You think you’ve seen the problem before, and therefore what’s available is your previous conception of it—and we have to be most careful about that. In any human setting, we also have to be careful about biases that are based on hierarchies, sometimes called sunflower bias. I’m sure, Hugo, with your teams, you make sure that the youngest team members speak first. Not the oldest team members, because it’s easy for people to look at who’s senior and alter their own creative approaches.

Hugo Sarrazin: It’s helpful, at that moment—if someone is asserting a point of view—to ask the question “This was true in what context?” You’re trying to apply something that worked in one context to a different one. That can be deadly if the context has changed, and that’s why organizations struggle to change. You promote all these people because they did something that worked well in the past, and then there’s a disruption in the industry, and they keep doing what got them promoted even though the context has changed.

Simon London: Right. Right.

Hugo Sarrazin: So it’s the same thing in problem solving.

Charles Conn: And it’s why diversity in our teams is so important. It’s one of the best things about the world that we’re in now. We’re likely to have people from different socioeconomic, ethnic, and national backgrounds, each of whom sees problems from a slightly different perspective. It is therefore much more likely that the team will uncover a truly creative and clever approach to problem solving.

Simon London: Let’s move on to step five. You’ve done your work plan. Now you’ve actually got to do the analysis. The thing that strikes me here is that the range of tools that we have at our disposal now, of course, is just huge, particularly with advances in computation, advanced analytics. There’s so many things that you can apply here. Just talk about the analysis stage. How do you pick the right tools?

Charles Conn: For me, the most important thing is that we start with simple heuristics and explanatory statistics before we go off and use the big-gun tools. We need to understand the shape and scope of our problem before we start applying these massive and complex analytical approaches.

Simon London: Would you agree with that?

Hugo Sarrazin: I agree. I think there are so many wonderful heuristics. You need to start there before you go deep into the modeling exercise. There’s an interesting dynamic that’s happening, though. In some cases, for some types of problems, it is even better to set yourself up to maximize your learning. Your problem-solving methodology is test and learn, test and learn, test and learn, and iterate. That is a heuristic in itself, the A/B testing that is used in many parts of the world. So that’s a problem-solving methodology. It’s nothing different. It just uses technology and feedback loops in a fast way. The other one is exploratory data analysis. When you’re dealing with a large-scale problem, and there’s so much data, I can get to the heuristics that Charles was talking about through very clever visualization of data.

You test with your data. You need to set up an environment to do so, but don’t get caught up in neural-network modeling immediately. You’re testing, you’re checking—“Is the data right? Is it sound? Does it make sense?”—before you launch too far.

Simon London: You do hear these ideas—that if you have a big enough data set and enough algorithms, they’re going to find things that you just wouldn’t have spotted, find solutions that maybe you wouldn’t have thought of. Does machine learning sort of revolutionize the problem-solving process? Or are these actually just other tools in the toolbox for structured problem solving?

Charles Conn: It can be revolutionary. There are some areas in which the pattern recognition of large data sets and good algorithms can help us see things that we otherwise couldn’t see. But I do think it’s terribly important we don’t think that this particular technique is a substitute for superb problem solving, starting with good problem definition. Many people use machine learning without understanding algorithms that themselves can have biases built into them. Just as 20 years ago, when we were doing statistical analysis, we knew that we needed good model definition, we still need a good understanding of our algorithms and really good problem definition before we launch off into big data sets and unknown algorithms.

Simon London: Step six. You’ve done your analysis.

Charles Conn: I take six and seven together, and this is the place where young problem solvers often make a mistake. They’ve got their analysis, and they assume that’s the answer, and of course it isn’t the answer. The ability to synthesize the pieces that came out of the analysis and begin to weave those into a story that helps people answer the question “What should I do?” This is back to where we started. If we can’t synthesize, and we can’t tell a story, then our decision maker can’t find the answer to “What should I do?”

Simon London: But, again, these final steps are about motivating people to action, right?

Charles Conn: Yeah.

Simon London: I am slightly torn about the nomenclature of problem solving because it’s on paper, right? Until you motivate people to action, you actually haven’t solved anything.

Charles Conn: I love this question because I think decision-making theory, without a bias to action, is a waste of time. Everything in how I approach this is to help people take action that makes the world better.

Simon London: Hence, these are absolutely critical steps. If you don’t do this well, you’ve just got a bunch of analysis.

Charles Conn: We end up in exactly the same place where we started, which is people speaking across each other, past each other in the public square, rather than actually working together, shoulder to shoulder, to crack these important problems.

Simon London: In the real world, we have a lot of uncertainty—arguably, increasing uncertainty. How do good problem solvers deal with that?

Hugo Sarrazin: At every step of the process. In the problem definition, when you’re defining the context, you need to understand those sources of uncertainty and whether they’re important or not important. It becomes important in the definition of the tree.

You need to think carefully about the branches of the tree that are more certain and less certain as you define them. They don’t have equal weight just because they’ve got equal space on the page. Then, when you’re prioritizing, your prioritization approach may put more emphasis on things that have low probability but huge impact—or, vice versa, may put a lot of priority on things that are very likely and, hopefully, have a reasonable impact. You can introduce that along the way. When you come back to the synthesis, you just need to be nuanced about what you’re understanding, the likelihood.

Often, people lack humility in the way they make their recommendations: “This is the answer.” They’re very precise, and I think we would all be well-served to say, “This is a likely answer under the following sets of conditions” and then make the level of uncertainty clearer, if that is appropriate. It doesn’t mean you’re always in the gray zone; it doesn’t mean you don’t have a point of view. It just means that you can be explicit about the certainty of your answer when you make that recommendation.

Simon London: So it sounds like there is an underlying principle: “Acknowledge and embrace the uncertainty. Don’t pretend that it isn’t there. Be very clear about what the uncertainties are up front, and then build that into every step of the process.”

Hugo Sarrazin: Every step of the process.

Simon London: Yeah. We have just walked through a particular structured methodology for problem solving. But, of course, this is not the only structured methodology for problem solving. One that is also very well-known is design thinking, which comes at things very differently. So, Hugo, I know you have worked with a lot of designers. Just give us a very quick summary. Design thinking—what is it, and how does it relate?

Hugo Sarrazin: It starts with an incredible amount of empathy for the user and uses that to define the problem. It does pause and go out in the wild and spend an enormous amount of time seeing how people interact with objects, seeing the experience they’re getting, seeing the pain points or joy—and uses that to infer and define the problem.

Simon London: Problem definition, but out in the world.

Hugo Sarrazin: With an enormous amount of empathy. There’s a huge emphasis on empathy. Traditional, more classic problem solving is you define the problem based on an understanding of the situation. This one almost presupposes that we don’t know the problem until we go see it. The second thing is you need to come up with multiple scenarios or answers or ideas or concepts, and there’s a lot of divergent thinking initially. That’s slightly different, versus the prioritization, but not for long. Eventually, you need to kind of say, “OK, I’m going to converge again.” Then you go and you bring things back to the customer and get feedback and iterate. Then you rinse and repeat, rinse and repeat. There’s a lot of tactile building, along the way, of prototypes and things like that. It’s very iterative.

Simon London: So, Charles, are these complements or are these alternatives?

Charles Conn: I think they’re entirely complementary, and I think Hugo’s description is perfect. When we do problem definition well in classic problem solving, we are demonstrating the kind of empathy, at the very beginning of our problem, that design thinking asks us to approach. When we ideate—and that’s very similar to the disaggregation, prioritization, and work-planning steps—we do precisely the same thing, and often we use contrasting teams, so that we do have divergent thinking. The best teams allow divergent thinking to bump them off whatever their initial biases in problem solving are. For me, design thinking gives us a constant reminder of creativity, empathy, and the tactile nature of problem solving, but it’s absolutely complementary, not alternative.

Simon London: I think, in a world of cross-functional teams, an interesting question is do people with design-thinking backgrounds really work well together with classical problem solvers? How do you make that chemistry happen?

Hugo Sarrazin: Yeah, it is not easy when people have spent an enormous amount of time seeped in design thinking or user-centric design, whichever word you want to use. If the person who’s applying classic problem-solving methodology is very rigid and mechanical in the way they’re doing it, there could be an enormous amount of tension. If there’s not clarity in the role and not clarity in the process, I think having the two together can be, sometimes, problematic.

The second thing that happens often is that the artifacts the two methodologies try to gravitate toward can be different. Classic problem solving often gravitates toward a model; design thinking migrates toward a prototype. Rather than writing a big deck with all my supporting evidence, they’ll bring an example, a thing, and that feels different. Then you spend your time differently to achieve those two end products, so that’s another source of friction.

Now, I still think it can be an incredibly powerful thing to have the two—if there are the right people with the right mind-set, if there is a team that is explicit about the roles, if we’re clear about the kind of outcomes we are attempting to bring forward. There’s an enormous amount of collaborativeness and respect.

Simon London: But they have to respect each other’s methodology and be prepared to flex, maybe, a little bit, in how this process is going to work.

Hugo Sarrazin: Absolutely.

Simon London: The other area where, it strikes me, there could be a little bit of a different sort of friction is this whole concept of the day-one answer, which is what we were just talking about in classical problem solving. Now, you know that this is probably not going to be your final answer, but that’s how you begin to structure the problem. Whereas I would imagine your design thinkers—no, they’re going off to do their ethnographic research and get out into the field, potentially for a long time, before they come back with at least an initial hypothesis.

Want better strategies? Become a bulletproof problem solver

Hugo Sarrazin: That is a great callout, and that’s another difference. Designers typically will like to soak into the situation and avoid converging too quickly. There’s optionality and exploring different options. There’s a strong belief that keeps the solution space wide enough that you can come up with more radical ideas. If there’s a large design team or many designers on the team, and you come on Friday and say, “What’s our week-one answer?” they’re going to struggle. They’re not going to be comfortable, naturally, to give that answer. It doesn’t mean they don’t have an answer; it’s just not where they are in their thinking process.

Simon London: I think we are, sadly, out of time for today. But Charles and Hugo, thank you so much.

Charles Conn: It was a pleasure to be here, Simon.

Hugo Sarrazin: It was a pleasure. Thank you.

Simon London: And thanks, as always, to you, our listeners, for tuning into this episode of the McKinsey Podcast . If you want to learn more about problem solving, you can find the book, Bulletproof Problem Solving: The One Skill That Changes Everything , online or order it through your local bookstore. To learn more about McKinsey, you can of course find us at McKinsey.com.

Charles Conn is CEO of Oxford Sciences Innovation and an alumnus of McKinsey’s Sydney office. Hugo Sarrazin is a senior partner in the Silicon Valley office, where Simon London, a member of McKinsey Publishing, is also based.

Explore a career with us

Related articles.

Strategy to beat the odds

Five routes to more innovative problem solving

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Perspective
• Published: 25 January 2022

Intelligent problem-solving as integrated hierarchical reinforcement learning

• Manfred Eppe   ORCID: orcid.org/0000-0002-5473-3221 1   nAff4 ,
• Christian Gumbsch   ORCID: orcid.org/0000-0003-2741-6551 2 , 3 ,
• Matthias Kerzel 1 ,
• Phuong D. H. Nguyen 1 ,
• Martin V. Butz   ORCID: orcid.org/0000-0002-8120-8537 2 &
• Stefan Wermter 1  

Nature Machine Intelligence volume  4 ,  pages 11–20 ( 2022 ) Cite this article

5209 Accesses

31 Citations

8 Altmetric

Metrics details

• Cognitive control
• Computational models
• Computer science
• Learning algorithms
• Problem solving

According to cognitive psychology and related disciplines, the development of complex problem-solving behaviour in biological agents depends on hierarchical cognitive mechanisms. Hierarchical reinforcement learning is a promising computational approach that may eventually yield comparable problem-solving behaviour in artificial agents and robots. However, so far, the problem-solving abilities of many human and non-human animals are clearly superior to those of artificial systems. Here we propose steps to integrate biologically inspired hierarchical mechanisms to enable advanced problem-solving skills in artificial agents. We first review the literature in cognitive psychology to highlight the importance of compositional abstraction and predictive processing. Then we relate the gained insights with contemporary hierarchical reinforcement learning methods. Interestingly, our results suggest that all identified cognitive mechanisms have been implemented individually in isolated computational architectures, raising the question of why there exists no single unifying architecture that integrates them. As our final contribution, we address this question by providing an integrative perspective on the computational challenges to develop such a unifying architecture. We expect our results to guide the development of more sophisticated cognitively inspired hierarchical machine learning architectures.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

$29.99 / 30 days

cancel any time

Subscribe to this journal

Receive 12 digital issues and online access to articles

$119.00 per year

only $9.92 per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Phy-Q as a measure for physical reasoning intelligence

Cheng Xue, Vimukthini Pinto, … Jochen Renz

Hierarchical motor control in mammals and machines

Josh Merel, Matthew Botvinick & Greg Wayne

Hierarchical generative modelling for autonomous robots

Kai Yuan, Noor Sajid, … Zhibin Li

Gruber, R. et al. New Caledonian crows use mental representations to solve metatool problems. Curr. Biol. 29 , 686–692 (2019).

Article   Google Scholar  

Butz, M. V. & Kutter, E. F. How the Mind Comes into Being (Oxford Univ. Press, 2017).

Perkins, D. N. & Salomon, G. in International Encyclopedia of Education (eds. Husen T. & Postelwhite T. N.) 6452–6457 (Pergamon Press, 1992).

Botvinick, M. M., Niv, Y. & Barto, A. C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113 , 262–280 (2009).

Tomov, M. S., Yagati, S., Kumar, A., Yang, W. & Gershman, S. J. Discovery of hierarchical representations for efficient planning. PLoS Comput. Biol. 16 , e1007594 (2020).

Arulkumaran, K., Deisenroth, M. P., Brundage, M. & Bharath, A. A. Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34 , 26–38 (2017).

Li, Y. Deep reinforcement learning: an overview. Preprint at https://arxiv.org/abs/1701.07274 (2018).

Sutton, R. S. & Barto, A. G. Reinforcement Learning : An Introduction 2nd edn (MIT Press, 2018).

Neftci, E. O. & Averbeck, B. B. Reinforcement learning in artificial and biological systems. Nat. Mach. Intell. 1 , 133–143 (2019).

Eppe, M., Nguyen, P. D. H. & Wermter, S. From semantics to execution: integrating action planning with reinforcement learning for robotic causal problem-solving. Front. Robot. AI 6 , 123 (2019).

Oh, J., Singh, S., Lee, H. & Kohli, P. Zero-shot task generalization with multi-task deep reinforcement learning. In Proc. 34th International Conference on Machine Learning ( ICML ) (eds. Precup, D. & Teh, Y. W.) 2661–2670 (PMLR, 2017).

Sohn, S., Oh, J. & Lee, H. Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies. In Proc. 32nd International Conference on Neural Information Processing Systems ( NeurIPS ) (eds Bengio S. et al.) Vol. 31, 7156–7166 (ACM, 2018).

Hegarty, M. Mechanical reasoning by mental simulation. Trends Cogn. Sci. 8 , 280–285 (2004).

Klauer, K. J. Teaching for analogical transfer as a means of improving problem-solving, thinking and learning. Instruct. Sci. 18 , 179–192 (1989).

Duncker, K. & Lees, L. S. On problem-solving. Psychol. Monographs 58, No.5 (whole No. 270), 85–101 https://doi.org/10.1037/h0093599 (1945).

Dayan, P. Goal-directed control and its antipodes. Neural Netw. 22 , 213–219 (2009).

Dolan, R. J. & Dayan, P. Goals and habits in the brain. Neuron 80 , 312–325 (2013).

O’Doherty, J. P., Cockburn, J. & Pauli, W. M. Learning, reward, and decision making. Annu. Rev. Psychol. 68 , 73–100 (2017).

Tolman, E. C. & Honzik, C. H. Introduction and removal of reward, and maze performance in rats. Univ. California Publ. Psychol. 4 , 257–275 (1930).

Google Scholar  

Butz, M. V. & Hoffmann, J. Anticipations control behavior: animal behavior in an anticipatory learning classifier system. Adaptive Behav. 10 , 75–96 (2002).

Miller, G. A., Galanter, E. & Pribram, K. H. Plans and the Structure of Behavior (Holt, Rinehart & Winston, 1960).

Botvinick, M. & Weinstein, A. Model-based hierarchical reinforcement learning and human action control. Philos. Trans. R. Soc. B Biol. Sci. 369 , 20130480 (2014).

Wiener, J. M. & Mallot, H. A. ’Fine-to-coarse’ route planning and navigation in regionalized environments. Spatial Cogn. Comput. 3 , 331–358 (2003).

Stock, A. & Stock, C. A short history of ideo-motor action. Psychol. Res. 68 , 176–188 (2004).

Hommel, B., Müsseler, J., Aschersleben, G. & Prinz, W. The theory of event coding (TEC): a framework for perception and action planning. Behav. Brain Sci. 24 , 849–878 (2001).

Hoffmann, J. in Anticipatory Behavior in Adaptive Learning Systems : Foundations , Theories and Systems (eds Butz, M. V. et al.) 44–65 (Springer, 2003).

Kunde, W., Elsner, K. & Kiesel, A. No anticipation-no action: the role of anticipation in action and perception. Cogn. Process. 8 , 71–78 (2007).

Barsalou, L. W. Grounded cognition. Annu. Rev. Psychol. 59 , 617–645 (2008).

Butz, M. V. Toward a unified sub-symbolic computational theory of cognition. Front. Psychol. 7 , 925 (2016).

Pulvermüller, F. Brain embodiment of syntax and grammar: discrete combinatorial mechanisms spelt out in neuronal circuits. Brain Lang. 112 , 167–179 (2010).

Sutton, R. S., Precup, D. & Singh, S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112 , 181–211 (1999).

Article   MathSciNet   MATH   Google Scholar  

Flash, T. & Hochner, B. Motor primitives in vertebrates and invertebrates. Curr. Opin. Neurobiol. 15 , 660–666 (2005).

Schaal, S. in Adaptive Motion of Animals and Machines (eds. Kimura, H. et al.) 261–280 (Springer, 2006).

Feldman, J., Dodge, E. & Bryant, J. in The Oxford Handbook of Linguistic Analysis (eds Heine, B. & Narrog, H.) 111–138 (Oxford Univ. Press, 2009).

Fodor, J. A. Language, thought and compositionality. Mind Lang. 16 , 1–15 (2001).

Frankland, S. M. & Greene, J. D. Concepts and compositionality: in search of the brain’s language of thought. Annu. Rev. Psychol. 71 , 273–303 (2020).

Hummel, J. E. Getting symbols out of a neural architecture. Connection Sci. 23 , 109–118 (2011).

Haynes, J. D., Wisniewski, D., Gorgen, K., Momennejad, I. & Reverberi, C. FMRI decoding of intentions: compositionality, hierarchy and prospective memory. In Proc. 3rd International Winter Conference on Brain-Computer Interface ( BCI ), 1-3 (IEEE, 2015).

Gärdenfors, P. The Geometry of Meaning : Semantics Based on Conceptual Spaces (MIT Press, 2014).

Book   MATH   Google Scholar  

Lakoff, G. & Johnson, M. Philosophy in the Flesh (Basic Books, 1999).

Eppe, M. et al. A computational framework for concept blending. Artif. Intell. 256 , 105–129 (2018).

Turner, M. The Origin of Ideas (Oxford Univ. Press, 2014).

Deci, E. L. & Ryan, R. M. Self-determination theory and the facilitation of intrinsic motivation. Am. Psychol. 55 , 68–78 (2000).

Friston, K. et al. Active inference and epistemic value. Cogn. Neurosci. 6 , 187–214 (2015).

Berlyne, D. E. Curiosity and exploration. Science 153 , 25–33 (1966).

Loewenstein, G. The psychology of curiosity: a review and reinterpretation. Psychol. Bull. 116 , 75–98 (1994).

Oudeyer, P.-Y., Kaplan, F. & Hafner, V. V. Intrinsic motivation systems for autonomous mental development. In IEEE Transactions on Evolutionary Computation (eds. Coello, C. A. C. et al.) Vol. 11, 265–286 (IEEE, 2007).

Pisula, W. Play and exploration in animals—a comparative analysis. Polish Psychol. Bull. 39 , 104–107 (2008).

Jeannerod, M. Mental imagery in the motor context. Neuropsychologia 33 , 1419–1432 (1995).

Kahnemann, D. & Tversky, A. in Judgement under Uncertainty : Heuristics and Biases (eds Kahneman, D. et al.) Ch. 14, 201–208 (Cambridge Univ. Press, 1982).

Wells, G. L. & Gavanski, I. Mental simulation of causality. J. Personal. Social Psychol. 56 , 161–169 (1989).

Taylor, S. E., Pham, L. B., Rivkin, I. D. & Armor, D. A. Harnessing the imagination: mental simulation, self-regulation and coping. Am. Psychol. 53 , 429–439 (1998).

Kaplan, F. & Oudeyer, P.-Y. in Embodied Artificial Intelligence , Lecture Notes in Computer Science Vol. 3139 (eds Iida, F. et al.) 259–270 (Springer, 2004).

Schmidhuber, J. Formal theory of creativity, fun, and intrinsic motivation. IEEE Trans. Auton. Mental Dev. 2 , 230–247 (2010).

Friston, K., Mattout, J. & Kilner, J. Action understanding and active inference. Biol. Cybern. 104 , 137–160 (2011).

Oudeyer, P.-Y. Computational theories of curiosity-driven learning. In The New Science of Curiosity (ed. Goren Gordon), 43-72 (Nova Science Publishers, 2018); https://arxiv.org/abs/1802.10546

Colombo, M. & Wright, C. First principles in the life sciences: the free-energy principle, organicism and mechanism. Synthese 198 , 3463–3488 (2021).

Article   MathSciNet   Google Scholar  

Huang, Y. & Rao, R. P. Predictive coding. WIREs Cogn. Sci. 2 , 580–593 (2011).

Friston, K. The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11 , 127–138 (2010).

Knill, D. C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27 , 712–719 (2004).

Clark, A. Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 36 , 181–204 (2013).

Clark, A. Surfing Uncertainty : Prediction , Action and the Embodied Mind (Oxford Univ. Press, 2016).

Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S. & Reyonolds, J. R. Event perception: a mind/brain perspective. Psychol. Bull. 133 , 273–293 (2007).

Eysenbach, B., Ibarz, J., Gupta, A. & Levine, S. Diversity is all you need: learning skills without a reward function. In International Conference on Learning Representations (ICLR, 2019).

Frans, K., Ho, J., Chen, X., Abbeel, P. & Schulman, J. Meta learning shared hierarchies. In Proc. International Conference on Learning Representations https://openreview.net/pdf?id=SyX0IeWAW (ICLR, 2018).

Heess, N. et al. Learning and transfer of modulated locomotor controllers. Preprint at https://arxiv.org/abs/1610.05182 (2016).

Jiang, Y., Gu, S., Murphy, K. & Finn, C. Language as an abstraction for hierarchical deep reinforcement learning. In Neural Information Processing Systems ( NeurIPS ) (eds. Wallach, H. et al.) 9414–9426 (ACM, 2019).

Li, A. C., Florensa, C., Clavera, I. & Abbeel, P. Sub-policy adaptation for hierarchical reinforcement learning. In Proc. International Conference on Learning Representations https://openreview.net/forum?id=ByeWogStDS (ICLR, 2020).

Qureshi, A. H. et al. Composing task-agnostic policies with deep reinforcement learning. In Proc. International Conference on Learning Representations https://openreview.net/forum?id=H1ezFREtwH (ICLR, 2020).

Sharma, A., Gu, S., Levine, S., Kumar, V. & Hausman, K. Dynamics-aware unsupervised discovery of skills. In Proc. International Conference on Learning Representations https://openreview.net/forum?id=HJgLZR4KvH (ICLR, 2020).

Tessler, C., Givony, S., Zahavy, T., Mankowitz, D. J. & Mannor, S. A deep hierarchical approach to lifelong learning in minecraft. In Proc. 31st AAAI Conference on Artificial Intelligence 1553–1561 (AAAI, 2017).

Vezhnevets, A. et al. Strategic attentive writer for learning macro-actions. In Neural Information Processing Systems ( NIPS ) (eds. Lee, D. et al.) 3494–3502 (NIPS, 2016).

Devin, C., Gupta, A., Darrell, T., Abbeel, P. & Levine, S. Learning modular neural network policies for multi-task and multi-robot transfer. In Proc. International Conference on Robotics and Automation ( ICRA ) (eds. Okamura, A. et al.) 2169–2176 (IEEE, 2017).

Hejna, D. J., Abbeel, P. & Pinto, L. Hierarchically decoupled morphological transfer. In Proc. International Conference on Machine Learning ( ICML ) (eds. Daumé III, H. & Singh, A.) 11409–11420 (PMLR, 2020).

Hamrick, J. B. et al. On the role of planning in model-based deep reinforcement learning. In Proc. International Conference on Learning Representations https://openreview.net/pdf?id=IrM64DGB21 (ICLR, 2021).

Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proc. 7th International Conference on Machine Learning ( ICML ) (eds. Porter, B. W. & Mooney, R. J.) 216–224 (Morgan Kaufmann, 1990).

Nau, D. et al. SHOP2: an HTN planning system. J. Artif. Intell. Res. 20 , 379–404 (2003).

Article   MATH   Google Scholar  

Lyu, D., Yang, F., Liu, B. & Gustafson, S. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proc. AAAI Conference on Artificial Intelligence Vol. 33, 2970–2977 (AAAI, 2019).

Ma, A., Ouimet, M. & Cortés, J. Hierarchical reinforcement learning via dynamic subspace search for multi-agent planning. Auton. Robot. 44 , 485–503 (2020).

Bacon, P.-L., Harb, J. & Precup, D. The option-critic architecture. In Proc. 31st AAAI Conference on Artificial Intelligence 1726–1734 (AAAI, 2017).

Dietterich, T. G. State abstraction in MAXQ hierarchical reinforcement learning. In Advances in Neural Information Processing Systems ( NIPS ) (eds. Solla, S. et al.) Vol. 12, 994–1000 (NIPS, 1999).

Kulkarni, T. D., Narasimhan, K. R., Saeedi, A. & Tenenbaum, J. B. Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation. In Neural Information Processing Systems ( NIPS ) (eds. Lee, D. et al.) 3675–3683 (NIPS, 2016).

Shankar, T., Pinto, L., Tulsiani, S. & Gupta, A. Discovering motor programs by recomposing demonstrations. In Proc. International Conference on Learning Representations https://openreview.net/attachment?id=rkgHY0NYwr&name=original_pdf (ICLR, 2020).

Vezhnevets, A. S., Wu, Y. T., Eckstein, M., Leblond, R. & Leibo, J. Z. Options as responses: grounding behavioural hierarchies in multi-agent reinforcement learning. In Proc. International Conference on Machine Learning ( ICML ) (eds. Daumé III, H. & Singh, A.) 9733–9742 (PMLR, 2020).

Ghazanfari, B., Afghah, F. & Taylor, M. E. Sequential association rule mining for autonomously extracting hierarchical task structures in reinforcement learning. IEEE Access 8 , 11782–11799 (2020).

Levy, A., Konidaris, G., Platt, R. & Saenko, K. Learning multi-level hierarchies with hindsight. In Proc. International Conference on Learning Representations https://openreview.net/pdf?id=ryzECoAcY7 (ICLR, 2019).

Nachum, O., Gu, S., Lee, H. & Levine, S. Data-efficient hierarchical reinforcement learning. In Proc. 32nd International Conference on Neural Information Processing Systems (NIPS) (eds. Bengio, S. et al.) 3307–3317 (NIPS, 2018).

Rafati, J. & Noelle, D. C. Learning representations in model-free hierarchical reinforcement learning. In Proc. 33rd AAAI Conference on Artificial Intelligence 10009–10010 (AAAI, 2019).

Röder, F., Eppe, M., Nguyen, P. D. H. & Wermter, S. Curious hierarchical actor-critic reinforcement learning. In Proc. International Conference on Artificial Neural Networks ( ICANN ) (eds. Farkaš, I. et al.) 408–419 (Springer, 2020).

Zhang, T., Guo, S., Tan, T., Hu, X. & Chen, F. Generating adjacency-constrained subgoals in hierarchical reinforcement learning. In Neural Information Processing Systems ( NIPS ) (eds. Larochelle, H. et al.) 21579-21590 (NIPS, 2020).

Lample, G. & Chaplot, D. S. Playing FPS games with deep reinforcement learning. In Proc. 31st AAAI Conference on Artificial Intelligence 2140–2146 (AAAI, 2017).

Vezhnevets, A. S. et al. FeUdal networks for hierarchical reinforcement learning. In Proc. 34th International Conference on Machine Learning ( ICML ) (eds. Precup, D. & Teh, Y. W.) Vol. 70, 3540–3549 (PMLR, 2017).

Wulfmeier, M. et al. Compositional Transfer in Hierarchical Reinforcement Learning. In Robotics: Science and System XVI (RSS) (eds. Toussaint M. et al.) (Robotics: Science and Systems Foundation, 2020); https://arxiv.org/abs/1906.11228

Yang, Z., Merrick, K., Jin, L. & Abbass, H. A. Hierarchical deep reinforcement learning for continuous action control. IEEE Trans. Neural Netw. Learn. Syst. 29 , 5174–5184 (2018).

Toussaint, M., Allen, K. R., Smith, K. A. & Tenenbaum, J. B. Differentiable physics and stable modes for tool-use and manipulation planning. In Proc. Robotics : Science and Systems XIV ( RSS ) (eds. Kress-Gazit, H. et al.) https://ipvs.informatik.uni-stuttgart.de/mlr/papers/18-toussaint-RSS.pdf (Robotics: Science and Systems Foundation, 2018).

Akrour, R., Veiga, F., Peters, J. & Neumann, G. Regularizing reinforcement learning with state abstraction. In Proc. IEEE / RSJ International Conference on Intelligent Robots and Systems ( IROS ) 534–539 (IEEE, 2018).

Schaul, T. & Ring, M. Better generalization with forecasts. In Proc. 23rd International Joint Conference on Artificial Intelligence ( IJCAI ) (ed. Rossi, F.) 1656–1662 (AAAI, 2013).

Colas, C., Akakzia, A., Oudeyer, P.-Y., Chetouani, M. & Sigaud, O. Language-conditioned goal generation: a new approach to language grounding for RL. Preprint at https://arxiv.org/abs/2006.07043 (2020).

Blaes, S., Pogancic, M. V., Zhu, J. J. & Martius, G. Control what you can: intrinsically motivated task-planning agent. Neural Inf. Process. Syst. 32 , 12541–12552 (2019).

Haarnoja, T., Hartikainen, K., Abbeel, P. & Levine, S. Latent space policies for hierarchical reinforcement learning. In Proc. International Conference on Machine Learning ( ICML ) (eds. Dy, J. & Krause, A.) Vol. 4, 2965–2975 (PMLR, 2018).

Rasmussen, D., Voelker, A. & Eliasmith, C. A neural model of hierarchical reinforcement learning. PLoS ONE 12 , e0180234 (2017).

Riedmiller, M. et al. Learning by playing—solving sparse reward tasks from scratch. In Proc. International Conference on Machine Learning ( ICML ) (eds. Dy, J. & Krause, A.) Vol. 10, 6910–6919 (PMLR, 2018).

Yang, F., Lyu, D., Liu, B. & Gustafson, S. PEORL: integrating symbolic planning and hierarchical reinforcement learning for robust decision-making. In Proc. 27th International Joint Conference on Artificial Intelligence ( IJCAI ) (ed. Lang, J.) 4860–4866 (IJCAI, 2018).

Machado, M. C., Bellemare, M. G. & Bowling, M. A Laplacian framework for option discovery in reinforcement learning. In Proc. International Conference on Machine Learning (ICML) (eds. Precup, D. & Teh, Y. W.) Vol. 5, 3567–3582 (PMLR, 2017).

Pathak, D., Agrawal, P., Efros, A. A. & Darrell, T. Curiosity-driven exploration by self-supervised prediction. In Proc. 34th International Conference on Machine Learning ( ICML ) (eds. Precup, D. & Teh, Y. W.) 2778–2787 (PMLR, 2017).

Schillaci, G. et al. Intrinsic motivation and episodic memories for robot exploration of high-dimensional sensory spaces. Adaptive Behav. 29 549–566 (2020).

Colas, C., Fournier, P., Sigaud, O., Chetouani, M. & Oudeyer, P.-Y. CURIOUS: intrinsically motivated modular multi-goal reinforcement learning. In Proc. International Conference on Machine Learning ( ICML ) (eds. Chaudhuri, K. & Salakhutdinov, R.) 1331–1340 (PMLR, 2019).

Hafez, M. B., Weber, C., Kerzel, M. & Wermter, S. Improving robot dual-system motor learning with intrinsically motivated meta-control and latent-space experience imagination. Robot. Auton. Syst. 133 , 103630 (2020).

Yamamoto, K., Onishi, T. & Tsuruoka, Y. Hierarchical reinforcement learning with abductive planning. In Proc. ICML / IJCAI / AAMAS 2018 Workshop on Planning and Learning ( PAL-18 ) (2018).

Wu, B., Gupta, J. K. & Kochenderfer, M. J. Model primitive hierarchical lifelong reinforcement learning . In Proc. International Joint Conference on Autonomous Agents and Multiagent Systems ( AAMAS ) (eds. Agmon, N. et al.) Vol. 1, 34–42 (IFAAMAS, 2019).

Li, Z., Narayan, A. & Leong, T. Y. An efficient approach to model-based hierarchical reinforcement learning. In Proc. 31st AAAI Conference on Artificial Intelligence 3583–3589 (AAAI, 2017).

Hafner, D., Lillicrap, T. & Norouzi, M. Dream to control: learning behaviors by latent imagination. In Proc. International Conference on Learning Representations https://openreview.net/pdf?id=S1lOTC4tDS (ICLR, 2020).

Deisenroth, M. P., Rasmussen, C. E. & Fox, D. Learning to control a low-cost manipulator using data-efficient reinforcement learning. In Robotics : Science and Systems VII ( RSS ) (eds. Durrant-Whyte, H. et al.) 57–64 (Robotics: Science and Systems Foundation, 2011).

Ha, D. & Schmidhuber, J. Recurrent world models facilitate policy evolution. In Proc. 32nd International Conference on Neural Information Processing Systems (NeurIPS) (eds. Bengio, S. et al.) 2455–2467 (NIPS, 2018).

Battaglia, P. W. et al. Relational inductive biases, deep learning and graph networks. Preprint at https://arxiv.org/abs/1806.01261 (2018).

Andrychowicz, M. et al. Hindsight experience replay. In Proc. Neural Information Processing Systems ( NIPS ) (eds. Guyon I. et al.) 5048–5058 (NIPS, 2017); https://papers.nips.cc/paper/7090-hindsight-experience-replay.pdf

Schwartenbeck, P. et al. Computational mechanisms of curiosity and goal-directed exploration. eLife 8 , e41703 (2019).

Haarnoja, T., Zhou, A., Abbeel, P. & Levine, S. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. International Conference on Machine Learning ( ICML ) (eds. Dy, J. & Krause, A.) 1861–1870 (PMLR, 2018).

Yu, A. J. & Dayan, P. Uncertainty, neuromodulation and attention. Neuron 46 , 681–692 (2005).

Baldwin, D. A. & Kosie, J. E. How does the mind render streaming experience as events? Top. Cogn. Sci. 13 , 79–105 (2021).

Download references

Acknowledgements

We acknowledge funding from the DFG (projects IDEAS, LeCAREbot, TRR169, SPP 2134, RTG 1808 and EXC 2064/1), the Humboldt Foundation and Max Planck Research School IMPRS-IS.

Author information

Manfred Eppe

Present address: Hamburg University of Technology, Hamburg, Germany

Authors and Affiliations

Universität Hamburg, Hamburg, Germany

Manfred Eppe, Matthias Kerzel, Phuong D. H. Nguyen & Stefan Wermter

University of Tübingen, Tübingen, Germany

Christian Gumbsch & Martin V. Butz

Max Planck Institute for Intelligent Systems, Tübingen, Germany

Christian Gumbsch

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Manfred Eppe .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary Boxes 1–6 and Table 1.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Eppe, M., Gumbsch, C., Kerzel, M. et al. Intelligent problem-solving as integrated hierarchical reinforcement learning. Nat Mach Intell 4 , 11–20 (2022). https://doi.org/10.1038/s42256-021-00433-9

Download citation

Received : 18 December 2020

Accepted : 07 December 2021

Published : 25 January 2022

Issue Date : January 2022

DOI : https://doi.org/10.1038/s42256-021-00433-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Efficient stacking and grasping in unstructured environments.

• Jinbiao Zhu

Journal of Intelligent & Robotic Systems (2024)

Four attributes of intelligence, a thousand questions

• Matthieu Bardal
• Eric Chalmers

Biological Cybernetics (2023)

An Alternative to Cognitivism: Computational Phenomenology for Deep Learning

• Pierre Beckmann
• Guillaume Köstner
• Inês Hipólito

Minds and Machines (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Learning, Design, and Technology pp 933–953 Cite as

Integrated Problem-Based Learning: A Case Study in an Undergraduate Cohort Degree Program

• Zain Ali 4 ,
• Nanxi Meng 4 ,
• Scott Warren 4 &
• Lin Lin-Lipsmeyer 5  
• Reference work entry
• First Online: 15 October 2023

26 Accesses

The tuition fee for education in the United States is skyrocketing, and 30% of students across US universities drop out after the first year. Faculty have reported having limited impact on students’ college careers, yet many universities continue delivering classes in a traditional manner with no major changes on the horizon. This chapter shares a case study of a transformational integrated undergraduate cohort-based BS degree program built on the foundation of problem-based learning at the Frisco Campus of the University of North Texas. The chapter identifies five major pillars of the degree program including (1) cohort-based and problem-based learning as the pedagogy, (2) partnership with industry organizations to provide hands-on experience for students, (3) integration of curriculum between all cohort-based classes, (4) internal partnership with student services to get students ready with life skills, and finally (5) organizational partnerships to provide three internships for students before graduation. This chapter shares the journey from concept of the program to the completion of the first year, along with what program components worked and did not work. The study concludes with the achievements and changes that were made for the academic year (2020–2021), and further research directions are suggested in addition to key lessons learned.

Download reference work entry PDF

Introduction

According to College Atlas (2018), it is projected that 70% of Americans will study at a four-year college, but less than 62% among them will graduate with a college degree in six years, and 30% of students will drop out of college at the end of their first year across all universities in the United States. Another survey by LendEDU (2018) found that 55% of students in the United States struggled to find the money to pay for college, and 51% dropped out of college because of financial issues. An increasing drop-out rate of undergraduate students in the United States has reminded higher education faculty, researchers, and stakeholders that increasing the credit hours and prolonging the dominant 4-year schedule of undergraduate studies are not viable solutions to the issue.

Facing the increasingly competitive job market, many post-secondary institutes have prioritized preparing students to acquire the most desired soft skills by employers to increase the chances of their graduates getting employment. The National Association of Colleges and Employers (NACE, Peck, 2017 ) ranked communication, problem-solving, and collaboration as the top three desired skills for employers. Krause (2009) pointed out that employers want to hire graduates that already have these above-mentioned skills; the consensus was that the essential skills should have developed in college and should only require refining onboard the job. Given the acknowledgment, Bauer-Wolf’s study ( 2019 ) still found employers reported having difficulty finding such candidates for their openings. These three major issues prevent undergraduate education in the United States from fulfilling the needs of students. An undergraduate degree that serves the needs of students and is financially affordable is highly desired.

To address these three issues, a reform of curriculum instruction and program design is desired in the interest of current and prospective students of higher education, faculty, stakeholders, and policy makers. In order to better prepare students facing the job market upon their graduation, one of the suggested reforms, among many others, is to expose students to real-world scenarios to develop problem-solving skills through hands-on experiences that require communicating and collaborating with others.

Problem-based learning (PBL) is an instructional method that originally comes from medical education; now it has been adopted and implemented as an instructional pedagogy in diversified fields and domains. Significantly different from traditional teacher-centered and lecture-based teaching methods, PBL is rooted in constructivist learning theory (Savery & Duffy, 1995 ). It relies largely on student autonomy, which requires students to work autonomously in a goal setting, take on responsibilities, collaborate, and communicate in their learning (Wijnia et al., 2011 ). While instructors serve the role of learning facilitators, PBL emphasizes on student self-directed discovery and learning through solving real-life issues. PBL instruction, hence, aids students to build on their soft skills and enhance them during their undergraduate studies.

To better support the implementation of PBL at the level of undergraduate degree design, cohort-based learning is introduced along with the PBL as the core to facilitate the cultivation of the soft skills. Additionally, four other pillars are added to achieve the overall success of the undergraduate degree education reform: integrated academic curriculum, student service integration, internship, and organizational partnership.

The current study shares the exploratory case study of the integration of the above five components at an undergraduate degree program level. The University of North Texas (UNT) Frisco campus launched UNT’s first PBL cohort program for First-Time-in-College (FTIC) students in fall 2019. Twenty-three freshmen students were admitted to the undergraduate degree program Project Design and Analysis (PD&A). So far students have worked on a transportation project and a business improvement project with a focus on improving efficiency in how restaurants were opened with the City of Frisco (COF) as an industry/organization partner.

The PD&A program is an innovative exploration for instruction and program design in higher education. The goal of UNT initiating such a program is to tackle two challenges faced by post-secondary institutes in the United States and worldwide: student retention and career readiness. In the current study, we propose to conduct an exploratory case study to showcase the ongoing PD&A program and seek to answer the following research questions:

What are the key processes of integrating five pillars into the PBL cohort PD&A program?

How does PD&A program address the typical three issues (employer desired skills, student drop-out rate, and financial burden) faced by US higher education institutes?

What key learnings were applied to the PD&A program after the first year in this design-based research approach of teaching?

Literature Review

Theoretical foundation of problem-based learning.

Problem-based learning (PBL) is an instructional method that originally comes from medical education in the early 1990s, but now it has been adopted and implemented as an instructional pedagogy in many diverse fields and domains. PBL roots are found in constructivist learning theory (Savery & Duffy, 1995 ), relying largely on student autonomy, which requires students to work autonomously in the learning goal setting, taking responsibilities for their learning, collaboration, and communications (Wijnia et al., 2011 ). PBL emphasizes on student self-directed discovery while instructors serve the role of learning facilitator and help students learn by solving real-life problems.

Barrows (2002) identified four key components of PBL: ill-structured problems that trigger not only students’ thoughts on the cause, but also the solution; student-centered approach that allows students to decide what they need to learn; teachers serve as facilitators and tutors, instead of traditional knowledge provider, and guide students to ask meta-cognitive questions and gradually release learning responsibilities to students; and authenticity of problems presented to students to learn, keeping alignment to professional or “real world” practice. With the four key components of PBL set forth, Savery (2006) defined PBL as an instructional approach that is learner centered and empowers learners to conduct research, integrate theory and practice, and apply knowledge and skills to develop a viable solution to a defined problem.

To implement PBL successfully, Savery and Duffy ( 1995 ) provided eight principles and commented on the critical features of PBL on learning goal setting, problem generation, problem presentation, and facilitator role. PBL model addresses collaboration and communication skills, and educators have achieved the common understanding that these skills cannot consistently improve through traditional curriculum and academic program setting. Consistent instillment is the pursued pedagogy of fostering long-lasting soft skills.

PBL in Undergraduate Business Education

An increasing number of instructors have recognized the importance of PBL and the implementation of PBL into various courses and curricula, through which evidence has been observed regarding the benefit of PBL in various domains and subjects in undergraduate studies, including medical education, biomedical engineering, chemical engineering, software engineering, and thermal physics. PBL has shown its effectiveness in the above fields, but studies also found that the connection between PBL and business and economic studies are tenuous, although these domains are highly practical.

Jones and McMaster ( 2004 ) applied PBL in a 3-year undergraduate Information Science degree program, where students across all years formed groups to solve real-life business problems, suggesting the possibility of successfully reconciling academic learning objectives with real-life project demands in business studies. However, Bigelow ( 2004 ) pointed out that simply implementing PBL in improving students’ problem-solving skills may not be sufficient after students graduate and face organizational problem-solving. More steps are in demand to further prepare students to solve real-world problems and collaborate with others. Brundiers, Wiek, and Redman ( 2010 ) proposed that collaborating with community and industrial partners provides students real-world learning opportunities, which emphasized the importance of connecting the learning experience closely with the real world. Martínez León ( 2019 ) reported teaching an engineering course with the PBL model through the execution of Lean Six Sigma (LSS) projects implemented through university–industry partnerships. This study reported that such model facilitated the integration and application of theoretical knowledge through the development of professional skills in undergraduate students, as demanded by business partners and organizations. However, the feasibility and the validity of such pedagogical exploration applied at the academic program level remain unclear.

Miller, Hill, and Miller’s ( 2016 ) study provided a good example of applying PBL in an undergraduate supply classroom to introducing Lean Six Sigma by providing students an ill-structured business problem in a setting students are familiar with, which focused their learning efforts on technical mastery of concepts and tools. It helped cultivating critical thinking, teamwork, and project management beyond the content of the course. Miller, Hill, and Miller ( 2019 ) further investigate PBL as in-class simulation to teach operations and process improvement concepts and emphasized problem-solving, teamwork, and intra-firm cooperation for large, semester-long process improvement projects for multiple student groups, which expand the application and instructional design of PBL.

In undergraduate business education, research on the application of PBL in curriculum and instructional design beyond one course and one semester is highly desired in order to explore the skills improvement, curriculum integration, and the best practices of adopting PBL as the main pedagogical strategy in undergraduate education.

Cohort-Based Learning Model

As educators become more aware of the importance of collaboration, an increasing number of educators and educational researchers have begun to appreciate the educational benefit of cohort-based learning. Cohort-based learning provides students opportunities to foster their creativity and innovation, as these skills are not as readily utilized in other instructional contexts. In the cohort-based learning model, students form long-lasting groups that allow them to extend the learning and collaboration beyond the classroom thus fostering long-term friendships, which indirectly contributes to keeping the drop-out rates to a minimum. Although cohort-based learning poses special requirements for program structure and curriculum design, the benefits of this model is evident (Saltiel et al., 2002 ).

PBL and Cohort-Based Learning

PBL being adopted and implemented as the major pedagogy program-wise is not commonly seen. PBL implementation throughout a cohort-based undergraduate degree program is an innovative design in post-secondary education. Throughout the documented literature, Jin et al. ( 2019 ) reported their achievements in adopting PBL in cohort-based classes of an undergraduate science academic program and found that students who underwent this model showed increased awareness and interest in solving problems. The program itself showed increased enrollment, doubled retention rate, and decreased average time of completing the degree by almost 2 years. However, such examples are uncommon, and little is known about the instructional design, student learning transition, and connection to the real world.

Based on the research findings listed above, one of the objectives for the current study is to seek to understand how to better prepare students to be job-ready by integrating industry partnerships and curriculum into a cohort-based PBL instruction for a new Bachelor of Science degree in Project Design and Analysis (PD&A) at University of North Texas (UNT) Frisco campus.

Methodology

The current chapter proposes to apply an exploratory case study to examine the impact of a PD&A undergraduate degree with PBL and cohort model as the core feature implemented on UNT Frisco campus. Case study is useful for the study of individual case or cases within a current, real-life context or setting. Given the timeliness and ongoing feature of the PD&A program, exploratory case study helps to identify causal relationships among factors and develop explanations for the ongoing scenario (Maxwell, 2004 ).

Overview of the PD&A

The initial motivation when designing the PD&A degree program with PBL and cohort learning design was to provide a feasible undergraduate degree plan that addresses the three major issues faced by many current US higher education institutes: cultivating employer-desired skills, lowering student drop-out rate, and reducing financial burden. Hence the goals of the PD&A program are to 1) empower the students in applying the knowledge gained in classroom into real life in timely fashion, 2) be “robot proof” and job-ready for their upcoming career after graduation, and 3) allow students to complete their undergraduate study with reasonable costs in 3 years. The new campus of UNT at Frisco, Texas, was assigned as the incubator for the launching of the program. The journey of PD&A design was initiated in the summer of 2018 with the objective of admitting the first cohort of students in the fall semester of 2019.

The key desired outcomes for the degree program are to 1) apply classroom knowledge in solving real-world problems, 2) prepare students to be job-ready for their upcoming career after graduation, 3) have high-demand employable skills, 4) have reduced cost of education (without lowered tuition fees), 5) be better connected with faculty and staff. The method that was chosen to achieve these objectives was to create a problem-based learning program with a cohort learning model that allows students to graduate in 3 years from UNT, a Tier 1 public majority minority university in Texas.

Program administrators and faculty consider the journey of the PD&A program, in regard to problem-based cohort learning (PBCL) model, to be an educational design-based research (DBR). A key characteristic of DBR is that the educational ideas for student or teacher learning are formulated in the design, but can be adjusted during the implementation of these ideas. Program administrators will continuously update and refine the framework and the course design based on the feedback received from the cohort students, partners, faculty, and staff. From the experience of Cohort 1’s program in the first year, changes are being made for the upcoming Cohort 2 students for fall 2020. More degree programs are being launched based on the developing PBL framework in UNT Frisco.

The PD&A degree designed with block scheduling enables students to complete the program in 3 years. This lays the foundation for students to save money by paying fixed tuition and fee rates for regular semesters, and the reduced tuition fee in the summer based on the existing “save and soar” tuition plan at UNT. Cohort students can also jumpstart their career 1 year sooner while eliminating all expenses surrounding education, which are not tuition and fees such as transportation and meals.

Five Pillars of PBL Cohort Model

This degree program is designed with PBL and cohort model as the core element with a foundational belief in “partnership” and four additional pillars supporting the degree program, as shown in Fig. 1 . Successful partnerships with academic colleges, student services, industry (organizations), student organizations, support organizations, students, staff, and faculty are integrated in the only way this program will be successful. The approach of this chapter is to introduce each of the elements shown in the diagram below in detail and share the lessons learned throughout the first-year journey, while discussing the changes that will be made to the program going forward.

Problem-based learning cohort model

Pillar 1: Problem-Based Learning Cohort (PBLC) Model

PBLC model is illustrated in Fig. 1 . In order to create a culture of collaboration, the concept of cohorts was added to the PD&A program – all the students start and graduate at the same time. Miller, Hill, and Miller ( 2019 ) suggest a class size of 20–30 students to be appropriate for PBL classrooms. The program hence builds cohorts of approximately 24–30 students to help students fully engage in collaboration with other students and allow faculty to provide sufficient attention and assistance to all.

In PD&A, a cohort of students are admitted every fall semester; they take core and major classes taught with PBL as major pedagogy (93 credit hours), along with required applied seminars (6 hours), elective courses (15 to 18 credit hours), and three internships (3 to 9 hours) in 3 years for a total of 120 hours. Students take approximately 17 credit hours for six 16-week semesters and 6 credit hours for three 8-week summer semesters. The distribution of credit hours is shown in Fig. 2 . One of the potential disadvantages of the cohort model is that once students drop out or transfer to other institutes, it is challenging to rejoin their original cohort. The program was designed to keep the option of receiving transcripts by semester with individual grades for each class in order to allow students to transfer to or from other programs (Fig. 3 ).

PD&A credit hours distribution

Sample project design and analysis degree program overview for 3-year period

The PBCL program marketing was formally launched in February 2019 for Cohort 1 to start classes in fall 2019 with 23 students signed up. The program was launched with three Texas Core classes in History, English, and Psychology plus a Project Connections class on collaborative thinking, an applied seminar, and one elective of 3 hours. In spring 2020, the students had four Texas Core Classes, Connections class on Professional Communications, and an applied seminar.

The challenge to delivering this concept arises from the fact that the New College Frisco campus had to bring in a large number of faculty members from different academic units (colleges, departments) and student service units (student services, recruitment, etc.) to collaborate, as the objectives, measurements, and agendas of each of these units could be difficult.

Pillar 2: Integrated Academic Curriculum

The degree was built on the framework of an existing undergraduate BS degree in Integrative Studies with three concentrations: project, design, and analysis. Project courses are designed as connection courses; the connection generated from these courses lays solid ground for partnering with industries and conducting learning in projects for students. The design and analysis courses along with the applied seminar serve as the other major classes for this degree. In the course design of the degree, eight competencies that employers want in graduate students and the key methodologies and certifications that employers are using to train their employees were instilled in the course curriculum with methodologies and tools such as StrengthsFinder, SCRUM, Lean Six Sigma, Agile, PMBOX, Social Styles, and DiSC, which are used by employers to improve the skills of their staff in major corporations around the globe. Additionally, by including the technological components into the instruction such as Microsoft Office, Teams, Slack, JIRA, Google Docs, and mini-tab, students are prepared for the digital capacity of future jobs.

A connection class in Collaborative Thinking with a focus on Six Sigma given in the first semester is taken by students with the intent to provide the competencies and skills that allow the students to work on a real partner problem from the first semester in a language that most businesses understand. The partner problem then serves as the center of the integrated curriculum from the first semester allowing all the semester classes to revolve around the project with the exception of electives and the applied seminar as shown in the Fig. 4 .

Major and core classes revolve around the project

The faculty of the PD&A program work together as a team to connect individual courses in the Core Curriculum to integrate around the partner project, as well as collaboratively developing integrative syllabus and course work. One example of the integrated curriculum from the fall 2019 semester is when the partner project revolved around mobility, including transportation challenges and driverless cars. In the history course, student learnings from prior to 1865 included the migrations of Mayans to associate with population growth in the City of Frisco as well as the introduction of railroad as a new mode of transportation for that time along with the acceptance criteria from the people at that time. In psychology, the discussions included what constitutes the psychological needs of people when they move and what facets one needs to consider when selecting public transportation routes and vehicles, such as disability. In English, the assigned articles were pertaining to mobility, and most of the writing assignments were focused around mobility, smart cities, etc. All the initial reading and class assignments were revolved around using Six Sigma tools on mobility use cases until students were prepared for being in a position to transition to the class mobility project with the City of Frisco (COF).

The connection course of fall 2019 titled “Collaborative Thinking” focused on adopting the Six Sigma methodology of the DMAIC (Define, Measure, Analyze, Improve, and Control) process. Additionally, the students built a code of conduct for their team; learned the Bruce Tuckman five-stage development process that most teams follow of norming, forming, storming, performing, and adjourning; and learned how to run effective meetings with clear objectives, action logs, and parking lot. The Six Sigma toolbox also exposes students to tools like scope development, interviewing techniques, process mapping, stakeholder management, data stratification, fishbone diagrams, 5 Whys, Gemba boards, and presenting skills. By doing so, students are expected to establish a good foundation for team building and an understanding of appropriate tools on the partner project. All the faculty in the cohort program facilitate the students to build their skills of collaboration and communication.

Assignments given in one class can be leveraged in other classes. For instance, in English the students learn to write good business memos that are presented to the client along with the client presentation three times in the semester. The business memo is graded in the English class, as well as the Connections class. Another example is that the students learned how to take surveys in the psychology class, and the results of the survey were analyzed and stratified with the help of the psychology professor. Later the results were used in the connections class in designing the routes and presented to the partner in the final presentation.

Pillar 3: Organizational Partnership

Working with real-world business partners can engage students with authentic real-world problems. The pillar named “Organizational Partnership” comes from the partnership established in this program with the Industry, Government institution, or a non-profit organization. The program has identified a Time, Treasure, and Talent partnering model for organizational partnership (3TPM) with expectations identified in each of the categories as shown in Fig. 5 .

3T Partnering model (3TPM) for PBL organizational partnerships

Guest Lecture - Industry partners are invited to provide four 45-minute workshops as applied seminar classes each semester on campus or at the organizational partners’ sites to share their experiences with what the students are learning, such as project management, Six Sigma, transportation, and communication. These workshops serve several purposes: it is an opportunity for the students to see how what they have learned in class can be applied in real life with real examples; it exposes students to industry personnel for potential internship opportunities, and it allows the students to learn how to network with industry professionals.

Project Sponsorship – Project sponsorship partners are required to commit to sharing their talent with academia to help the students learn by applying what they have learned in class with the project. In return, the partners benefit by receiving free consultation on a business challenge that they might be struggling with. The first-year partner, the City of Frisco (COF), and the second-year partner, nThrive, helped us in 1) providing a real business scenario or business problem for the students to work on, 2) committing and coming into the class three times, and 3) providing access to facilities, people, data, and policies for the students to acquire additional knowledge and providing deep insights on the project.

Of the three visits, the first was at the beginning of the semester to provide an overview of the business scenario/problem, a second visit to review student progress, and a third visit to attend the final presentation at the end of the semester. Occasional visits were made during the semester as well.

There are five scenarios that faculty could potentially use to work with the students in the PD&A program (shown in Fig. 6 ):

Fictitious Projects: Projects created for the students based on research. The project outcomes are likely to be predetermined and known to the faculty.

Case Study: Faculty use case studies from databases like Harvard Business Review to create projects for the students. These are sometimes used to help derive the background for the anchor project and are typically done with results already defined.

Case Interview: The partner company recaps an event that has happened in the past and creates a project with the faculty by taking out the name of the company and the parties involved in the execution of the project and modifying the data, in order to create a scenario for the students to work on.

Business Scenario: The faculty work with companies on projects that are based on real business needs and without a predefined answer. In business scenario projects, some dependency on the client exists, and student contact is not overwhelming. In this scenario, the project teams in the class are competing to conduct research on competing companies, products, performing surveys, or interviews to provide different perspectives to the partner on the most ideal business scenario.

Business Problem: These projects revolve around industry business problems and have all the traits of the business scenario, but has medium to high dependency on the partner. In this scenario, the partner provides a large complex business problem, and students work in small teams to deliver a solution to the client. More than one of the stakeholders have to be involved in the student journey to make the project meaningful for the students as well as the recommendation meaningful for the partner. As shown in Fig. 6 , the projects students worked on with COF are categorized as P3 (Mobility project in fall 2019) and P4 (Business Process Improvement project in spring 2020) accordingly, which indicates that the partner involvement levels in these projects are high.

5 Methods of incorporating projects into PBL curriculum

Advisory Board – An organizational partnership advisory board is intended to be launched with representatives from Industry and non-profit organizations in fall 2020 or spring 2021.

Internships: Internships with organizational partners are so important to the success of the program that they are considered to be one of the five pillars of the program.

Pillar 4: Student Services Integration

The fourth pillar, integrated student services, is intended to provide students full support for their success in PD&A program (Fig. 7 ).

Organization and academic partnering opportunities in PBL

Different from the traditional applied seminars hosted by Student Services with no grades assigned, PD&A applied seminars are designed to facilitate mutual understanding between faculty and students on the course load and assigned 1 credit hour with 3 hours of contact time, which gives students incentive to attend the seminar. The applied seminars in the first semester guide students starting their college journey by providing them life and learning skills. The focus areas of the student services classes include building life skills, such as coping with stress, timely delivery when the stress is high – especially after week 4, when many tests come up at the same time.

Fostering time management, personal finance, and learning strategy skills.

Preparing students to be internship ready by teaching resume writing, LinkedIn profile development, etc.

Preparing students to be job search ready by teaching interviewing techniques, dressing for success, mockup interviews, etc.

Student services and academic services work together in this program by focusing on helping students build life skills over 3 years and bringing professionals from the industry so the students can get firsthand feedback and be ready for their future business career. Providing opportunities for the students to start building a network of professionals for their career can equip students with a cutting-edge advantage for job market competition.

Pillar 5: Internships

The final pillar in the model is internships. Partnering companies work with academia and student services to provide internships so the students can obtain applied experience. The partners benefit from receiving students with great skills at a low cost while they are still in school, and full-time well-prepared talents upon graduation. Three different internships with a total 9 credit hours in 3 years are required before graduation for each student. The internship curriculum focuses on industry partners helping the university to provide opportunities for the students to gain experience in the eight NACE job-ready competencies as previously mentioned, as well as leveraging the learnings from the classroom and projects.

UNT’s external organizational partners along with UNT’s academic internal partners, such as Finance, Student Services, and other administrations of UNT, collaborated with New College to provide internships for almost all of the students in the program.

Discussions

Exploring the PBL pedagogy in cohort model is an innovation in reformation for higher education. As of now, the following experiences and lessons are learned. First, how to manage dual credit in the cohort program. The original plan was that students with dual credit would only audit the class without enrolling and paying, since the learning from the class would apply to the project. However, without incentive to get grades, the students were not motivated to be in class and maintain scholarships without full-time enrollment status. The stance in the program was changed to welcome students to attend core classes without mandatory attendance requirement, even though portions of the class content would apply to the project. Administrators of the program select teams with enough representation from all classes to fill the gap of missing skills on the project. Additionally, most of the students that came in with dual credit had taken classes in English, History, or Political Science, which are mandatory core classes in Texas. In order to address this challenge, these six classes have been split over six semesters to balance the class loads for students.

The second learning was the challenge of the Connections class. The Connections class was designed to be a lab with 3CHR of credit and 6 hours of contact time. This was also the class where all faculty attend the class together. This class had created a number of challenges: 1) increased work load for faculty members – a lecturer in Core was using 50% of their load catering to a cohort of 25–30 students; 2) ineffective time use – having four to five faculty members in a class prevent them from working on their other duties; 3) keeping the students in the classroom for too many hours. To address these challenges, the curriculum committee allowed faculty to shift some course content to the lab (example 6 Sigma methodology for collaborative thinking), adjust the class from 3 CHR to 4 CHR, and match the contact time to the credit hours. Another change to be made is to have all the faculty meet for 30 minutes prior to the weekly Connections class. The implementation of this meeting could provide more flexibility to all the faculty teaching in the PD&A program who are dual appointed in two colleges and increase the efficiency in the integrated curriculum design by focusing on the themes of the projects, and entitle faculty to plan their classes around the themes with a couple of touchpoints instead of managing all the details of the integration. Although significant time was spent planning the integration, the team came up with a similar course cycle with the traditional courses. The weekly 30-minute faculty meeting can not only help increase the work efficiency for faculty, but also help students to achieve greater success.

The faculty and staff are in the process of implementing the second learning by implementing a new class integration partnering model (CIPM). In this model, each faculty participates in the decision of how much integration they will participate in for the semester based on the provided project. The integrated curriculum methodology is shown in Fig. 8 . The methodology has four one-time events: project overview, level of integration discussion, weekly syllabus adjustment, and reflections. The methodology also has three recurring events over the semester: 63-minute touchpoint, 16 invites to the lab, and 4 client presentations.

PBL Integrated curriculum methodology (ICM)

The third learning is the time limitation of executing real problem projects. In the sixteen weeks of the semester, students completed project design, measured current trends for analysis and then suggested recommendations for implementation with measurement controls for success for the project at COF. The timeline of sixteen weeks could be too short for executing projects. The students would have a better learning experience if these learning items are spread over two semesters. Built on the empirical evidence, the quantity of projects remained the same, but the design and continuation of the project into two semesters is one of the changes that was incorporated. Taking projects to completion with proper analysis will improve the learning experience of the students in the current and future cohorts.

The fourth learning is the requirement of re-thinking the students and faculty evaluation in the PD&A program. The PBL pedagogy differentiates the UNT at Frisco campus from all other programs at UNT main campus that follow the traditional pedagogy of teaching. At Worcester Polytechnic Institute (WPI), an experienced PBL college, there are no prerequisites, and the minimum grade a student gets is a C, which would be more relevant in Frisco, yet we follow the traditional pedagogy policies from Denton. At UNT Frisco’s PBL program, students work with partners on real business problems that do not have a predefined answer, which requires significantly more time and effort for the student as well as faculty. Additionally, student teams work intensively with faculty, especially for three partner presentations that happen over the course of each semester; hence the grades, faculty load, and sequences of classes should all be reviewed from a PBL pedagogy perspective vs. the traditional pedagogy.

The last major learning is that a better model is needed to collaborate with the organizational partners. This is the first time that UNT worked with a PBL model to involve organizational partners, like nThrive, for long-term projects, which entails a steep learning curve for both parties. UNT needs to have a more efficient model for partnering with both organizations to develop long-term partnership for curriculum integration, internships, and guest lectures.

Interviews were conducted by media from UNT, Frisco, and the Dallas–Fort Worth metroplex, as well as independent observers and researchers, who collected some empirical evidence from the Cohort 1 students, faculty, staff, and the industrial partners. Some representative quotes from each category of contributors that provide insights from the participants of the program and reveal their experience are listed below. Some quotes from students are:

“It has just really been an awesome experience so far like I have really gotten a lot more experiences with the real world and business, I had not imagined myself doing this as I graduated from high school.”

“I am really excited to kind of experiences we get with some of the businesses around Frisco and be able to work with them more hands-on and I am really excited to be working with a closer-knit group of people.”

“Whenever I interact with my professors, any question I ever have, anytime I ever have a problem, or I need to talk to them, they are always there, super supportive.”

These quotes reflect that students appreciated the length of the program being 3 years, a bachelor’s degree in science, equipped with a lot of real-world hands-on experience for them in a close-knit group of students, faculty, staff, and industry partners. Besides, the partnership design has played an important role in students’ learning experience and vividly elaborated the essence of PBL. These quotes also reflect the close relationship among students and student-faculty, brought by the cohort model. Familiarity and friendship are the foundation of collaboration, which is one skill that employers seek in all graduates.

As for the PBL program model, here are some of the quotes from faculty:

“I really love the program, it is great to watch the interaction between students and how they work together while they are building life skills in team work, in collaborative thinking, leadership, and project management at a very young age.”

“What you learned was in the service of connecting the content that you gained the knowledge and skills that you gained with real-world skills that could actually have an impact around you.”

Faculty participating in this program also clearly see the beneficial features of the program and enjoy the benefit of working closely with students and teaching with full engagement. From implementing this program, the faculty, staff, and administrators are discovering new paths and possibilities of undergraduate program design in US higher education institutes. This lays the foundation for developing a transformational program and will provide opportunities for building the empirical evidence for applying PBL and cohort program as the UNT Frisco’s instructional pedagogical strategy.

Further Studies and Conclusion

The launch of the PD&A program is in its infancy at the UNT Frisco campus, and the plan is to continue this longitudinal study with design-based research to constantly reflect on its results. Based on the objectives of the program, as mentioned in this chapter, there is a need to measure if the students are ready for jobs on day 1 by conducting a comparative study on the enhancement of soft skills and/or NACE competencies for the students in the PD&A program compared to their peer group in traditional pedagogy. More comparative studies are also needed on job placement between peer groups and the performance the students have after they start their jobs.

The development of business cases and the evaluation scale of Cost Benefit Analysis of PD&A based on cohort size of 25–30 students are also desired to see the retention levels and student partnering after graduation, in order to evaluate the impact of PBL and cohort model on the initial objectives. The scaling should include organization models, partnering models, faculty readiness, traditional faculty workload measurements, student services integrations, etc.

Two of the pillars in the PBL Cohort Model require Higher Education Institutions to partner with organizations for the organizations to contribute to the areas of Time, Treasure and Talent. Future studies should be conducted on the development of a long-lasting organizational partnering model that allows both the educational institution and the organization to benefit from this partnership.

There is a firm belief that when a student from a problem-based learning cohort is able to experience six projects over six semesters and three internships with partners working in a project-based learning environment where the role of the faculty is that of a of a facilitator and a network of industry professionals, they will be the movers and shakers of the twenty-first century.

Bauer-Wolf, J. (2019). Survey: Employers Want 'Soft Skills' From Graduates. Retrieved from https://www.insidehighered.com/quicktakes/2019/01/17/survey-employers-want-soft-skills-graduates .

Bigelow, J. D. (2004). Using problem-based learning to develop skills in solving unstructured problems. Journal of Management Education, 28 (5), 591–609.

Google Scholar  

Brundiers, K., Wiek, A., & Redman, C. L. (2010). Real-world learning opportunities in sustainability: From classroom into the real world. International Journal of Sustainability in Higher Education, 11 (4), 308–324.

Article   Google Scholar  

Jin, L., Doser, D., Lougheed, V., Walsh, E. J., Hamdan, L., Zarei, M., & Corral, G. (2019). Experiential learning and close mentoring improve recruitment and retention in the undergraduate environmental science program at an hispanic-serving institution. Journal of Geoscience Education, 67 (4), 384–399.

Jones, M. C., & McMaster, T. (2004). Addressing commercial realism and academic issues in group-based IS undergraduate project work. Journal of Information Systems Education, 15 (4), 375–381.

Martínez León, H. C. (2019). Bridging theory and practice with lean six sigma capstone design projects. Quality Assurance in Education, 28 (1), 41–55.

Maxwell, J. A. (2004). Using qualitative methods for causal explanation. Field Methods, 16 (3), 243–264.

Miller, K. E., Hill, C., & Miller, A. R. (2016). Bringing lean six sigma to the supply chain classroom: A problem-based learning case. Decision Sciences Journal of Innovative Education, 14 (4), 382–411.

Miller, K. E., Hill, C., & Miller, A. R. (2019). Applying performance measures and process improvements in the classroom: The loch dots company simulation. Decision Sciences Journal of Innovative Education, 17 (4), 302–323.

Peck, A. (2017). Engagement and employability: Integrating career learning through Cocurricular experiences in postsecondary education . NASPA-Student Affairs Administrators in Higher Education.

Saltiel, I. M., Russo, C. S., & Dawson, J. (2002). Cohort programming and learning: Improving educational experiences for adult learners. The Canadian Journal for the Study of Adult Education, 16 (2).

Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist framework. Educational Technology, 35 (5), 31–38.

Wijnia, L., Loyens, S. M., & Derous, E. (2011). Investigating effects of problem-based versus lecture-based learning environments on student motivation. Contemporary Educational Psychology, 36 (2), 101–113.

Download references

Author information

Authors and affiliations.

Learning Technologies Department, University of North Texas, Denton, TX, USA

Zain Ali, Nanxi Meng & Scott Warren

Department of Learning Technologies, University of North Texas, Denton, TX, USA

Lin Lin-Lipsmeyer

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Zain Ali .

Editor information

Editors and affiliations.

J. Michael Spector

Instructional Design and Technology, Virginia Tech School of Education, Blacksburg, VA, USA

Barbara B. Lockee

Marcus D. Childress

Section Editor information

Massachusetts Bay Community College, Wellesley Hills, MA, USA

Bernadette Sibuma Ed.D.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Ali, Z., Meng, N., Warren, S., Lin-Lipsmeyer, L. (2023). Integrated Problem-Based Learning: A Case Study in an Undergraduate Cohort Degree Program. In: Spector, J.M., Lockee, B.B., Childress, M.D. (eds) Learning, Design, and Technology. Springer, Cham. https://doi.org/10.1007/978-3-319-17461-7_170

Download citation

DOI : https://doi.org/10.1007/978-3-319-17461-7_170

Published : 15 October 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-319-17460-0

Online ISBN : 978-3-319-17461-7

eBook Packages : Education Reference Module Humanities and Social Sciences Reference Module Education

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

An integrated ethical decision-making model for nurses

Affiliation.

• 1 Department of Nursing, Kyungwon University, San 65 Bokjeong-Dong, Gyeonggi-Do, Korea. [email protected]
• PMID: 22156941
• DOI: 10.1177/0969733011413491

The study reviewed 20 currently-available structured ethical decision-making models and developed an integrated model consisting of six steps with useful questions and tools that help better performance each step: (1) the identification of an ethical problem; (2) the collection of additional information to identify the problem and develop solutions; (3) the development of alternatives for analysis and comparison; (4) the selection of the best alternatives and justification; (5) the development of diverse, practical ways to implement ethical decisions and actions; and (6) the evaluation of effects and development of strategies to prevent a similar occurrence. From a pilot-test of the model, nursing students reported positive experiences, including being satisfied with having access to a comprehensive review process of the ethical aspects of decision making and becoming more confident in their decisions. There is a need for the model to be further tested and refined in both the educational and practical environments.

Publication types

• Research Support, Non-U.S. Gov't
• Attitude of Health Personnel
• Decision Support Techniques*
• Ethics, Nursing*
• Models, Nursing*
• Nursing Evaluation Research
• Pilot Projects
• Students, Nursing / psychology

The development of situational problem-based learning model integrated with technological pedagogical and content knowledge in teaching mathematics for grade 10 students

• Sorntachoti, Kampien
• Buaraphan, Khajornsak

Problem-based Learning (PBL) is widely used in teaching mathematics in Thailand. However, Situation-based Learning (SBL) is rare in mathematics education literature in Thailand. The combination of PBL with SBL will bring students to learning through solving the problem dealt with the situation being relevant to students' real lives. In the high technology world of 21st century, teachers are also demanded to attain Technological Pedagogical and Content Knowledge (TPACK). In addition, the COVID-19 pandemic situation forces teachers to apply several technologies in teaching especially on-line. So that, the integration of TPACK with Situational Problem-based Learning to become the TPACK-SPBL model is needed. This article aimed to develop the TPACK-SPBL model for teaching mathematics for Grade 10 students. The authors analyzed the literature related to PBL, SBL and TPACK and then synthesized the three bodies of knowledge into the TPACK-SPBL model. The TPACK-SPBL model was consisted of seven teaching steps: Step 1: Introduce the situation in targeted context, Step 2: Identify the creative problem in context, Step 3: Plan way to solve creative problem, Step 4: Solve the problem creatively, Step 5: Synthesize and present knowledge, Step 6 Apply knowledge in new context, Step, and Step 7: Summarize and evaluate. The seven teaching steps according to the TPACK-SPBL model was utilized to design the lesson plans in the topic of Set for Grade 10 students in mathematics. The TPACK-SPBL lesson plans were sent to a panel of five experts to check their content validity through the Index of Item-Objective Congruence (IOC). The calculation of IOC showed that the TPACK-SPBL lesson plans were qualified with their IOC higher than 0.60. Finally, the authors provided an example of a lesson plan that used the TPACK-SPBL model to teach grade 10 students about sets.

• MATHEMATICS EDUCATION