Portions of this article are directly reprinted with permission from this original blog post by Regional Educational Laboratory (REL) Southwest. We are grateful to the authors, REL Southwest, and the Institute of Education Sciences (IES) for allowing us to help spread the word about this new tool with you!

There is limited literature on how to formatively assess the effectiveness of research-practice partnerships (RPPs). More tools and processes are needed so that RPP members, regardless of how long the partnership may be in place, are able to use their time more efficiently and to accelerate progress toward achieving their goals. We identified these needs in our own work, and developed a new tool to address these needs, which we introduce here. 

REL Southwest partners with state education agencies, school districts, and other educational organizations to address pressing education issues through RPPs in Arkansas, Louisiana, New Mexico, Oklahoma, and Texas. Our website’s partnership pages list partnership members and describe the focus of work in each state. After year one of the RPPs, we identified a need to improve their participation, engagement, and overall functionality. To inform our next steps, we needed a research-based tool to assess RPP health. For this reason, REL Southwest developed the Tool for Assessing the Health of Research-Practice Partnerships.

How Did We Develop This Tool?

As the basis of the new tool, REL Southwest used an existing framework defined in Assessing Research-Practice Partnerships: Five Dimensions of Effectiveness (2017) by Henrick, et al., which readers of this magazine might be familiar with through previous articles discussing different aspects of this framework (see here, here, and here, for example). The Henrick, et al. (2017) framework provides dimensions and indicators that researchers and practitioners must address to build and sustain an effective RPP. The authors of this framework intended the indicators to serve as a guide for future RPPs to develop more specific measures of RPP effectiveness, and the framework was intended to be customized for specific RPP needs.

In addition to meeting the research community’s need to address the challenge of RPP evaluation and improvement, the tool provides a means for participation by practitioner partners. Teachers, administrators, and other practitioners can use the tool to identify challenges and issues with the RPP’s process and goals, ensuring they have a voice in its assessment and continuous improvement. REL Southwest developed the new tool for individuals—researchers or practitioners—who want to formatively assess and continuously improve an RPP in collaboration with its members. Together the partners set short- and medium-term measurable goals, establish processes and structures to support these goals, and reflect on the degree to which their work accomplishes these goals.

How Can You Use This Tool?

The REL Southwest Tool for Assessing the Health of RPPs includes a guidance document and an Excel workbook that consists of three parts. The tool’s components are also outlined in this infographic.

The tool guidance includes information on how to use each of the workbook’s three parts, including:

1. Guidance to prioritize dimensions and indicators.
2. Guidance to establish short- and medium-term goals and measures.
3. Interview protocol designed to help conduct interviews with the RPP members to assess the health of prioritized dimensions and indicators.

Individual RPP members may use the tool effectively without a significant investment of time, even if they have limited experience evaluating or participating in RPPs.

By using a formative lens rather than a summative one, the tool encourages RPP members to purposefully and honestly reflect on their collaborative work and to make necessary adjustments and improvements over time to achieve the partnership’s intended outcomes. Using the tool at the onset of a partnership could help ensure all members have a common understanding of the process and goals as the partnership gets underway. Periodic assessment could help keep the RPP focused on goals or identify a need to adjust them. By conducting the interviews with RPP members, evaluators can discover additional needs that the partnership may not be meeting for its members.

An Example of How the Tool Was Used and Adapted in an RPP

The REL Southwest team at American Institutes for Research (AIR) decided to pilot this tool with an RPP outside of the REL program. The goal was to see if the tool and related processes would be feasible in an RPP with a single district, one of the predominant configurations of RPPs across the country. Here is a brief overview of how the RPP used the tool and of some of the outcomes associated with its use.

PUMP-CS RPP pilots the tool. Preparing Urban Milwaukee Pathways in Computer Science (PUMP-CS) is an NSF-funded Computer Science for All RPP. We worked with them starting in 2019 to pilot our tool: We first introduced participating RPP members to the Henrick et al. (2017) framework so they could familiarize themselves with the five dimensions and the indicators for each dimension, then used our tool to facilitate a process for the RPP members to collaboratively determine quarterly priorities for the RPP based on the framework’s dimensions and indicators.

The RPP enthusiastically decided to prioritize all five dimensions and prioritized at least one indicator per dimension to focus on in the first quarter. AIR used the tool’s protocol to interview each member of the RPP leadership team (2 district staff members and 1 researcher and 1 technical assistance partner). After discussing the interview results, the RPP leadership team identified the following action items aligned to the prioritized indicators listed below.

Dimension 1: Building Trust and Cultivating Partnership Relationships

Prioritized Indicator: Researchers and practitioners routinely work together

• Action item for PUMP-CS: Continue to develop the partnership’s logic model during the RPP meetings and share it with other stakeholders (i.e., the partnership’s steering committee) to further inform its development as well as establish a shared understanding of the partnership’s efforts and activities.

Prioritized Indicator: The RPP establishes routines that promote collaborative decision making and guard against power imbalances

• Action item for PUMP-CS: Establish bi-weekly touchpoints for the RPP leadership team and ground conversations in the logic model so the prioritized activities and outcomes inform key decisions.

Dimension 2: Conducting Rigorous Research to Inform Action

Prioritized Indicator: The RPP establishes systematic processes for collecting, organizing, analyzing, and synthesizing data

• Action items for PUMP-CS: 
  • Develop a timeline for district data requests
  • Identify the audiences for the data analyses and establish a portal for aggregating data so the RPP may access findings on an as-needed basis
  • Establish processes that invite the district to co-interpret results

Prioritized Indicator: Findings are shared in ways that take into account the needs of the practice organization

• Action item for PUMP-CS: Identify existing research that may address the district’s immediate needs while waiting for findings from current data collection efforts

Dimension 3: Supporting the partner practice organization in achieving its goals

Prioritized Indicator: The RPP provides research and evidence to support improvements in the partner organization

• Action item for PUMP-CS: Collaboratively articulate measurable, timebound short- and medium-term goals for each prioritized indicator that align to the logic model

Dimension 4: Producing knowledge that can inform educational efforts more broadly

Prioritized Indicator: The RPP develops two dissemination plans, one that supports partnership goals and a second plan for broader dissemination

• Action item for PUMP-CS: Identify which district stakeholders need to increase their understanding of the partnership’s efforts and develop a strategy for communicating with them

Lessons learned. Meaningful conversations take time. Prioritizing dimensions, selecting indicators, and setting SMART (specific, measurable, achievable, relevant, time bound) goals required up to 1.5 hours of meeting time. Rather than rush these conversations and gather feedback via interviews every quarter, in the second year of the partnership, the leadership team decided to use the tool every 6 months and to prioritize 2-3 dimensions and a subset of their indicators during that 6-month period. This adjustment allowed the RPP to accomplish more significant, targeted work with fewer action items needing to be pushed to the following 6-month period.

In addition to interviewing the RPP members every six months, AIR also added a brief survey modeled after the rubric developed by Henrick et al. called “Are We a Partnership Yet”? This helps to serve as a quick temperature check of how each partner is feeling about the different dimensions at play. The survey data along with the themes from the qualitative data are discussed every six months. According to the most recent survey findings, the RPP members reported that the RPP is on track toward maturing on the “building trust and cultivating partnership relationships” dimension. Feedback during the interviews suggested that: (1) the RPP members believe the partnership is changing the status quo of CS instruction in the district and (2) the diversity of expertise strengthens the partnership.

PUMP-CS RPP members’ reflection on the use of the tool

“I think that the structure that you guys are putting in place here is really essential. It’s going to change the way we do business, and I think that’s really important.”

– RPP district staff member

“It has been a breath of fresh air to have another partner joining us in this work, and to actually get us to look up…at the larger context of the project. That’s been something that has been fun and informative.”

– RPP researcher representative

In Conclusion

We hope that by providing guidance on how the dimensions and indicators of the Henrick et al. framework can be used to establish goals and monitor how an RPP is progressing towards these goals, our tool will be useful to individuals working in RPPs that wish to assess the health of their partnership.  The REL Southwest team invites you to contact us with questions or to share your experiences using the tool.

Elizabeth Barkowski is a senior researcher and Carrie Scholz is a principal researcher at American Institutes for Research.

Suggested citation: Barkowski, E., & Scholz, C. (2021). Measuring RPP Health: Introducing a New Tool Based on the Five Dimensions of RPP Effectiveness. NNERPP Extra, 3(4), 11-13.