“[We] can’t continue to bet tens of millions of dollars each year on RPPs without a better sense of what they are doing, what they are accomplishing, and what factors are associated with their success.”

– Mark Schneider, Director, Institute of Education Sciences

(July 30, 2018)

In late July 2018, shortly after assuming the role of Institute of Education Sciences (IES) director, Mark Schneider announced at the National Network of Education Research-Practice Partnerships Annual Forum that he would like to see more evidence of positive impact on educational practice from recent IES investments in research-practice partnerships (RPPs). As executive director of one of three RPPs that Schneider identified as arguably the most successful in the nation, I truly believe in the value of RPPs but, admittedly, struggle to quantify our positive impact.

Place-based RPPs change their local ecosystem by being an empirical voice—a trusted, accessible source that can verify or validate the experience of children in our schools. How do we quantify that value? This is a conundrum I have pondered since I began working as part of an RPP.

For example, since the Baltimore Education Research Consortium (BERC) began examining municipal transportation data on Baltimore students’ school commuting, our work contributed to a change in the community conversation about local transportation, which until that point had taken little account of student ridership. Organizations became aware of the challenges of transportation for after school programs and daily attendance, especially on-time attendance. Such changes have a large impact, but how can they be measured?

A typical strategy to determine return on investment (ROI) needs explicit inputs and outputs. While student achievement is one output measure, there are others produced by an RPP that are not as easy to quantify, such as raising awareness of how transit impacts high school attendance or how changes in college enrollment patterns impact college degree completion.

At BERC, we haven’t shied away from the challenge, however, and have made the typical researcher attempts to quantify our efforts. First, we developed an internal survey to try to capture stakeholder satisfaction. Unfortunately, it had an extremely low response rate. When we discussed it with the school system CEO, she shared that she would rather tell us to our face what she thinks and not complete a survey. Next, in partnership with other RPPs, we crafted a survey tool (Wentworth, Mazzeo, & Connolly, 2017) to measure data and research use. More recently, we have used the Wilder Collaboration Factors Inventory to measure partnership qualities (Mattessich, Murray-Close, & Monsey, 2001). Other research teams have developed a framework in which to think about the multiple “dimensions of effectiveness”  (Henrick, Cobb, Penuel, Jackson, & Clark, 2017).

While the results from each effort offered us useful feedback, none captured the essence of the value of RPPs—and thereby specific information that would prove our value to funders as well as provide feedback to strengthen our RPP. After much internal conversation, we have a more comprehensive way to describe our value included here as Figure 1.

Our RPP focuses on being a local institution that provides support, research, and analysis for school district leaders, as well as community stakeholders and the assorted partners we engage with through our work. We think of this work in terms of growing and nurturing our partnerships via meeting their associated research needs.

Figure 1. Growing and nurturing a research-practice partnership to achieve valued outcomes.

Everything grows from a strong foundation (our roots), from which the work of rigorous research (our trunk) creates meaningful insights that impact youth (our crown of created value). When our collaborative efforts inform action, this further strengthens our partnerships as a result of demonstrating the value of working together, which in turn helps to institutionalize the work.

The Roots: Our Foundation

Requirement 1: Trust and Relationships. To exist and function, the RPP requires a number of foundational pieces to grow. Trust is essential for partners to have honest, transparent conversations and be vulnerable with each other. This is necessary for the partners to do work that matters. An RPP is not simply a researcher and a practitioner working collaboratively on a project. Instead, it is a partnership that begins with problem identification based on honest conversations about challenges and creating a much deeper, trusting relationship (Bryk and Schneider, 2002).

Requirement 2: Knowledge of Local Context. Local RPPs bring numerous partners to the table based on important recognition of its local contexts. External researchers wishing to initiate a project can be well-served partnering with an RPP, as it typically brings more stakeholders to the table for project development, implementation, and evaluation. Additionally, the deep understanding local researchers have of the community and context of the work is another powerful asset inherent in RPPs that can be leveraged by external actors. In addition to providing a richer and deeper understanding of local context, we know that context matters for school improvement (Hallinger and Murphy, 1986; McLaughlin and Rowan, 1993).

Requirement 3: Aligning Goals. The value of aligned goals works in both directions for practitioners and academics. Researchers can help practitioners think about their work within theoretical frameworks, but also help them focus on specific problems of practice (Roderick, Easton, and Sebring, 2009). For example, attendance is a challenge in general, but reducing chronic absences among first-time ninth graders is a more focused problem for which a research project and intervention can more manageably be defined. For researchers listening to district and school staff describe challenges and perceptions (e.g., “no one trusts the attendance data”), this helps create more realistic theoretical frameworks and better connects the research to more specific challenges on the ground (Coburn and Penuel, 2016).

The Trunk: Our Core Strength

Capacity: Rigorous Research. The core strength of any RPP is its expertise in conducting rigorous research that can be used by practitioners to change policy and practice. Its conception and completion must be empirical, as well as independent. Any reports or products must be credible to the local community as well as to peer researchers. Maintaining a clear impartial and technical analysis of the work is essential, or the value of the RPP vanishes.

The Crown: Our Created Value

With deep roots supporting its ability to conduct rigorous research, an RPP will produce valuable research and information that can be used by practitioners and researchers alike to inform policy and practice.

Valued Outcome 1: Inform Action. A priority for all research produced by an RPP is utility. Can partners use the findings to do their job better? It is a focus of RPP work to produce research and/or data that can be used by policymakers, practitioners, or community members. Its overarching goal is to inform action through policy development, policy and practice change, and creation of a local community knowledge base. To change the conversation about students and the institutions that serve them, all stakeholders—including parents, funders, and community members—must understand the research implications supplied by the RPP. Disseminating the research thus requires multiple forms of communications, with different take-home points, products, and styles (Finnigan, and Daly, (eds), 2014).

Valued Outcome 2: Impact Youth. As policy and practice changes, there is an expectation that outcomes for youth will also be impacted. One value of an RPP is to highlight the links between research, changes to policy and practice, and changes in youth outcomes. For instance, in our Early Education Data Collaborative, longitudinal research on programs and pathways of youth have led early-learning providers to streamline pass-offs between programs to better serve children and their families. The research also showed differential outcomes depending of youth characteristics which led one program to  re-prioritize enrollment in programs.

Valued Outcome 3: Strengthen Partnership. As projects grow and the work becomes deeper, the partnership itself becomes stronger. Partners are more eager to reach out for help or advice, and additional challenges can be identified and overcome. These collaborative projects lead to innovation and new ways of thinking about the work of both research and practice (Powell, Koput, & Smith-Doerr, 1996). Ten years ago, or even five years ago, the district would not have reached out to BERC to sit on advisory panels, attend strategic planning meetings, and work as thought partners. In addition to that, local funders and community groups also look to BERC for the same support.

Valued Outcome 4: Institutionalize Partner Work. As localized RPPs mature over time, their roles, processes and behaviors become institutionalized so that even when staffing changes occur, these qualities are known and instituted as new staff come on board both at schools as well as the district office and the universities. By creating common knowledge on school policy, data, practice, and outcomes, the RPP becomes a lever for change by increasing awareness across its locality and through other institutions in its community.

Summary

The work conducted by RPPs is valuable in both its formative aspects (i.e., developing relational trust, using rigorous research to provide data and perspective to our community, strengthening partners, and institutionalizing the knowledge) as well as its more formal returns (i.e., informing action, impacting youths’ lives, and eventually changing our city).  

We will continue to ponder how to measure this value. It is a challenge that will likely deepen as our connection to Baltimore expands and becomes more nuanced. Our hope is to develop a system of metrics that will provide feedback to strengthen and sustain an RPP and transcend the limitations of traditional ROI calculations to capture an RPP’s full impact, in all of its complexity and productivity in the local context.

The author would like to thank Elaine Allensworth, Bonnie Legro, and Laura Wentworth for their important feedback and suggestions on early drafts.

Faith Connolly is Executive Director of the Baltimore Education Research Consortium.

References

Bryk, A. S.; and Schneider, B. (2002) Trust in Schools: A Core Resource for Improvement, Russell Sage Foundation, New York, NY, USA.

Coburn, C. E., and W. R. Penuel. (2016). “Research-Practice Partnerships in Education: Outcomes, Dynamics, and Open Questions.” Educational Researcher 45 (1): 48–54.

Finnigan, K. S.; Daly, A. J. (eds) (2014). Using Research Evidence in Education from the Schoolhouse Door to Capitol Hill. Policy Implications of Research in Education 2, DOI 10.1007/978-3-319-04690-7_11, Springer International Publishing Switzerland.

Hallinger, P.; Murphy, J.F.  (1986). The Social Context of Effective Schools, American Journal of Education, Vol. 94, No. 3, pp. 328-355.

Henrick, E.C., Cobb, P., Penuel, W.R., Jackson, K., & Clark, T. (2017). Assessing Research-Practice Partnerships: Five Dimensions of Effectiveness. New York, NY: William T. Grant Foundation.

Mattessich, P., Murray-Close, M., & Monsey, B. (2001). Wilder Collaboration Factors Inventory. St. Paul, MN: Wilder Research.

McLaughlin, T. J.; and Rowan, B. (1993). Understanding Context Effects on Secondary School Teaching. Teachers College Record Volume 95 Number 1, p. 45-68.

Roderick, M.; Easton, J. Q.; and Sebring, P. B. (2009).  CCSR: A New Model for the Role of Research in Supporting Urban School Reform, University of Chicago, Chicago, IL.

Wentworth, L; Mazzeo, C.; Connolly, F. (2017). Research Practice Partnerships: A Strategy for Promoting Evidence-Based Decision-Making in Education Educational Research, 59(2) pp. 241-255.

Powell, W. W, Koput, K. W.; and Smith-Doerr, L. (1996).  Interorganizational Collaboration and the Locus of Innovation: Networks of Learning in Biotechnology Administrative Science Quarterly, Vol. 41, No. 1, pp. 116-145.

Schneider, M. (2018). IES Director’s Blog. Retrieved from https://ies.ed.gov/blogs/director/post/rpps

Suggested citation: Connolly, F. (2019). Measuring the Value of a Research-Practice Partnership. NNERPP Extra, 1(1), 6-9.