DOES MONEY MATTER FOR THE PRODUCTION OF RELEVANT RESEARCH?

INCENTIVIZING RESEARCHERS TO WORK ON RESEARCH RELEVANT TO PRACTICE, POLICY, AND COMMUNITIES

Laura Wentworth | California Education Partners + Stanford-SFUSD Partnership, Erin O’Hara | Erin O’Hara Consulting, and Beth Vaade | Madison Metropolitan School District + Madison Education Partnership

Volume 6 Issue 2 (2024), pp. 10-22

One of the most commonly cited influences leading to the research-practice divide in education is academics’ lack of motivation or incentive to research topics that are relevant to the educational challenges faced in practice, policy, and communities. Universities award tenure to academics based on multiple factors, typically prioritizing the number of publications in peer-reviewed academic journals. While educational research is an applied discipline –the topics explored by educational researchers can be quite applicable to practice– challenges in education are complex and require multi-level systems thinking that goes beyond the solutions suggested by a single study. Consequently, educational research published in journals focuses on findings that may or may not be relevant to the dynamic challenges faced by education professionals and community members, and if relevant, may only address one challenge in a multi-pronged quandary. Gamoran (2023) refers to these realities as the tensions between the aims of engaged scholarship and the reward structure of a research university.

The research-practice partnership (RPP) approach to research is an example of a structure meant to facilitate the connectivity between research and practice (Coburn and Penuel, 2016) and may thus be a promising way to motivate academics to engage in relevant, timely research that addresses the more complex educational problems. RPPs help researchers and practitioners work at the boundaries of their roles and change routines and practices used for producing and using research (Penuel et al., 2015). However, most academics engaged in RPP work still face the above-mentioned tensions between the RPP aims of engaged work done in partnership and the typical reward structures of universities that are not necessarily supportive of collaborative work. To address this, university grant-making programs designed to incentivize faculty to work in RPPs are emerging as a structure to encourage university researchers to work on research that is relevant to practice, policy, and communities and works towards the more multi-pronged solutions needed in the field of education.

Generally, these financial incentives are awarded through grant-making programs but can also be provided through access to resources like infrastructure for accessing school district or state data (e.g., Kim et al. 2021) or other forms of social infrastructure like routines and meeting structures (e.g., Penuel, 2019). The financial incentives from grant-making come through payment of course releases, summer salary, funding of faculty research teams like doctoral students and lab staff, or funding supporting the practice or community partner to engage in the partnership. Little research currently exists about such incentive structures and the type of supports needed to implement them, as well as the kind of work and other impacts resulting from such efforts.

In this article, we document the use of financial incentives to support university faculty work in RPPs with practitioners and policymakers by examining the grant-making programs of three universities that have been administered starting in 2014, 2016, and 2017 (one of the programs is still being administered today). For the purposes of our article, we limited our analysis to when the grants were administered during the 2014-2021 school years. The universities are the Wisconsin Center for Education Research at the University of Wisconsin-Madison, Vanderbilt University, and Stanford University Graduate School of Education, and the three RPPs supported by these grant-making programs are the Madison Education Partnership, Tennessee Education Research Alliance, and Stanford-SFUSD Partnership. We are three RPP directors (two current and one former), one from each of the RPPs, and were directly involved in supporting the implementation of the respective grant-making program.

UNIVERSITY GRANT-MAKING PROGRAMS SUPPORTING RESEARCH-PRACTICE PARTNERSHIPS

Here, we describe each of the three grant-making programs and highlight the main effects we have seen these programs have, both the positive effects and the stumbling blocks.

(I) UNIVERSITY OF WISCONSIN-MADISON + MADISON EDUCATION PARTNERSHIP

Overview

University of Wisconsin-Madison’s Grant-Making Program to Support Madison Education Partnership

The Madison Education Partnership (MEP) was established in 2016 as a partnership between Wisconsin Center for Education Research (WCER) at the University of Wisconsin-Madison and the Madison Metropolitan School District (MMSD). WCER and MMSD created MEP to collaboratively engage in “high-quality, problem-based” research that influences policy, practices, and the understanding of educational processes more generally. In addition to expedited approval processes, easier access to district administrative data, and support for relationship development with district leaders, the university also provided financial incentives for university researchers to engage in MEP through a grant-making program. The grants provided university faculty, research faculty, and graduate students (supported by a Principal Investigator) an opportunity to submit proposals for funding of up to $50,000 for one year. Typically, two proposals per year were selected (with $100,000 in total available each year) through a thorough scoring and discussion process.

Submitted projects needed to align with these four criteria:

  • Focus on a MEP problem of practice.
  • Produce findings within one year of the award.
  • Present research findings twice to school district leaders.
  • Create two reports about the findings (one interim and one final).

MEP released a request for proposals and interested researchers submitted a short application that included a cover page, a project description (maximum three pages), project budget, project timeline, and a description of project personnel (including biographical sketches and CVs). The proposal information also included a scoring rubric (see here for the rubric). Once submitted, the MEP co-directors and the Steering Committee reviewed proposals and made recommendations for funding to the school district superintendent and the WCER director for final approval. MEP has offered the grant competition four times since Fall 2016, with the last competition in 2020-21. By 2024, MEP no longer offered sponsored research grants.

Effects of UW-Madison’s Grant-Making Program

Four grant cycles have taught MEP about the effects that financial incentives do and do not have on researchers. The grant-making program:

  • Achieved many of its intended goals of producing faculty-led relevant research. By 2019, MEP funded nine studies, all aligned to the partnership’s problems of practice (four-year-old kindergarten and student attendance). Of the funded cohorts, all projects have conducted new research and submitted multiple reports to MEP and MMSD. Several projects have also led to academic publications and further, larger grant funding from external organizations.
  • Sparked more robust methodological approaches to and new faculty perspectives on the problem of practice and MEP. All nine studies collected new data and used qualitative methods as at least one component of their work.
  • Attracted new faculty to MEP: The nine studies also came from faculty in a diverse set of departments, schools, and colleges across the University of Wisconsin-Madison, as well as researchers who were new to MMSD and education.

Despite these successes, there have been learnings for MEP about the limitations of using financial incentives to spark researcher interest in partnership work:

  • MEP saw limited interest from faculty in applying for and participating in the grant program for a number of reasons. The faculty lacked knowledge of MEP and/or the grants, had little interest in the problem of practice named by the district, or were concerned that the funding amount and/or timeline were infeasible.
  • MEP directors had to coordinate multiple research projects across the same problem of practice. It was challenging for the school district personnel to coordinate the operations of multiple research projects. School district staff, at times, struggled to see how the projects related to each other and to MEP. Additionally, it was challenging for researchers to understand the contours of each other’s research projects, which led to researchers making repetitive requests to the school district.
  • University researchers needed more support than the MEP directors had expected to write reports on findings that practitioners found accessible. The university researchers had trouble writing interim and final reports of their findings in language friendly to their practice partners.
(II) VANDERBILT PEABODY COLLEGE + TENNESSEE EDUCATION RESEARCH ALLIANCE

Overview

Vanderbilt Peabody College’s Grant-Making Program to support the Tennessee Education Research Alliance

The Tennessee Education Research Alliance (TERA) is a research-policy-practice partnership between Vanderbilt University’s Peabody College (Peabody) and the Tennessee Department of Education (TDOE). Formally launched in 2016, TERA builds on a long history in Tennessee of supporting high-quality research that directly impacts state policy. Starting in 2017, Peabody offered faculty the opportunity to apply for research start-up grants twice a year of up to $15,000. After initial interest from Peabody faculty in data housed within TERA, TERA launched a new larger grant opportunity in 2020. The TERA Research Partnership Fund allowed for larger grants (up to $40,000) for faculty to partner with TERA and TDOE aiming to attract faculty with diverse expertise and disciplines to partnership work. 

TERA formed a selection committee that included the then TERA Executive Director and Faculty Director and Peabody’s Associate Dean for Research. Because of the gubernatorial transition and subsequent change in leadership at the TDOE, the initial selection committee did not include TDOE leaders. 

After initial iterations, the TERA Research Partnership Fund settled on four key criteria for funding:

  • Alignment and relevance: Is the project aligned with TERA and TDOE research priorities? Does the proposal demonstrate the potential and plan for the results to be of use to policymakers and/or practitioners seeking to enhance the quality and/or equity of education in Tennessee? Does the proposal clearly address a problem of policy or practice?
  • Methodological rigor: Does the proposal clearly outline an empirical strategy and does that strategy serve to address the research questions? Will the empirical strategy produce valid, replicable results? 
  • Personnel: Does the Principal Investigator have relevant substantive and methodological expertise and experience to do this research? Has the Principal Investigator secured a co-Principal Investigator from TDOE to help plan, operationalize, and use the research? 
  • Feasibility: Does the proposal detail adequate time and personnel to carry out a high-quality research program?

Applying faculty were asked to submit a 2-3 page brief summarizing their project. Through three cycles of grant applications, eight total grants were submitted and six awards were granted. (See here and here for examples of the scoring rubric used in the grant-making competition.) In 2024, the TERA Research Partnership Fund program is currently on hold.

Effects of Vanderbilt Peabody College’s Grant-Making Program

Peabody College launched their initial grant competition in Fall 2017, and ran the grant competition in Spring 2018 and Fall 2018, before ultimately launching a TERA specific program in 2020. The programs had the intended effects of getting more researchers involved in TERA’s partnership work, but with some stumbling blocks:

  • In the first round, the proposals were not co-developed with TDOE; therefore, department staff at the time were initially hesitant to approve the research outlined in the winning proposal. Subsequent conversations led to slight adaptations to the research design to address some concerns by the state and university officials and to an understanding between the faculty member and staff at the Department of Education about the potential policy impact of the research.
  • In the second round, TERA changed the proposal structure to align with TDOE’s topics of interest, but they ultimately did not receive any proposals. When TERA’s leadership team worked with the Peabody Research Office to revise the request for proposals to include specific topics of interest to TDOE, the small grants focused on TERA did not receive any proposals.
  • With the launch of the TERA specific competition, the TERA Research Partnership Fund, the submitted proposals differed from previous proposals in that the research had been developed in partnership between TDOE and the faculty members applying. Additionally, TERA and TDOE added TDOE’s (then) Chief Research and Strategy Officer to the review committee at Peabody. The winning proposals had significant input both at Peabody and at TDOE.

The TERA and Peabody leaders running the grant competitions realized a couple of important lessons learned:

  • The TERA team needed to help both partners understand the incentives for the other partner: Specifically, the team needed to help articulate what motivates the Peabody faculty to participate in partnership research and what motivates TDOE to participate in and approve partnership research, and how these motivations could be aligned.
  • Vanderbilt researchers’ participation in TERA increased when they received the support of a broker to help develop research questions and connect to the right staff at the partner organization. (A broker is someone who facilitates the interactions and connections between and across members of the research and practice/policy side of the partnership. See here for more information about RPP brokering.)
(III) STANFORD UNIVERSITY GRADUATE SCHOOL OF EDUCATION + STANFORD-SFUSD PARTNERSHIP

Overview

Stanford University Graduate School of Education’s Grant-Making Program to Support the Stanford-SFUSD Partnership

Since it was established in 2009, the Stanford-SFUSD Partnership aims to cultivate joint projects, which result in Stanford University Graduate School of Education (GSE) researchers conducting useful, generalizable research and San Francisco Unified School District (SFUSD) administrators using the research in key decisions, in hopes of advancing better outcomes for students. The partnership is run by a Partnership Director, employed by a third party, California Education Partners. Starting in 2014, Stanford GSE in partnership with SFUSD created a grant-making program for projects in SFUSD aimed at incentivizing professors to work on projects aligned with SFUSD’s priorities and useful to SFUSD administrators. After the request for proposals (RFP) was announced, interested faculty had one-on-one meetings with the Partnership Director to find a practice partner and learn more about the RFP.  Stanford faculty were encouraged to write the proposal in collaboration with an SFUSD leader. A joint selection committee composed of two SFUSD representatives and two Stanford representatives selected projects. The members of the committee included SFUSD’s Deputy Superintendent of Instruction and Assistant Superintendent of Research, Planning, and Assessment, and two Stanford faculty representatives selected by the Stanford GSE Dean. There were three criteria the selection committee used to award incentive funds:

  • The project is aligned with the school district priorities.
  • The project meets Stanford GSE’s standard for generalizability – likely to produce findings that will be relevant and useful to other school districts.
  • The project has a district leader acting as the owner/sponsor of the research who helps plan, operationalize, and use the research.

As seen here, the committee used a rubric to guide selection of the projects based on these criteria. 

After awarding the first set of Incentive Fund grants in 2014, the Stanford GSE gave out about 4-6 grants a year which were valued on average around $25,000-$100,000 each, with a few projects receiving over $100,000. By 2024, the grant-making program still existed, although the budget and grants were smaller at $50K each, with about two to three grants awarded each year.

Effects of Stanford GSE’s Grant-Making Program

The grant-making program has had a number of impacts on both the RPP and faculty members’ research. Here we share a few reflections from our efforts in supporting this program so far:

  • More faculty worked in partnership with SFUSD than did prior to the incentive fund. Between 2014-2019, of the 23 faculty who received incentive fund awards, over half had not worked with SFUSD before. 
  • The selection committee wanted more evidence of district support and “impact” of projects pursuing a second year of funding. For example, the committee added a letter of recommendation as one of the requirements to help understand the role of the district leader acting as the owner or sponsor of the research. 
  • The incentive fund required the school district to improve how they articulated their research priorities. It was challenging for SFUSD to explain the ideas or questions they wanted to research that could coincide with the one-year timeline of the grants.
  • The need for research aligned with district priorities moved under a faster timeline than the average development of research. This challenge emphasized the importance of the match between the researchers’ interests and the practitioners’ interests for making a good project.
    SIMILARITIES, DIFFERENCES, IMPLICATIONS FOR UNIVERSITY GRANT-MAKING PROGRAMS FOR RPPs

    When comparing across the three university grant-making programs, we see several similarities and a number of notable differences. Similarities include:

    • The grant-making programs all shared a common goal – to increase the number of faculty working in the RPPs on research seen as aligned and useful to the respective practice partner.
    • Grants were awarded on an annual cycle with the expectation that findings were also shared over the course of one year. 
    • All three of the programs had the requirement that the research had potential to make an impact on policy and practice. This required the district and state partners to name their priorities for research. 
    • In all three RPPs, the RPP director needed to provide social infrastructure (scaffolded meeting routines and orchestrated events) for the researchers and practitioners to interact
    • The grant-making programs all required researchers to widen their portfolio of research. To explore the topic of interest to the district or state leaders, some researchers tried a new method or explored another discipline, thereby working at the boundary of their expertise. For example, many faculty who traditionally look for causal warrants in their research designs pursued descriptive research.

    Some differences we noted across the grant-making programs were:

    • The amount of money awarded was different. UW Madison’s grant-making program awarded about $100,000 in total each year (typically, two proposals for funding of up to $50,000 were selected), the Peabody program offered research start-up grants twice a year of up to $15,000 and the subsequent TERA program offered up to $40,000 for around two to three projects a year, and Stanford GSE’s program awarded four to six grants a year, ranging from $25,000 to $100,000 each.
    • The point at which the programs were launched with respect to the RPP’s journey was different. For example, the Stanford GSE launched its grant-making program after working in partnership with SFUSD for four years, whereas UW Madison and Vanderbilt launched their grant-making programs at the beginning of their RPPs. For the latter two universities, the grant-making program aimed to jumpstart faculty engagement with the RPPs, whereas for Stanford GSE, the RPP was already established, and the grants were meant to accelerate faculty engagement.
    • Each university used a different approach to sharing criteria and awarding funding. Vanderbilt used a university-based committee to award funding, whereas UW Madison and Stanford used committees mixed with university faculty and district leaders. The TERA program did add a TDOE representative to the review committee at Peabody. The universities used different criteria for awarding funding, which are contrasted in Table 1 below. Two of the three universities used a rubric to guide decision-making about which projects to award grants.

    Table 1: Elements of Research Practice (Policy) Partnerships and Incentive Fund Programs

    Madison Education Partnership

    Tennessee Education Research Alliance

    Stanford-San Francisco Partnership

    Year Established

    2016

    2016

    2009

    Year Incentive Fund Program Established

    2016

    2017

    2014

    Size of Grants

    up to $50,000

    up to $40,000

    $25,000-$100,000+

    Number of Grants Awarded in First Year of Grantmaking Program

    3 projects at $50,000 for one year

    3 projects at $40,000 for one year

    6 projects ranging in amount from $30,000 to over $100,000 for one year

    Timeline for research

    Annual

    Annual

    Annual

    Criteria for Grant Competition

    • Address a problem of practice selected by the partnership 
    • Produce findings within one year of the award
    • Present research findings twice to district leaders
    • Create two reports about the findings (interim and final)
    • Add to existing TERA and TDOE knowledge base
    • Add to faculty’s existing research program
    • Clear research questions and data to be analyzed
    • Clear research plan aligned to TERA and TN Dept of Ed priorities
    • Potential to lead to other funding
    • Advance the policy and practice of TN Dept of Ed
    • The project is aligned to the school district priorities
    • The project meets Stanford GSE’s standard for generalizability – likely to produce findings that will be relevant and useful to other school districts
    • The project has a district leader that helps plan, operationalize, and use the research
    IMPLICATIONS

    This cross-case analysis surfaced a handful of potential implications that may be useful for other universities that want to launch grant-making programs in support of enhancing academic researchers’ ability to participate in local RPP efforts. In order to be successful, university grant-making programs might require the following:

    • Practice-side capacity to articulate priorities for research topics and where research can support their work. If universities want their faculty to produce ideas for research that are useful to their district and state partners, then the practice partners will need to have the capacity to articulate their goals and describe the types of research topics, questions, and timelines that could be useful.
    • Research-side capacity to develop proposals and engage in RPP work. In our three cases, the directors provided necessary capacity building for the RPP that supported the grant-making program. The RPP directors matched faculty with district and state leaders who had similar interests in research. They also supported faculty when developing their proposals by providing feedback and ideas for improving the proposal. Capacity-building efforts need to be accounted for in grant-making programs, either by providing additional time for these added support mechanisms or allocating additional funding to directors to help them engage in this work.
    • Structures and routines for developing proposals and awarding grants (this takes time and resources). All three universities relied on RPP directors to run their grant-making programs. The directors developed the request for proposals in collaboration with other university and practice leaders and acted as the broker to operationalize the grant-making programs (see Wentworth et al., 2023 for a detailed description of RPP brokering).
      CONCLUSION

      As we conclude our analysis of three university grant-making programs designed to incentivize university researcher participation in RPP work, we return to this article’s original question asked in the title: Does money matter for the production of relevant research? 

      For RPPs involving universities, there is often a larger goal around structural changes needed to address misaligned incentives, such as different university promotion and tenure guidelines that actually do prioritize research that is relevant to and co-produced with practice, policy, and community partners. As we have heard many times over, this will likely take time. In the meantime, the three cases outlined here, which we experienced first-hand, seem to suggest that money does help and matter, with a couple of caveats and conditions. For once, money alone only gets you so far in changing systems in universities – it’s a necessary, though not sufficient solution. For example, we had faculty who would engage because of the money, but generally had limited interest in engaging in deeper partnership research. Many faculty needed a guide –or what we call a broker– to be able to benefit from the larger social infrastructure of a partnership and navigate relationship building with their new practice partners. The district and state leaders also benefited from engaging with the broker to learn about and engage in important RPP practices, such as naming researchable questions. Both the development and execution of the research benefit from tight alignment between researcher and practitioner.  In short, in view of our own experiences with these grant-making programs, we can confidently say money does matter for promoting relevant research in the field of education – but it is only one part of the solution.

      We also wondered: Is there an ideal amount of money for these grants to achieve the goal of getting more faculty engaging in the RPPs? Generally, we think it may be less about the amount of money, and more about the capacity, infrastructures, and supports that may prohibit or further support faculty from engaging in partnership work. Things like data infrastructure (simple access to administrative data), technical infrastructure (streamlined agreements), and social infrastructure (processes, routines, or events that support relationship development and maintenance) all play an important role in fostering faculty participation in an RPP. The grant funding from universities in our three cases mainly served the purpose of funding faculty and researcher time and did not account for the larger supporting infrastructure enabling faculty to work in the RPP. Taking this into account could make such programs even more useful.

      As RPPs continue to make headway as a more useful approach to the production and use of relevant research, we hope to see the support for this kind of work grow as well so that the research-practice divide becomes, in turn, smaller and less dividing.

      Laura Wentworth is Director of the Research-Practice Partnership Program and Director of the Stanford-SFUSD Partnership at California Education Partners; Erin O’Hara is Principal at Erin O’Hara Consulting and was previously the founding Executive Director of the Tennessee Education Research Alliance; and Beth Vaade is Senior Executive Director of Research, Assessment, and Improvement at Madison Metropolitan School District and Co-Director of the Madison Education Partnership.

      REFERENCES

      Coburn, C.E. & Penuel, W. R. (2016). Research Practice Partnerships in Education: Outcomes, Dynamics, and Open Questions. Educational Researcher. 45(1): 48-54.

      Gamoran, A. (2023). Advancing Institutional Change to Encourage Faculty Participation in Research-Practice Partnerships. Educational Policy37(1), 31 55.  https://doi.org/10.1177/08959048221131564

      Kim, M., Shen, J., Wentworth, L., Ming, N., Reininger M., and Bettinger, E. (2020). “The Stanford-SFUSD Partnership: Development of Data-Sharing Structures and Processes.” In: Cole, Dhaliwal, Sautmann, and Vilhuber (eds), Handbook on Using Administrative Data for Research and Evidence-based Policy. Accessed at https://admindatahandbook.mit.edu/book/v1.0/sfusd.html on 2024-02-28.

      Ozer, E. J., Langhout, R. D., & Weinstein, R. S. (2021). Promoting institutional change to support public psychology: Innovations and challenges at the University of California. American Psychologist, 76(8), 1293–1306. https://doi.org/10.1037/amp0000877

      Penuel, W.R. (2019). Co-design as Infrastructuring with Attention to Power: Building Collective Capacity for Equitable Teaching and Learning Through Design-Based Implementation Research. In: Pieters, J., Voogt, J., Pareja Roblin, N. (eds) Collaborative Curriculum Design for Sustainable Innovation and Teacher Learning. Springer, Cham. https://doi.org/10.1007/978-3-030-20062-6_21 

      Penuel, W. R., Allen, A. R., Coburn, C. E., & Farrell, C. (2015). Conceptualizing research–practice partnerships as joint work at boundaries. Journal of Education for Students Placed at Risk, 20(1), 182-197.

      Wentworth, L., Arce-Trigatti, P., Conaway, C., Shewchuk, S. (2023) Brokering in Education Research-Practice Partnerships: A Guide for Educational Professionals and Researchers. Routledge.

      Suggested citation: Wentworth, L., O’Hara, E. & Vaade, B. (2024). Does Money Matter for the Production of Relevant Research? Incentivizing Researchers to Work on Research Relevant to Practice, Policy, and Communities. NNERPP Extra, 6(2), 10-22.

      NNERPP | EXTRA is a quarterly magazine produced by the National Network of Education Research-Practice Partnerships  |  nnerpp.rice.edu