This is the seventh installment of Improving Improvement, our quarterly series focused on leveraging the power of research-practice partnerships (RPPs) to build schools’, districts’, and states’ capacity to improve. So far this school year, we’ve laid out the work ahead for our partnerships, the questions we hoped to answer, reflected on how to answer those questions, and evaluate the success of improvement efforts. While we won’t have data to report out on our current improvement efforts for another year or so, we do have some takeaways worth sharing after a full school year working with 15 new partners. In this installment, we start with a well-known but rarely addressed barrier to engaging in improvement best practices: Time.

Doing Too Much May Be the Biggest Barrier to Improvement

The idea that people who work in education have busy days is not novel. Everyone who has worked in or with a school district or charter management organization is familiar with the time pressure of days filled with meetings, emails, and endless “fires” to put out. In the battle of urgent versus important, urgent usually wins. This is so common that it is expected. It is taken as part of the context of the work, fixed and unavoidable. We design our RPPs to work around it as best we can, limiting what we ask of practitioners and fitting this work into the few cracks of daylight in their workdays. Here is the novel realization the Proving Ground team recently had: that makes us part of the problem.

We are part of the problem because we are not addressing the problem’s cause. Our partners do not have time to work with us because they are doing too much already. Working with us means doing more, at least in the short term. We knew going into this year that we had to address this. We partnered with states to make the improvement work part of the work our partners were already required to do. We kicked off our efforts by hosting sessions on creating the conditions for continuous improvement, one of which was “clearing the path,” which is a shorthand for carving out time. We encouraged partners to select interventions to pilot that aligned with existing plans. A year in, we are confident that our efforts fell short. 

We know this because we work closely with our partners. We see them pulled away from working sessions. They email us to cancel check-ins. They transparently admit when the team has not had time to meet to work on their problems of practice. We also have survey data to back this up. After every session with us, our partners provide feedback that we use to improve future sessions. The feedback is telling: Across 14 sessions, more than 80% of participants –and usually more than 90%– agree or strongly agree that the sessions were valuable and that their knowledge of the content in those sessions improved. However, there is one item that consistently falls below 80% : “This session was a good use of time.” The open-ended responses to the survey are the most telling. The most common comments involve some version of, “It’s hard to be away for a full day.” Also tellingly, participants routinely acknowledge the realization that doing quality improvement work requires spending more time unpacking and planning than they ordinarily spend on typical work tasks. However, they question how they could ever possibly spend this much time on more than one initiative. 

This is a problem that needs to be solved if we hope to have long-term impact. In the short term, our partners will likely succeed in completing this improvement cycle – they always have. Despite the burdens they face, they have engaged in the work; they have attended seven day-long workshops and they are executing the process in between. They will launch their thoroughly planned interventions next year, gather evidence and decide whether to stop, adapt, or scale. But we cannot be confident that they will continue this process after our engagement ends, let alone make it part of the ordinary course of business. It’s not because they have not embraced the process. It is because they do not have time. And they don’t have time because they are doing too much.

Doing Less is the Solution and the Result of Good Continuous Improvement

To improve, our partners need to do less. Practicing what we preach, we need to start solving the problem by understanding its cause. Here is our hypothesis: Our partners do too much because much of what they are doing does not advance their goals. They do a lot of ineffective or inefficient work because they do not have sufficient evidence of what is actually working and because the default is continuing everything unless it is proven ineffective (or enough stakeholders complain). This creates a vicious circle: Doing a lot of work that does not improve outcomes means outcomes are not improving – creating pressure to try more things. This is of course not the only reason education agencies do too much. There are structural and institutional factors that make education the preferred solution to a wide range of societal ills. But our hypothesis is that the most leverage comes from addressing the tendency to do too much that is not effective. 

The proposed solution is therefore to turn this from a vicious circle into a virtuous one. The core of good improvement practice, like the core of the scientific method, is to systematically generate evidence and use it to decide whether to abandon or continue with a hypothesis. In this context, the education agencies’ hypotheses are that their chosen interventions – policies, practices, or programs – will improve outcomes. If education agencies routinely generated quality evidence to test their hypotheses and used that evidence to stop doing anything where the impact was insufficient to justify the effort and cost, they would do less. With the extra bandwidth, they could improve the fidelity of the process, stopping more ineffective interventions and devoting more resources to effective ones. As the composition of interventions improves, so do outcomes, decreasing the demand for additional interventions. Moreover, the process gets easier and faster the more they do it and it can therefore be applied to more problems of practice in any given timeframe. 

None of this is to suggest that generating quality evidence is easy, especially where outcomes are difficult to measure. But this is the work researchers and RPPs exist to do. The key is to make this more efficient and easier for practitioners to do on their own. Proving Ground has put a lot of energy into this and in some ways, we have succeeded. But this analysis also suggests that evidence is a necessary not a sufficient condition. Shifting the vicious circle to a virtuous one requires both evidence and a fundamental shift in the default decision. Rather than continuing everything unless proven ineffective, education agencies should be continuing interventions only when they are proven cost effective. This would require a huge shift in culture, countering inertia and several decision-making biases. We are currently working on ways to do just that.

We are also working on addressing the “chicken-egg” problem implicit in all this. We need to help our partners clear enough space to get started and feel confident they can continue the work in the time the process requires. We are therefore working on a protocol to help them identify activities they are already doing that do not align with priorities and can be stopped to create a little bandwidth so they can get started. Stay tuned for more on that protocol.

Looking Ahead

In the next installment of Improving Improvement, we will share more detailed updates on the progress of our intrastate networks, the Georgia Improvement Network and the Rhode Island LEAP Support Network, including lessons learned from partnering with states to support districts on their improvement journeys.

We are also always open to additional suggestions for topics for future editions of Improving Improvement. Reach out to us with any questions you have about our networks, continuous improvement process, or ideas you’d like to see us tackle.

David Hersh (david_hersh@gse.harvard.edu) is Director of Proving Ground.

Suggested citation: Hersh, D. (2022). Improving Improvement: What’s the Biggest Barrier to Improvement? NNERPP Extra, 4(2), 19-21.