OVERVIEW

The Research Artifact

“Impact of the CPS Computer Science Graduation Policy on Student Access and Outcomes”

By Steven McGee, Lucia Dettori, and Andrew Rasmussen (2022)

The RPP: Mission

The Chicago Alliance for Equity in Computer Science (CAFÉCS) is an RPP among Chicago Public Schools (CPS) teachers and administrators, university computer science faculty from University of Illinois at Chicago, DePaul University, and Loyola University Chicago, and educational researchers from The Learning Partnership and Partner to Improve. CAFÉCS places the accent on equity to emphasize our computer science education goals for CPS. CAFÉCS focuses on research and development that enables CPS to ensure that all students in CPS participate in engaging, relevant, and rigorous computing experiences, to increase opportunities for all students to pursue computing pathways, and to prepare all students for the future of work.

The RPP: History and Context

In 2009, CPS computer science teacher Don Yanek, district administrator Brenda Wilkerson, and three university computer scientists, Lucia Dettori (DePaul University), Ron Greenberg (Loyola University Chicago), and Dale Reed (University of Illinois Chicago) came together to collaborate on a shared goal: to provide all CPS students access to compelling and relevant computer science experiences. At the time, the nation was experiencing a dramatic decrease in the number of college graduates majoring in computer science (McGee et al., 2013). For several years, the group explored various options for supporting computer science in CPS. In 2011, the team identified the Exploring Computer Science (ECS) program, which includes curriculum and teacher professional development, as a promising option (Goode et al., 2014; Margolis et al., 2012). Three teachers traveled to Los Angeles in the summer of 2011 to participate in the ECS professional development in preparation for a pilot test of the program in Chicago. The professional development uses a Teacher-Learner-Observer model followed by a debrief allowing teachers to reflect and discuss the impact of equity and inquiry strategies in the lesson. ECS was developed for the Los Angeles public schools. In 2012, CPS became the first school district to adopt the ECS program outside of Los Angeles (Reed et al., 2015). At the time, computer science programs and curricula in public school districts were nascent. Since that time, ECS has expanded to at least 34 states and Puerto Rico, including the seven largest school districts, as well as some rural locations and reservations.

In parallel, the group sought funding to scale up ECS professional development in Chicago. The first collaborative grant from the National Science Foundation (NSF)—Taste of Computing—was awarded to the group in 2011 (Dettori et al., 2011). As part of this grant, Steven McGee of The Learning Partnership joined the collaborative to support the research and evaluation activities. CAFÉCS research on the initial implementation of ECS provided preliminary evidence of the benefits of ECS as the foundational course for high school computer science pathways. Not only does ECS support equivalent outcomes regardless of students’ race/ethnicity and gender (McGee, McGee-Tekula, Duck, McGee, et al., 2018), it also equivalently increases interest in pursuing additional computer science coursework (McGee, McGee-Tekula, Duck, Dettori, et al., 2018). As the ECS program spread to more CPS schools, the mayor’s office and CPS launched the CS4All initiative in 2013 to expand computer science opportunities across all grade levels in Chicago (Zumbach, 2013). A goal of the CPS CS4All initiative was to provide computer science courses in every high school and work towards incorporating computer science into high school graduation requirements (City of Chicago, Office of the Mayor, 2013). After four years of pilot implementation of ECS, CPS became the first school district in the nation to enact a high school computer science graduation requirement, with ECS serving as the foundational course for the requirement. Students either had to complete a yearlong computer science class or apply for a waiver if they were participating in a Career and Technical Education (CTE) program, the International Baccalaureate (IB) program, or a similar career-oriented or college prep program. Although less than half of the high schools in CPS offered any computer science at the time, CAFÉCS research showing the benefit of ECS as a foundational course (described above) provided the Board with confidence that the ECS curriculum and professional development could spread to all high schools in CPS.  

When NSF launched the CS4All RPP program, the Chicago team received one of the first two large grants awarded, and CAFÉCS was formalized as an official RPP in 2017. At that time, Partner to Improve (led by Erin Henrick) joined the partnership as the external evaluator. Starting in October 2017, the goal of CAFÉCS was to understand and address variation in the implementation of ECS across CPS. In 2019, CAFÉCS joined NNERPP. In 2021, Kristan Beck became the 4th director of the Office of Computer Science in CPS.

WHY THIS WORK

In June 2020 and June 2021, more than fourteen thousand students in the Chicago Public Schools (CPS) graduated each year with one year of high school computer science credit in fulfillment of CPS’ computer science graduation requirement. With the graduation of the first two cohorts of CPS students subject to the graduation requirement, the time was right for CAFÉCS to undertake a systematic analysis of the state of computer science in CPS as well as the full impact of the graduation requirement.

WHAT THE WORK EXAMINES

Previous Preliminary Evidence of the Benefits of ECS

The CAFÉCS analysis of the impact of CPS’s CS graduation requirement builds on previous partnership work exploring the benefits of the ECS course: CAFÉCS research found that ECS not only supports equivalent outcomes regardless of students’ race/ethnicity and gender (McGee, McGee- Tekula, Duck, McGee, et al., 2018), it also equivalently increases interest in pursuing additional computer science coursework (McGee, McGee-Tekula, Duck, Dettori, et al., 2018). The participation of African American and Latinx students in Advanced Placement (AP) computer science doubled two years after the graduation requirement was enacted. African American and Latinx students were 1.5 times more likely to have taken ECS prior to their AP class than white and Asian students (Boda & McGee, 2021). ECS also serves as effective preparation for AP CS A, as the students who took ECS prior to AP CS A were 3.5 times more likely to pass the AP exam with a score of 3 or higher than those who did not take ECS first (Boda & McGee, 2021).

Both previous and current CAFÉCS research priorities are shaped by the problems of practice facing the CPS Office of Computer Science, as identified and shared by Office of Computer Science staff. CAFÉCS uses a variety of meeting formats to foster collaboration in its weekly meetings. In the early days of CAFÉCS, the entire collaborative met monthly to support implementation of the graduation requirement; now, we meet on an ad hoc basis. During the meetings of the entire collaborative, CPS leaders share the district priorities for computer science and the CAFÉCS team aligns its activities to the priorities of CPS. When CPS identifies problems of practice, CAFÉCS addresses them through a collaborative problem-solving cycle (Lewis et al., 2022) involving six stages:

1. CPS leaders identify a problem they are facing. 
2. CAFÉCS partners brainstorm hypotheses during meetings of the entire collaborative. 
3. The CAFÉCS leadership team filters and prioritizes the hypotheses to test during the weekly meetings. 
4. The research team engages in data analysis. 
5. The research team shares the results in both leadership team meetings and in meetings of the entire collaborative. 
6. The results inform both the Office of Computer Science strategies and the CAFÉCS research agenda.

This problem-solving cycle was used in CAFÉCS research exploring factors that correlate with students failing the ECS class (McGee et al., 2018). The results pointed to the importance of ECS professional development for ECS teachers in reducing the course failure rate. CPS computer science integration specialists used these results in communicating with principals about the importance of the ECS professional development. The results also laid the foundation of further research and development around high-quality credit recovery (Johnson et al., 2022).

Exploring the Impact of the CS Graduation Requirement

After the second cohort of students subject to the computer science graduation requirement graduated from CPS, CAFÉCS launched an evaluation of the graduation requirement, funded by the Chicago Mercantile Exchange (CME) Group Foundation. In summer 2022, The Learning Partnership established a summer fellowship program for advanced graduate students and early career researchers (also funded by the CME Group Foundation) to conduct this analysis on the impact of CPS’s CS graduation requirement. The fellowship program provided an opportunity to get an external examination of the impact while at the same time to provide early career researchers with exposure to partnership work. A national search was conducted, and six fellows were invited to participate through a competitive selection process. As part of the 8-week fellowship, the fellows attended an initial 3-day orientation session in June 2022 in Chicago. The orientation included a discussion with the CAFÉCS leadership team and members of the Office of Computer Science at CPS. After the orientation, the fellows worked remotely to conduct their analyses. They had weekly virtual check-ins with The Learning Partnership, and also had access to one of three quantitative mentors who met with the fellows virtually twice during the summer and provided feedback. The fellows also met remotely with staff from the Office of Computer Science at the midpoint of their analyses to get feedback. At the end of the fellowship in August, the fellows participated in a symposium event that served as a celebration of the success of the graduation requirement (Dettori et al., 2022).

The CAPE framework (Fletcher & Warner, 2021) provided the conceptual framing guiding the impact evaluation of the graduation requirement. In particular, each letter in “CAPE” stands for a dimension of work that needs to be attended to in support of a specific aim. In our case, for example, to ensure that CPS supports equal access to and course performance in computer science, the district needs to develop Capacity for schools to offer computer science, increase Access to computer science, ensure equal Participation, and then examine how computer science Experiences lead to equal outcomes. Applying this framework, the six fellows conducted analyses related to different aspects of the following four research questions:

1. How did CPS’s Capacity to offer computer science change after the graduation requirement was enacted?
2. How did Access to computer science courses change after the graduation requirement was enacted?
3. How did Participation in computer science change after the graduation requirement was enacted?
4. How consistent were the student Experiences in and outcomes from their computer science courses before and after the enactment of the graduation requirement?

FINDINGS

The summer fellows presented the following results across all four dimensions of the CAPE framework: 

1. CAFÉCS research provided evidence of the key role that teachers play in the success of students in computer science (McGee, Greenberg, et al., 2018). Lack of qualified computer science teachers restricts schools’ Capacity to offer computer science. The ECS professional development program supported a rapid expansion of school Capacity after the enactment of the graduation requirement by increasing the number of qualified computer science teachers. At the time the graduation requirement was enacted, roughly half of the schools did not offer any computer science and 2/3 did not have sufficient capacity to support computer science for all students. Larger schools with fewer low-income students and a strong college-going climate were more likely to offer computer science just before the enactment of the graduation requirement. By the time the first cohort graduated, almost 90% of high schools had sufficient capacity to offer computer science to all students. 
2. There was a statistically significant increase in Access to computer science after the computer science graduation requirement. All high schools offered computer science by the time that the first cohort graduated. 
3. There was a statistically significant increase in Participation in computer science across all demographic groups after the graduation requirement. By the time the second cohort graduated after the requirement, the demographics of students taking computer science matched the demographics of the district.
4. Students’ Experiences with ECS led to equivalent course performance between students taking ECS before and after the enactment of the graduation requirement. The number of students pursuing computer science pathways in CPS doubled after the enactment of the graduation requirement.

At the celebration event, two Black female CPS high school students shared their experiences with ECS:

“I would absolutely consider continuing this and not just because it’s very good at helping me out with computer science, also it’s the way that they taught it. It actually took time for me to understand it because even though I love computer science, I was struggling in the beginning but it was always the different ways they would explain it and the different ways that they would backtrack and make sure everyone understood. It’s definitely something I would want to continue not just for computer science but also some kids as they said beforehand you actually have an opinion and you actually do have a voice and it may take time but you will understand.” – CPS sophomore computer science student

“The words “computer” and “science” together can give some people the wrong impression. I would have to say to take the class seriously and to actually try to enjoy it. At first learning all the different code blocks can seem confusing but once you actually get it, it’s a pretty fun class. I would definitely tell them to try and enjoy it.” – CPS sophomore computer science student

IMPACT AND USE OF THE WORK

The analyses conducted by the summer fellows provided an opportunity for the community to celebrate the success of the computer science graduation requirement. DePaul University hosted a hybrid event in which the fellows presented the results of their analyses interspersed with opportunities for CPS administrators, teachers, and students to share how the graduation requirement has impacted them (Dettori et al., 2022). The development of the report also provided an opportunity for CAFÉCS to reflect on the strategies they used to support the enactment and implementation of the graduation requirement. CAFÉCS identified four key strategies for enabling sustainable change in school districts, which are organized into the PROSPER framework (Programs, Research, Organizational Structure, and Policy for Equitable Results):

1. Programs: CAFÉCS impacted the selection and implementation of computer science programs 
2. Research: CAFÉCS conducted research to understand and address problems of practice facing the Office of Computer Science 
3. Organizational Structure: CAFÉCS funding impacted organizational structures in CPS
4. Policy: CAFÉCS impacted the enactment and implementation of the district graduation requirement policy. The successful implementation of the graduation requirement has increased Equitable Results in computer science education in CPS by expanding capacity, access, and participation leading to equitable experiences

This research may serve as a blueprint for other districts who are pursuing computer science access for all students. One consistent theme across all of the findings is the important role that teachers played in the success of the initiative: Teachers are the most important ingredient for ensuring that schools have the capacity to offer computer science courses that lead to equivalent course experiences for all students. Access to computer science was constrained by the availability of qualified teachers. In addition, the consistency of student outcomes before and after the implementation of the graduation requirement rested upon the consistency and quality of the professional development. CPS was able to achieve success in the implementation of the graduation requirement policy because the district focused on expanding access to high quality professional development, leading to an increase in the number of qualified teachers as a means to build school capacity to offer computer science. Likewise, for school districts considering enacting a policy like a high school computer science graduation requirement, it is essential that districts have a plan for capacity building before enacting such a significant policy, given the critical role teachers play.

OPEN QUESTIONS AND NEXT STEPS 

After the release of the report, members of the research team met with the entire Department of Computer Science to discuss the results of the analyses and next steps. Two major interrelated open questions emerged from the discussion: 

1. Given that one of the mechanisms by which students can secure a waiver from the computer science requirement is through participation in career & technical education (CTE) and International Baccalaureate (IB) programs, members of the Department of Computer Science raised concerns around the possibility that there may be inequitable access to advanced computer science coursework at schools offering CTE information technology pathways. How does the waiver process impact equity of access to computer science? 
2. A second observation emerged around possible barriers to access. In particular, schools with higher percentages of Black/African American students are less likely to offer an AP computer science course. What additional factors affect access to and experience of advanced computer science coursework?

CAFÉCS will host summer fellows in 2023 to investigate these open questions.

This article was written by members of the CAFÉCS leadership team: Andrew Rasmussen is Miscellaneous Employee at Chicago Public Schools; Kristan Beck is Director of Computer Science at Chicago Public Schools Department of Computer Science; Don Yanek is a teacher at Mather High School; Steven McGee is President at The Learning Partnership; Lucia Dettori is Interim Dean at DePaul University; Ronald Greenberg is Professor of Computer Science at Loyola University; Dale Reed is Clinical Professor in Computer Science at University of Illinois Chicago; and Erin Henrick is President of Partner to Improve.

REFERENCES

Boda, P.A. & McGee, S. (2021). Broadening participation and success in AP CS A: Predictive modeling from three years of data. In Proc. ACM Tech. Sym. Comp. Sci. Ed. (SIGCSE’21). ACM, Virtual Event, USA, 6 pages. https://doi.org/10.1145/3408877.3432421

City of Chicago, Office of the Mayor. (2013, December 9). Mayor Emanuel and CPS CEO Barbara Byrd-Bennett announce comprehensive K–12 computer science program for CPS students [Press release]. https://www.chicago.gov/city/en/depts/mayor/press_room/press_releases/2013/december_2013/mayor-emanuel-and-cps-ceo-barbara-byrd-bennett-announce-comprehe.html

Dettori, L., Wilkerson, B., Chapman, G., McGee, S., Winters, R., Brannon, F., Yin, D., Williams, T., Park, A., Lansford, L., Easter, J., Dunketell, J., Pham, D., Kang, S., & Beck, K. (2022). Celebration of the Success of the CPS Computer Science Graduation Requirement [Webinar]. Chicago, IL: Chicago Alliance for Equity in Computer Science. https://doi.org/10.51420/report.2022.1

Dettori, L., Steinbach, T., Reed, D., Yanek, D., & Greeberg, R. (2011). Collaborative Research: Type I: Taste of Computing: Adding a CS Entree to the Education Choices in a Large Urban School District [Grant]. https://doi.org/10.51420/grant.2011.1

Fletcher, C. L., & Warner, J. R. (2021). CAPE: a framework for assessing equity throughout the computer science education ecosystem. Communications of the ACM, 64(2), 23–25. https://doi.org/10.1145/3442373

Goode, J., Margolis, J., & Chapman, G. (2014). Curriculum is not enough: The educational theory and research foundation of the exploring computer science professional development model. In the Proceedings of the 45th ACM Technical Symposium on Computer Science Education, 493-498. http://www.exploringcs.org/wp-content/uploads/2010/09/Curriculum-is-not-Enough.pdf.

Johnson, M., Chapman, G., Dettori, L., McGee, S., Peterson, K., & Rasmussen, A. (2022, May). Hybrid/Online ECS: An Equity-Focused Credit Recovery Course [Paper presentation]. RESPECT annual conference. Philadelphia, PA.

Lewis, C., Henrick, E., Friedkin, S., & McGee, S. (2022). Model Variation in Inquiry Processes. In D. J. Peurach, J. L. Russell, L. Cohen-Vogel, & W. R. Penuel, The Foundational Handbook on Improvement Research in Education. Lanham, MD: Rowman & Littlefield.

Margolis, J., Ryoo, J. J., Sandoval, C. D. M., Lee, C., Goode, J., & Chapman, G. (2012). Beyond access: broadening participation in high school computer science. ACM Inroads, 3(4), 72-78. doi: 10.1145/2381083.2381102

McGee, S., Greenberg, R. I., Reed, D. F., & Duck, J. (2013, Summer). Evaluation of the IMPACTS computer science presentations. Journal for Computing Teachers. 

McGee, S., Greenberg, R. I., Dettori, L., Rasmussen, A. M., McGee-Tekula, R., & Duck, J. (2018). An examination of the factors correlating with course failure in a high school computer science course. The Learning Partnership. https://doi.org/10.51420/report.2018.1

McGee, S., McGee-Tekula, R., Duck, J., Dettori, L., Greenberg, R.I., Reed, D.F., Wilkerson, B., Yanek, D., & Rasmussen, A.M. (2018, April). Does Exploring Computer Science Increase Computer Science Enrollment? [Paper presentation] American Education Research Association, New York City. https://doi.org/10.51420/conf.2018.1

McGee, S., McGee-Tekula, R., Duck, J., McGee, C., Dettori, L., Greenberg, R.I., Snow, E., Rutstein, D., Reed, D., Wilkerson, B., Yanek, D., Rasmussen, A., & Brylow, D. (2018).  Equal Outcomes 4 All: A study of student learning in Exploring Computer Science. Proceedings of the 49th SIGCSE Technical Symposium, Association for Computing Machinery, 50-55. https://doi.org/10.1145/3159450.3159529

Reed, D., Wilkerson, B.,Yanek, D., Dettori, L., and Solin, J. How Exploring Computer Science (ECS) came to Chicago. ACM Inroads, 6(3):75–77, September 2015.

Zumbach, L. (2013, December 10). CPS making computer science core subject. Chicago Tribune.

Suggested citation: Rasmussen, A., Beck, K., Yanek, D., McGee, S., Dettori, L., Greenberg, R., Reed, D., & Henrick, E. (2023). How has the Chicago Public Schools’ Computer Science Graduation Requirement Impacted Students? NNERPP Extra, 5(2), 2-10. https://doi.org/10.25613/DPCN-BB11