American Scientist

Thus far the research on schools and COVID-19 has been quite muddy, but two new studies are helping clear the waters. How can we make sense of the latest studies related to COVID-19 and schools? Think through the three C’s for critical thinking: comparison, chance, and context.

How credible is the comparison between those exposed to the “treatment” and those who aren’t? In these studies, the “treatment” is in-person schooling, so we’re interested in whether treated places—those that reopened school—have experienced different COVID-related outcomes relative to places where schools remained virtual or were in a hybrid mode. Like trials for new vaccines, the best way to test cause and effect for new treatments is randomization. This process ensures that the groups being compared are very similar in every away except for their mode of instruction. Of course, school districts are not too keen on being randomized to in-person or remote schooling, so we have to try the next best thing, sometimes called a natural experiment. We look for differences in exposure that are “as good as random”—for example places that are very similar in most ways but happen to differ in their school policies.

Two U.S.-based studies on schools and COVID-19 using this strong research design were recently released by education policy research centers, the National Center for Analysis of Longitudinal Data in Education Research (CALDER) and the National Center for Research on Education Access and Choice (REACH). In both studies, the researchers tried to compare apples to apples by comparing counties with similar characteristics and prior COVID-19 trends but with different choices regarding in-person schooling versus hybrid or all-virtual instruction. Imagine comparing COVID-19 trajectories in the “Twin Cities” of Minneapolis and St. Paul, if one city chose in-person and the other remote schooling. These statistical analyses to try to create “twin” counterfactuals for each location in the dataset.

The recent Morbidity and Mortality Weekly Report (MMWR) released by the U.S. Centers for Disease Control and Prevention (CDC) on January 13, in contrast, only described COVID-19 cases over time by age group, without testing whether those patterns were related directly to school reopening policies. This type of description can be informative for seeing whether kids have high or low rates in general, but it doesn’t tell us much about cause and effect.

How likely were the results to have arisen just by chance? We want to know that what we observe in data represents reality, rather than statistical noise. The best protection from faulty conclusions based on statistical noise is a large sample size. Looking at many school districts for which we know details about reopening, rather than just a handful, better ensures that we’re not just picking up a statistical fluke (such as a particularly COVID-unlucky school district).

The CALDER and REACH studies have large samples of school districts with data on school reopening policies. The CALDER Center study used data from Michigan and Washington, and the REACH study used data from the vast majority of U.S. school districts. The CDC MMWR analysis of COVID-19 incidence by age group has a large sample covering 44 states, Washington, DC, and three territories, but does not include any information on school reopening policies or instructional modalities.

How well do the results translate beyond the research setting—beyond the context of the study? To inform what we do, both in our individual decisions and in policymaking, we have to consider whether the findings of any particular study generalize beyond that specific study’s sample and setting. The best protection against a misapplication of results is replication across multiple geographies, time periods, and education systems. For example, when assessing the effect of different school reopening modes on COVID-19, it’s helpful to have data from lots of different places, both across the United States and around the world. These recent studies bolster the knowledge base because they have broad geographical coverage, large samples, and credible research designs, but they couldn’t account for how different schools approached the details of COVID-19 mitigation, such as ventilation standards or masking policies and their enforcement.

“The REACH study defined 'high prevalence' as more than 36 to 44 new COVID-19 hospitalizations per 100,000 people per week in the county.”

• Facebook
• Twitter

Previous studies were limited to Europe, mostly from the summer months, when case numbers and school occupancy rates were low and when mitigation measures such as distancing and ventilation were facilitated by warmer weather. The U.S.-based CALDER Center and REACH studies demonstrate just how critical context is: Both studies find evidence that in-person schooling does not contribute to COVID-19 spread and health outcomes when baseline community spread is low, but that in-person schooling does contribute to worse COVID-19 metrics when pre-existing prevalence is high. The REACH study defined “high” as more than 36 to 44 new COVID-19 hospitalizations per 100,000 people per week in the county.

With SARS-CoV-2 transmission high across much of the country, this new evidence is an important consideration in school closing and reopening policies. The REACH team provides a searchable database to gauge where your community stands relative to the problematic range of hospitalizations. Although the conclusions are less rosy than early evidence from the summer months in the United States and Europe, the findings are consistent with the fact that this early evidence came from contexts with low community spread.

Of course, science is a method, not a stable set of findings. We should expect new and potentially changing guidance as scientists learn more. This changing guidance is not suspicious; rather, it's a hallmark of the scientific method. We know that decisions about kids’ schooling, the ones both parents and school administrators are making, are complex and involve consideration of difficult trade-offs. These conversations are high stakes, heated, and characterized by uncertainty. We intend to present the latest evidence with transparency and humility, and hope it helps parents, policymakers, and school leaders as we all navigate these challenging times.

Editor's Note: This blog post is republished and adapted from the Dear Pandemic blog. Members of the Dear Pandemic team first wrote about school reopenings during the pandemic for American Scientist in July 2020.