School Closures Caused Large Declines in U.S. Test Scores

There’s ample evidence that the switch to remote learning not only harmed children’s education, but also led to large drop in U.S. school enrolment. Okay, you might say, but have those findings held up? Maybe more recent studies have reached less dramatic conclusions about the impact of school closures during the pandemic.

It appears not. The latest study – a working paper by economists from Brown and MIT – suggests that school closures were every bit as damaging as we originally thought.

Clare Halloran and colleagues analysed data on schools districts in twelve U.S. states from 2015 to 2021, and tracked changes in pass rates in maths and English for grades 3 through 8.

The authors began by checking whether the pandemic year was an outlier in terms of changes in pass rates, as compared to previous years. They confirmed that it was: the 2020–21 school year saw substantial declines in pass rates for both maths and English. Changes in pass rates for maths are shown in the chart below:

States are arrayed along the x-axis, while the y-axis gives the year-on-year change in pass rate. The units are percentage points, meaning that Virginia (“VA”) saw a staggering 32 percentage point decline in the percentage of students passing maths. In most other states, the decline was around 12 percentage points.

Next, the researchers examined whether school districts that switched to remote learning saw greater declines in pass rates than those that stuck with in-person learning. Once again, they confirmed that such districts did see greater declines. Changes in pass rates for maths are shown in the image below:

Each dot corresponds to the average change in pass rates in school districts that used a particular learning mode (most, intermediate or least in-person learning).

Looking at the chart on the left (for pre-pandemic years), the dots of different shades are all clustered around zero, indicating no relationship between changes in pass rates and learning mode instruction during the pandemic. This is not surprising, as learning mode during the pandemic can’t influence changes in pass rates during pre-pandemic years.

Turning to the chart on the right (for the pandemic year), you can see that – within each state – the black dots are higher than the grey or clear dots. This indicates that school districts which relied most on in-person learning saw the smallest declines in pass rates. (Those that relied least on in-person learning generally saw the greatest declines).

For example, school districts in Virginia that relied most on in-person learning saw pass rates decline by ‘only’ around 22 percentage points, whereas those that relied least on in-person learning saw them decline by more than 32 percentage points.

This basic pattern of results held up when the researchers carried out more sophisticated, multivariate analyses. Overall, math pass rates declined by 14 percentage points, but this declined would have been 10 percentage points (i.e., 71%) smaller if all school districts had stuck with in-person learning.

The researchers say their findings “may provide a caution when considering school closures in the future” – which is what I’d call an understatement.