Despite lower student performance, more Milwaukee schools meet or exceed expectations — after state officials move the goalposts
42% of MPS schools earned higher ratings because of newly relaxed standards
Top-line results from new Wisconsin Report Cards appear to show that a number of Milwaukee schools improved enough to move out of the state’s two lowest rating categories (Figure 1) — even as overall student achievement dropped (Figure 2).
How can this be? State officials quietly and without explanation changed the score ranges (Table 1), making it easier for schools to earn a higher rating level without actually improving student outcomes.
These new goalposts artificially inflated the performance ratings of 74 Milwaukee schools, about 28% of the city’s 261 publicly funded school buildings.
The citywide impact of this change is best seen by imagining which ratings schools would have received if the old score thresholds had been maintained. Figure 3 shows exactly this, indicating the new rating standards resulted in fewer schools falling in the two lowest categories, and instead earning a ‘Meets Expectations’ or ‘Exceeds Expectations’ rating.
The biggest beneficiary of this change is Milwaukee Public Schools, where 51 school buildings (42% of all MPS-operated schools) received a higher rating under the new rating standards than they would have under the old standards. Public Charter Schools and Private Choice Schools also benefited, but to a proportionally smaller degree: 9 charters saw improved ratings under the new standards (27% of all charters), as did 14 Choice schools (19% of all Choice schools). (Please see ‘About the Data’ below for an explanation of how schools are grouped by sector for this analysis.)
MPS likewise benefited on its District Report Card, where state officials have also applied the same easier criteria. MPS scored a 58.4 out of 100 in 2019, earning a rating of ‘Meets Few Expectations.’ In 2021, the district’s score decreased slightly to 58.1 out of 100 in part due to lower student performance — but its rating improved to ‘Meets Expectations.’
Five more things you need to know
1. This year’s report cards should be interpreted carefully.
Report Cards aggregate a variety of school performance data and synthesize it into an overall score out of 100 points. The primary component is students’ academic performance on standardized tests in the subjects of math and reading, which are mandated by state and federal law.
In a typical year, most schools see more than 95% of students take these tests.
The past two years have not been typical. Tests were canceled altogether in 2020 under a federal waiver. In 2021, participation varied widely, with 0% student participation at some schools and 100% at others.
The tests can only be taken in person. When tests were administered in spring 2021, MPS schools were offering only virtual instruction. At the time, district officials said they would not push for students to participate, and instead would leave the decision to parents. By contrast, most Public Charter Schools and Private Choice Schools were offering in-person instruction by the time the testing window arrived.
As a result, only about 50% of students citywide took the tests. As shown in Figure 4, participation was markedly lower in MPS schools than in Charter and Choice schools.
The bottom line: Lower and wide-ranging test participation rates among schools and across sectors create concerns about the accuracy of these Report Cards, especially in schools and sectors with lower test participation rates.
This situation could also create significant challenges for Public Charter School Authorizers like the City of Milwaukee, UW-Milwaukee, and Milwaukee Public Schools. The performance contracts between these entities and the Public Charter Schools they authorize sometimes include provisions that require schools to meet or exceed the test score outcomes of Milwaukee Public Schools. These contract provisions may no longer be enforceable, since it is not statistically valid to compare the performance of a school that had high test participation rates to schools directly operated by MPS, which had average test participation around 34% and participation as low as 0% at some schools.
2. This year’s report cards cannot be directly compared to prior years, or used to compare schools.
A primary use of Report Cards is to compare change in school performance over time, and to compare performance between schools. Unfortunately, this year’s report cards should not be used for either of these purposes.
Why? First, state officials made significant “under the hood” changes to how Report Cards are calculated — including relaxing the grading scale as previously discussed. This means overall school ratings and most Report Card sub-categories are not comparable to Report Cards published in prior years.
Secondly, as already noted, significant variation in 2021 school-to-school test participation rates means Report Card ratings are not comparable across schools.
One potential exception: The “Student Achievement” subcategory has not seen any methodological changes, and is therefore more comparable across years, but only for schools with higher test participation rates.
3. The pandemic negatively impacted student learning. The new Report Cards likely understate that impact.
New state test data and Report Cards finally quantify the negative impact of the COVID-19 pandemic on student achievement.
Report Card scores are based only on the performance of students who took the test. Among those students, citywide proficiency dropped from 20% to 18% in reading and from 18% to 11% in math.
The state’s WISEdash public data portal, unlike the Report Cards, reports test scores in a way that counts all untested students as not proficient. By this measure, proficiency drops were much more extreme, falling 19.9% to 10.2% in reading and from 17.7% to 6.5% in math. These outcomes are more consistent with drops seen in other states.
Unfortunately, as Figure 5 illustrates, test participation rates were lowest among more vulnerable student groups, like students with disabilities and those from low-income households. As a result, data on the current academic status of these students are the least reliable.
Low and variable test participation combined with relaxed standards for Report Card categories mean that these data almost certainly understate the extent to which students’ academic progress has been harmed by the pandemic.
4. The new data paint a stark and sobering picture of student academic performance in Milwaukee.
Looking past the top-line Report Card ratings, student achievement as depicted in Figure 6 show that fewer than 1 in 5 students can read and do math on grade level.
Figure 6 shows that students attending Private Choice Schools appear to have lost the least ground relative to the other two main school sectors. One likely explanation: In general, Private Choice Schools began offering at least some in-person learning for at least some students in fall 2020 and continuing throughout the year. A small number of charter schools also reopened buildings in fall 2020, but most opened later in 2020 or early 2021. Milwaukee Public Schools remained fully closed until mid-April 2021, with most high school students remaining fully virtual until September 2021.
Outcomes for Milwaukee students were poor before COVID-19, and pandemic-related learning loss has made matters worse. Figure 7 shows that gaps in student proficiency by race, income, and disability status remain unacceptably large, and in some cases have widened.
More than ever before, there is a critical need for high-quality schools to remediate and accelerate student learning over the next 2–3 years. Unfortunately, Figure 8 shows that students with disabilities, Black students, and students from low-income households remain most likely to attend low-quality schools.
5. Report Cards still contain a glaring methodological flaw: by legislative mandate, they significantly overweight student academic growth for high-poverty schools.
Report Cards rate schools in part on how much academic growth students make during a school year. That student growth score is balanced with a student achievement score, which is an absolute measure of whether students are performing on grade level.
In high-poverty communities, students in any given grade are much more likely to come into school performing below grade level. That means a high-quality school serving a high-poverty community must be especially good at helping students catch up by remediating and accelerating student learning to achieve significant academic growth. Catching up can take multiple years, so even the strongest of these schools likely has a number of students who are still below grade level in any given year despite their progress.
For this reason, it makes sense to consider student growth alongside student proficiency levels when evaluating the performance of schools. Wisconsin’s Report Card does exactly this for high-poverty schools, but to an extreme. By legislative mandate, Report Card ratings for nearly all Milwaukee schools weight student growth 9:1 over student achievement.
A high-quality school can’t just move students forward effectively, it must eventually catch them up to performing on grade level. Under the current 9:1 weighting of growth, some schools which are failing to get students fully caught up are nevertheless awarded top Report Card ratings. Even more perversely, some high-poverty schools with high levels of student achievement receive lower Report Card ratings because their already-caught-up students are growing at a more typical rate.
At the aggregate level, the current Report Card design has resulted in average Report Card ratings for Milwaukee schools rising over time, implying schools are getting better. However, over that same period student achievement has actually fallen, as shown in Figure 9.
Conclusion
Wisconsin residents deserve a school accountability system that provides transparent, accurate, useful, accessible, and consistent information about how schools and districts are performing over time and relative to one another. Unfortunately, the 2021 Report Cards provide few of those things because of four key issues:
- Changes in score ranges for performance categories artificially inflated Report Card ratings of some Milwaukee schools, creating a false sense of improvement — even as student achievement has actually gotten worse.
- Lower and wide-ranging participation rates on state achievement tests mean Report Cards can’t be compared school-to-school, and likely limit the ability to compare the 2021 Report Cards with those published in future years.
- Significant methodological changes to how Report Card scores are calculated will make for a stronger report card in future years but mean Report Cards published after 2021 can’t be compared with Report Cards from earlier years.
- Overweighting student growth in high-poverty schools awards top ratings to some schools which are failing to get students fully caught up, while punishing some schools where students already at grade level are growing at a more typical rate.
Where do we go from here?
Some of these things cannot be fixed and are temporary in nature. For example, we cannot go back in time to increase student participation in the 2021 state tests. The only solution is to wait until these flawed data are crowded out by newer and more complete data in future years.
There are other issues which can and must be addressed by state officials:
- Reverting to the previous Report Card score ranges for the five rating categories would be a relatively fast and easy fix.
- Re-examining how student growth and student achievement are weighted when rating high-poverty schools would be more difficult and potentially divisive, but the ultimate long-term benefit of a stronger, more accurate, and more reliable Report Card is worth the effort.
As Wisconsin’s leaders continue to perfect the state Report Card, they must thoughtfully consider its core purpose and its impact, consult with parents and other end-users, and make any new design choices in transparent ways.
For more information about City Forward Collective, visit www.cityforwardcollective.org or contact Isral DeBruin at isral.debruin@cityforwardcollective.org.
