On January 30, the Prichard Committee sent the following letter to the Kentucky Board of Education requesting that they to reconsider the methodology used for reporting on “statistically and practically significant” achievement gaps in the accountability system and on the School Report Card.
The system’s current construct downplays inequities for Kentucky’s poorest and most under-served students and is therefor in direct conflict with the primary intention behind the state’s accountability model.
You can see President & CEO Brigitte Blom Ramsey’s comments to the KBE here:
https://www.facebook.com/prichardcommittee/videos/813364672473394/
January 30, 2020
Dear Chairman Karem and Members of the Kentucky Board of Education:
Thank you for taking on a new and important role in service to our children and our Commonwealth.
On behalf of the Prichard Committee for Academic Excellence, I write to ask you to renew the attention given to achievement gaps in the Kentucky School Report Card, with the explicit intention of alerting communities and families to education challenges that need their engagement.
Since 1983, the Prichard Committee has worked to study priority issues, inform the public and policy makers about best practices and engage Kentuckians in a shared mission to move the Commonwealth to the top tier of states for education excellence and equity for all children. It is from that lens that I implore you to lend serious consideration to the issue outlined below.
From the implementation of the Every Student Succeeds Act (ESSA) in 2015, to 2019 when the Kentucky Department of Education launched the Kentucky School Report Card, citizens, business leaders, families and other stakeholders conducted wide discussions of the best ways to report on school results. These discussions included broad agreement that the main purpose of the reporting was to inform families and communities and start important conversations about how to serve all students—regardless of race, socioeconomic background or disability—with increasing effectiveness.
The resulting regulation, 703 KAR 5:270, defined school ratings in which significant gaps would matter—but did not specify the standard of significance. The significance methodology used was not publicly discussed or announced until 2019 test scores were released and the school report cards were published. The methodology revealed then, we submit, was poorly designed and allows large gaps to go unnoticed.
Under the regulation, gaps are calculated by comparing reading and math scores for each student group, reported in a single number on a 0-125 scale. Here are four illustrations:
- For African American students, 170 schools delivered results twenty points or more below results for their white (non-Hispanic) classmates*, but only 28 were reported as significant.
- For Hispanic or Latino students, 66 schools delivered results twenty points or more below results for their white (non-Hispanic) classmates, but only 5 were reported as significant.
- For economically disadvantaged students, 489 schools delivered results twenty points or more below results for their classmates, but under the significance definition unveiled October 1, only 23 of those gaps were reported as significant.
- For students with identified disabilities, 472 schools delivered results twenty points or more below results for their classmates, but only 180 of those gaps were reported as significant.
Under this methodology, students, parents, community members, and educators are being actively told their schools do not have gaps that need sustained attention and collaboration. Kentucky’s current approach misleads our communities and, as a result, discourages shared problem solving. This is a disservice to our students, their families, and communities who strive for better and honest outcomes.
The regulation called for reporting on “statistically and practically significant” gaps, and then an administrative decision was made on how to implement that rule. The final decision was to look at effect size using a measure known as Cohen’s d and to require very large effect sizes before gaps are reported as significant. Cohen, the developer of the measure, recommended treating an effect size of 0.2 as small, 0.5 as moderate, and 0.8 as large—but 2019 Kentucky school report cards require an effect size of 1.0 before a gap between two student groups is reported as significant. That is, only gaps that are “larger than large” have been treated as significant. That decision, we submit, was a mistake.
We ask you to revise the regulation. Specifically, we ask you to make a policy decision on what size gap needs strong community attention, and then revise the accountability regulation to be sure gaps that size get robust attention.
Four brief amendments to 703 KAR 5:270 would be enough to make that change effective.
- Revise Section 4(8)(c) to add the words “of twenty points or greater” after “achievement gap,” so that the text will read “If achievement gaps of twenty points or greater are found in schools and LEAs earning a four (4) or five (5) star rating, the star rating will be reduced by one (1) star.
- Revise Section 5(6) to remove “statistically and practically significant” and add “of twenty points or greater,” so that the text will “Federal designations and achievement gaps of twenty points or greater will be reported for each school, LEA, and state.”
- Delete Section 1 (12), which currently says “Practical significance” means a measure of the differences between student groups has real meaning.”
- Delete Section 4(8)(c)1b, which currently says “Using a statistical analysis for each pair of comparison and reference groups, the department shall determine if a gap between the comparison group and reference group is both statistically and practically significant.”
We think twenty points is a reasonable standard for gaps that should lead to a dashboard signal that local attention and engagement may be needed. We have analyzed the 2019 accountability data in the attached document to illustrate how this standard would impact the number of 4- and 5-star schools.
We would also like to propose another step to draw attention to where we can do better for our students. We submit that results for each student group ought to be visible on the first page of the school report card. During the multi-year discussions that developed our current accountability system, there was steady discussion of showing a single chart that would make those group results very easy to find. At the time, the page being discussed was called “the dashboard,” but the key concept was that the gap data would be on the page that parents, educators, and others would see first. In the current version of the school report card, a viewer has to drill down four layers to find that information. Please help everyone involved with our schools see the challenges quickly. Please join us in calling for the primary data on group reading and math results to be visible on the first page of each school report card.
Since 2015, when Kentucky began work to design our new accountability system, the dashboard’s central goal has been to start important discussions about how to improve teaching and learning for each and every child. We have supported the new design based on the explicit commitment to make gaps a central part of those discussions. We were deeply saddened to learn that the current, unexpected approach to gaps has dishonored that commitment. Indeed, the current approach implies that difficult conversations are unnecessary in many situations where community engagement is urgently needed.
We respectfully ask you to correct this regulation so that all community members can easily tell when results delivered for some students are significantly disproportionate than those delivered for others, so that we can all contribute to the collaborative work of ensuring each and every student is adequately served – and so that we can all own our truth and actively work to improve.
Sincerely,
Brigitte Blom Ramsey
President & CEO
Attachments:
GapsSortedBySignificanceA
20 Point Model
* A few school report cards compare African American or Hispanic student results to those for Asian students or another race or language group. To demonstrate the problem quickly, this letter looks only at the reporting that uses white (non-Hispanic) students as the reference group for gap comparisons.
Comments are closed.