{"id":4887,"date":"2019-10-08T14:01:07","date_gmt":"2019-10-08T14:01:07","guid":{"rendered":"https:\/\/prichard-backup.com\/July\/2019\/10\/08\/kentuckys-gap-rules-need-more-discussion\/"},"modified":"2022-09-27T19:49:18","modified_gmt":"2022-09-27T19:49:18","slug":"kentuckys-gap-rules-need-more-discussion","status":"publish","type":"post","link":"https:\/\/prichard-backup.com\/July\/kentuckys-gap-rules-need-more-discussion\/","title":{"rendered":"Kentucky&#8217;s Gap Rules Need More Discussion"},"content":{"rendered":"<p>Only 30 Kentucky schools have significant achievement gaps between African American students and the school\u2019s top scoring racial or ethnic group? That\u2019s what this year\u2019s school report cards say.<\/p>\n<p>I hope someday to live in a commonwealth where that claim makes sense.<\/p>\n<p>To me, this year, it doesn&#8217;t.<\/p>\n<p>I see 152 schools that had 2018-19 gaps of 20 points or more between African American students and other groups, including 49 with gaps of 30 or higher and two with gaps of 40 points or beyond.<\/p>\n<p>These gaps are based on a score that combines reading and math data on a zero to 125 scale, with full credit for proficient scores, half credit for apprentice results, and extra credit for distinguished work. Some parts of Kentucky school report cards call that number the proficiency indicator score, while others call it the gap rate, but it\u2019s the same calculation.<\/p>\n<p>Here\u2019s a chart showing six different types of gaps, again with many schools having gaps listed as not significant:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-9179\" src=\"https:\/\/prichard-backup.com\/July\/wp-content\/uploads\/2019\/10\/Screen-Shot-2019-10-07-at-4.49.46-PM.png\" alt=\"\" width=\"431\" height=\"450\" \/><\/p>\n<p>Our new state accountability regulation said gaps would matter if they were \u201cstatistically and practically significant,\u201d but the regulatory language never offered the mathematical specifics that would be used. The final criteria weren\u2019t clear to most of us until after the report cards came out. The Department&#8217;s\u00a0&#8220;<a href=\"https:\/\/education.ky.gov\/AA\/distsupp\/Documents\/Achievement%20Gap%20Identification%2020190927.pdf\">Accountability Gap Identification<\/a>&#8221; document (dated September 27) explains the technical details:<\/p>\n<ul>\n<li>\u201cA Cohen\u2019s <em>d<\/em> is used to determine statistical and practical significance. Cohen\u2019s <em>d<\/em> provides a measure of effect size for comparisons of groups with differing sizes and variability as seen in student groups across the state.\u201d<\/li>\n<li>Cohen recommended that a <em>d<\/em> of 0.2 be considered a small effect size, with 0.5 counted as moderate, and 0.8 as large.<\/li>\n<li>\u201cIn Kentucky\u2019s school accountability system, the 1.0 level is used to determine if the achievement gaps are statistically and practically significant.\u201d<\/li>\n<\/ul>\n<p>That puzzles me. I&#8217;m not fluent in this type of statistical analysis, but it sure sounds like Kentucky\u2019s rule is that moderate and even large gaps will not be considered significant.<\/p>\n<p>Kentucky is also treating gaps very differently from other achievement issues. When two <span class=\"highlight\"><em><strong>schools<\/strong><\/em><\/span>\u00a0differ by 20 points or so, they are rated differently on our school dashboard\u2019s proficiency indicator. When two <span style=\"color: #339966\"><em><strong>groups <\/strong><\/em><\/span>have the same kind of 20 point difference, that does not get dashboard attention. Here&#8217;s an illustration using real schools&#8217; data to show that inconsistency.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-medium wp-image-9172\" src=\"https:\/\/prichard-backup.com\/July\/wp-content\/uploads\/2019\/10\/Screen-Shot-2019-10-06-at-2.44.59-PM.png\" alt=\"\" width=\"450\" height=\"377\" \/><\/p>\n<p>Here&#8217;s my puzzle:<\/p>\n<ul>\n<li>Caverna High and Eminence High <span style=\"color: #008000\"><em><strong>differ by <\/strong><\/em><em><strong><span style=\"color: #008000\"><em>1<\/em><\/span>9.0<\/strong><\/em><em><strong> points<\/strong><\/em><\/span>, with proficiency indicators of <em><strong><span style=\"color: #008000\">35.5 and\u00a054.5<\/span><\/strong><\/em> respectively. The dashboard gives them\u00a0very different ratings for that indicator: one very low and one medium.<\/li>\n<li>African American and white student results <span style=\"color: #008000\"><em><strong>differ by 19.6 points<\/strong><\/em><\/span> at Jeffersontown High, with scores of <span style=\"color: #008000\"><em><strong>34.7 and 54.3<\/strong><\/em><\/span>, respectively. The dashboard says the school has no significant gaps.<\/li>\n<li>The school difference has a high-visibility impact on dashboard ratings. \u00a0<span style=\"color: #008000\"><em><strong>Why isn&#8217;t the very similar group difference worthy of dashboard attention?<\/strong><\/em><\/span><\/li>\n<\/ul>\n<p>Over and over, state leaders remind us that the school report cards are supposed to start important conversations. That they aren\u2019t a judgment. That no one gets more money for high ratings or less money for lower ones. That they\u2019re meant to create opportunities for communities to discuss issues that matter in their local schools.<\/p>\n<p>On gaps, I think our current system is closing off conversations that need to happen. When citizens look at their school report cards and see \u201cno significant gaps,\u201d that invites them to think they don\u2019t need to dig deeper to understand how the school is serving different groups of students.<\/p>\n<p>So I want to start a conversation, too, and here are my starting questions:<\/p>\n<ul>\n<li>Is a Cohen\u2019s <em>d<\/em> of 1.0 the right cut point?<\/li>\n<li>What was the reasoning that led to selecting that cut point?<\/li>\n<li>Should Kentucky choose a different cut point to reflect our serious intent to close these gaps?<\/li>\n<li>Even more radically, should we make a policy choice that gaps of 20 points are always substantive and definitely worthy of public engagement?<\/li>\n<\/ul>\n<p>I think those gaps, and the children caught in them, matter.<\/p>\n<p><span style=\"color: #999999\"><strong><em>Source notes<\/em><\/strong><\/span><br \/>\n<span style=\"color: #999999\"><em>This analysis draws on the <a style=\"color: #999999\" href=\"https:\/\/openhouse.education.ky.gov\/Home\/SRCData\">research data files<\/a> for the 2018-19 school report cards. Click on Assessments\/Accountability at the top of the page to locate the files I used:<\/em><\/span><\/p>\n<ul>\n<li><span style=\"color: #999999\"><em><strong>Gap<\/strong>\u00a0shows a score for each group with enough students at each school and then shows a gap calculation next to each of the historically underserved groups.<\/em><\/span><\/li>\n<li><span style=\"color: #999999\"><em><strong>Accountability Proficiency By Level<\/strong>\u00a0contains each school\u2019s percent novice, apprentice, proficient, and distinguished for each group in reading and mathematics. I used that data to verify that the scores in the Gap file do indeed use the formula for the proficiency indicator.<\/em><\/span><\/li>\n<li><span style=\"color: #999999\"><em><strong>Accountability Profile<\/strong>\u00a0includes the proficiency indicator score based on all students at a school and also provides a column identifying all the gaps counted as significant at that school.<\/em><\/span><\/li>\n<\/ul>\n<p><span><i style=\"color: #999999\">The use of Cohen\u2019s d and effect sizes of 1.0 or greater is specified in the Department\u2019s September 27 \u201c<\/i><a style=\"color: #999999\" href=\"https:\/\/education.ky.gov\/AA\/distsupp\/Documents\/Achievement%20Gap%20Identification%2020190927.pdf\">Achievement Gap Identification<\/a><i style=\"color: #999999\">\u201d document.<\/i><\/p>\n<p><i style=\"color: #999999\">To review data on a single school, go to <a href=\"http:\/\/kyschoolreportcard.com\">kyschoolreportcard.com<\/a>. After you select a school,\u00a0<\/i><\/span><span style=\"color: #999999\"><i>click the link that says &#8220;View Accountability Data&#8221; to see a dashboard of major indicators and a summary of the school&#8217;s significant gaps. Then click &#8220;Explore Data&#8221; and use the left-side menu to see\u00a0<\/i><i>achievement<\/i><i>\u00a0gaps by groups, including those not considered significant.<\/i><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Only 30 Kentucky schools have significant achievement gaps between African American students and the school\u2019s top scoring racial or ethnic group? That\u2019s what this year\u2019s school report cards say. I hope someday to live in a commonwealth where that claim makes sense. To me, this year, it doesn&#8217;t. I see 152 schools that had 2018-19<\/p>\n","protected":false},"author":4,"featured_media":4830,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"passster_activate_protection":false,"passster_protect_child_pages":"","passster_protection_type":"password","passster_password":"","passster_activate_overwrite_defaults":"","passster_headline":"","passster_instruction":"","passster_placeholder":"","passster_button":"","passster_id":"","passster_activate_misc_settings":"","passster_redirect_url":"","passster_hide":"no","passster_area_shortcode":"","footnotes":""},"categories":[78],"tags":[],"class_list":["post-4887","post","type-post","status-publish","format-standard","has-post-thumbnail","category-numbers-up"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/posts\/4887","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/comments?post=4887"}],"version-history":[{"count":1,"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/posts\/4887\/revisions"}],"predecessor-version":[{"id":5880,"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/posts\/4887\/revisions\/5880"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/media\/4830"}],"wp:attachment":[{"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/media?parent=4887"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/categories?post=4887"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prichard-backup.com\/July\/wp-json\/wp\/v2\/tags?post=4887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}