Test changes ‘inflate’ API scoresState failed to adjust for easier test
On annual report card day Tuesday, the State Department of Education announced that schools averaged double-digit increases on the Academic Performance Index, the chief measure of school progress, with highest gains for minority children. While this is very good news, a retired school testing executive from Monterey with an eye for data says “Curb some of that enthusiasm.” More than a quarter of the gain over the past three years – 11 of 39 points on average for elementary schools – is undeserved. The reported figures are the result, Doug McRae says, of changes in methodology that “inflated” scores.
Starting three years ago, students with disabilities who did poorly on the annual California Standards Tests, the chief component of schools’ and districts’ API scores, started taking a new and easier version of the test: the California Modified Assessments. On the CMA, large numbers scored proficient, whereas they had previously scored below basic and far below basic on the CSTs or STAR tests. As a result, schools were given credit for higher API scores than they should have received.
McRae, who also has sharply criticized the Department of Education’s method for awarding money to low-performing schools, doesn’t dispute state officials that the CMA is a good test for disabled students; to the contrary, he writes in a six-page analysis “Without question the CMAs provide more meaningful individual student information” than the CSTs. But he says that the state should have adjusted for changing tests, either by ratcheting down the latest API scores, or by raising the scores of previous years. The state did nothing to correct the distortion. (In his analysis, McRae suggests a relatively easy way that adjustment could have been made.)
The CMA has been phased in over the past three years. According to McRae, 145,000 students in grades 3 through 11, including 4.5 percent of third through eighth graders, took the test in 2010. API scores range from 200 to 1000. Were it not for the CMA effect, elementary schools scores would have risen from 761 to 789, not 800. Over the past two years, scores for grades 7 and 8 would have risen 22 points, to 757, not 765 as the state is reporting.
During a press conference on Tuesday, Deputy Superintendent Deb Sigman said she was aware of McRae’s data and defended the CMA as the better test for students with disabilities. (McRae doesn’t disagree.) She declined to comment when I asked whether McRae’s numbers were correct. I forwarded his analysis and spreadsheets to the Department on Tuesday, but heard nothing back.
The news on API scores still would have been good, just not as impressive, without the inflated CMA impact. Superintendent of Public Instruction Jack O’Connell reported that all student subgroups improved between 11 and 17 points in 2010. Hispanic students’ and low-income students’ scores rose 17 points compared with 11 points for White students, creating a slight narrowing of the achievement gap. African-American students’ scores rose 15 points.
See here for the tables and complete breakdown of the API scores, as the state calculated them.