Test changes ‘inflate’ API scores

State failed to adjust for easier test
By

On annual report card day Tuesday, the State Department of Education announced that schools averaged double-digit increases on the Academic Performance Index, the chief measure of school progress, with highest gains for minority children. While this is very good news, a retired school testing executive from Monterey with an eye for data says “Curb some of that enthusiasm.” More than a quarter of the gain ­over the past three years – 11 of 39 points on average  for elementary schools  – is undeserved. The reported figures are the result, Doug McRae says, of changes in methodology that “inflated” scores.

Starting three years ago, students with disabilities who did poorly on the annual California Standards Tests, the chief component of schools’ and districts’ API scores, started taking a new and easier version of the test: the California Modified Assessments. On the CMA, large numbers scored proficient, whereas they had previously scored below basic and far below basic on the CSTs or STAR tests. As a result, schools were given credit for higher API scores than they should have received.

McRae, who also has sharply criticized the Department of Education’s method for awarding money to low-performing schools, doesn’t dispute state officials that the CMA is a good test for disabled students; to the contrary, he writes in a six-page analysis “Without question the CMAs provide more meaningful individual student information” than the CSTs. But he says that the state should have adjusted for changing tests, either by ratcheting down the latest API scores, or by raising the scores of previous years. The state did nothing to correct the distortion. (In his analysis, McRae suggests a relatively easy way that adjustment could have been made.)

The CMA has been phased in over the past three years. According to McRae, 145,000 students in grades 3 through 11, including 4.5 percent of third through eighth graders, took the test in 2010. API scores range from 200 to 1000. Were it not for the CMA effect, elementary schools scores would have risen from 761 to 789, not 800. Over the past two years, scores for grades 7 and 8 would have risen 22 points, to 757, not 765 as the state is reporting.

During a press conference on Tuesday, Deputy Superintendent Deb Sigman said she was aware of McRae’s data and defended the CMA as the better test for students with disabilities. (McRae doesn’t disagree.) She declined to comment when I asked whether McRae’s numbers were correct. I forwarded his analysis and spreadsheets to the Department on Tuesday, but heard nothing back.

The news on API scores still would have been good, just not as impressive, without the inflated CMA impact. Superintendent of Public Instruction Jack O’Connell reported that all student subgroups improved between 11 and 17 points in 2010. Hispanic students’ and low-income students’ scores rose 17 points compared with 11 points for White students, creating a slight narrowing of the achievement gap. African-American students’ scores rose 15 points.

See here for the tables and complete breakdown of the API scores, as the state calculated them.

For a detailed explanation of McRae’s calculations, check here, here, here and here.

3 Comments

  1. Federal regulation allow only for up to 2% of students to use modified assessments like the CMA for accountability (AYP) purposes. This limit reflects the best research available and has been validated by empirical studies (see http://www2.ed.gov/rschstat/eval/disadv/nclb-disab/nclb-disab.pdf ).
    CMA represents a way for California to game the system by administering this assessment to more students than necessary.  The conspicuous silence of Deb Sigman on this issue is very bothering in light of 4.5% CMA taking, that Sigman certainly knows is more than twice the research-based federal limit. The onus should be on the state to monitor that school districts (and the state itself!) are not abusing the CMA to claim more gains than they deserve, and harm special ed students in the process. CDE record on such openness is less than stellar.

    Report this comment for abusive language, hate speech and profanity

  2. Doug McRae’s assertions misrepresent the integrity of the state accountability system and ignore completely the key component of the API system, measuring improvement in student achievement from one year to the next.
     
    Students with disabilities can access California’s assessment system in a number of ways. Many students with disabilities access the California Standards Tests (CSTs) with or without accommodations or modifications. For other students, alternate assessments are available and are more appropriate assessments for them to demonstrate what they know and are able to do.
     
    One of the alternate assessments, the California Modified Assessment or CMA has been introduced into the statewide assessment system over the past three years following federal law that encouraged states to develop assessments “to more appropriately measure the achievement of certain students with disabilities.” In addition, the California Alternate Performance Assessment or CAPA is available for students with the most severe cognitive disabilities.
     
    Only certain students are eligible to participate in these alternate assessments. Participation is guided by the student’s individualized education program (IEP) team as well as by the eligibility criteria adopted by the State Board of Education.
     
    The CMA was first administered in 2008 to students in grades three through five in English-language arts (ELA) and mathematics and science in grade five. In 2009, it was expanded to grades six through eight in ELA, grades six and seven in mathematics, and grade eight in science. And this past spring, the CMA was administered in grade nine in ELA, Algebra I, and grade ten in life science.
     
    Data from the Standardized Testing and Reporting or STAR Program show that the number of students participating in the CMA is between 2% and 3% for English-language arts and mathematics.
     
    The Academic Performance Index or API compares student performance in a baseline year (i.e., Base API) to student performance the next year (i.e., Growth API). The Base and Growth API scores within an API cycle are computed using the same methodology and the same assessments. It is through this comparability that we achieve an “apples to apples” comparison. 
     
    The CMA for grades three through five was introduced into the accountability system starting with the 2008 Base API (based upon results from the 2008 testing cycle). The 2008 Base API is compared to the 2009 Growth API (based upon results from the 2009 testing cycle) to measure improvement in student achievement. Thus, the performance of students who took the CMA in grades three through five in 2009 is compared to the performance of students who took the CMA in grades three through five in 2010. As a result, the API scores only increase if the students are performing better on the same assessment one year to the next. The availability of the CMA has led to higher percentages of students with disabilities being reported in the STAR system as proficient or above. This is due to the fact that students with disabilities are now being assessed with an appropriate instrument against appropriate performance standards and these students are in fact making achievement gains. The API is a reflection of that progress.
     
    Because student performance improved on the CSTs and the CMA in 2010, we saw an overall increase in California’s API scores. The CMA is working as intended — to appropriately measure gains in achievement by students who have moderate to severe learning disabilities. 
     
    Developing the CMA was a task that was long overdue. With the introduction of this assessment, we should be celebrating the fact that California now has a series of assessments that allow our students with disabilities to demonstrate what they know and are able to do, and I am confident that our accountability system is providing an appropriate measure for gauging the progress of our schools and districts.
     
    Deborah V.H. Sigman
    Deputy State Superintendent
    Curriculum, Learning, and Accountability Branch
    California Department of Education

    Report this comment for abusive language, hate speech and profanity

  3. I do challenge the integrity of California’s API trend data, not by misrepresentations but rather by raising an issue that the California Department of Education has not addressed to date.  Deb Sigman’s comment does not address the problem that I raised with API gain data.  The problem that I raised has to do with multi-year trend data, not 1-year API Base-to-Growth gain data.  The 1-year API Base-to-Growth gain data that Deb describes are reasonable apples-to-apples gain data.  Deb’s comment is fine with only one note – that the percentage of students participating in STAR via CMAs (the 2-3 percent she claims) is understated in that the percentage participating for grades 3-8 (where CMA has been fully implemented) is now 4.5 percent and likely still growing, while the percent data Deb cited is based on a grades 2-11 denominator (not a grades 3-8 denominator).  When one looks at multi-year API trend data, the growth data are not adjusted for replacement of the more rigorous CST tests with the easier CMA tests for 4.5 percent of the enrollment (grades 3-8) now taking the easier tests.  Finally, I might note that other states have had problems with larger than expected numbers of Spec Educ students [compared to the advertised expected percentages when modified assessments were initially being developed] taking their modified assessments [see Pearson's  TrueScores blog for details]; while the overall numbers of Spec Educ students taking CMA perhaps should be an issue addressed by CA, it is not the direct issue I raised in terms of inflated or distorted API trend data. 

    Report this comment for abusive language, hate speech and profanity

"Darn, I wish I had read that over again before I hit send.” Don’t let this be your lament. To promote a civil dialogue, please be considerate, respectful and mindful of your tone. We encourage you to use your real name, but if you must use a nom de plume, stick with it. Anonymous postings will be removed.

10.1Assessments(37)
2010 elections(16)
2012 election(16)
A to G Curriculum(27)
Achievement Gap(38)
Adequacy suit(19)
Adult education(1)
Advocacy organizations(20)
Blog info(5)
CALPADS(32)
Career academies(20)
CELDT(2)
Character education(2)
Charters(82)
Common Core standards(71)
Community Colleges(66)
Data(25)
Did You Know(16)
Disabilities education(3)
Dropout prevention(11)
© Thoughts on Public Education 2014 | Home | Terms of Use | Site Map | Contact Us