CST results need an *
Fewer special ed students taking CSTs
Since 2008, increasing numbers of low-performing special education students have been taking a new and easier standardized test rather than the California Standardized Tests. While it’s appropriate for many of them to take the California Modified Assessment, the exclusion of 185,000 special education students this year had the effect of appreciably boosting the statewide CST scores.
That’s been the contention of Doug McRae, a retired executive who oversaw the design of standardized tests nationwide and has raised the issue with the State Board of Education. On Monday, he once again did the calculations to show the impact.
Factoring in the CMA effect, the statewide increase in proficiency on the CSTs in 2011 would have been 1.6 percent, not 2.05 percent, as the state Department of Education has stated. And the four-year effect of diverting special education students to CMA had an inflationary effect of 26 percent, according to McRae. The cumulative increase in proficiency would have been 8.95 percent instead of 11.25 percent, had the students taken the CST instead of the CMA.
California is one of two dozen states to create a modified assessment for special education students, as permitted by the federal Department of Education. Only those students who do poorly on the CST the previous year, with a below basic or far below basic score in math or English Language Arts, are eligible to take the CMA in that subject area. The state introduced the CMA for grades 3-5 in 2008, grades 6-8 in 2009, and for grades 9-11 over the past two years. The number of students taking the CMA increased from 39,000 to 184,000 in four years (see chart).
(source: Doug McRae)
The federal Dept. of Education expected that no more than 2 percent of a state’s student body would take the modified exam. This year, 4.4 percent of students took the CMA, more than 40 percent of special education students. According to McRae, “the number of students taking the CMA far exceeds initial plans for this test, and are likely to increase to an alarming percentage of the entire group of Special Education students in California.”
(The CMA data are part of McRae’s analysis of the latest CST scores. Readers interested in reading more can find the full text here.)
Reading through the sample tests, the modified test is still requiring some definite learning and ability, though they are distinctly easier.
I am curious if students are ever given both tests in the same year, and how much difference is seen in the results. (I would suspect it might matter which test was given first, too, because practice is a factor for some kids.) Do kids routinely jump up one cohort? Is it typical that their level stays the same? Do they ever drop cohorts or jump 2 or more cohorts?
Report this comment for abusive language, hate speech and profanity
Good questions, el. I hope readers with expertise in the area can help out here.
Report this comment for abusive language, hate speech and profanity
To el: To my knowledge no students have been administered both tests in the same content area the same year. In terms of differences of results for the two tests, that question is addressed in my Initial Observations document to which there is a link at the end of John’s post [thank you, John, for the link], on the bottom of page 4. The short answer is that we don’t know the degree of differences in the results, but it is possible to generate estimates for those differences and policymakers badly need this information to use CMA results with integrity in things like API calculations as well as other contentious things like, for instance, use of STAR scores as alternative means for the CAHSEE graduation requirement.
Report this comment for abusive language, hate speech and profanity
I’ve got respect for Doug McRae, but knowing some students who had to take the regular test… who looked at it mostly blankly the entire day , lost, while everyone hoped they could mark something useful (and missing recess, etc. as per testing directions and the maximum allotted time) and others that cried and cried, and those whose fear when they saw the test brought on some other physical symptoms…. do we need an asterisk that we don’t deserve the bump, or is it that the previous results might be unfairly low as so many special ed students have previously been required to take the regular test?
We are talking about a test that doesn’t have 1 right answer and 3 wrong answers in elementary school, one that might look to see if students have basic knowledge. We’re talking about one that mimics an adult test with little trickery involved, with 1 right answer, and 1 “almost right” answer depending on a kid’s point of view might appear to be the better answer. Some of the questions are poorly written, so even adults could debate which of the two is the better answer. Some have some obvious bias, and special ed students risk being at a disadvantage in more than one category on the same question.
Report this comment for abusive language, hate speech and profanity
Thank you, Pamela Trounstine, for stating the obvious and most important facts: special ed kids are best served by taking the “alternate” test and whatever its effect on overall CST scores may be, it is the right approach. I hope no argument will erode California’s commitment to this sound educational practice.
Report this comment for abusive language, hate speech and profanity
Do any kids score proficient or advanced on the Modified Assessment?
Report this comment for abusive language, hate speech and profanity
@el: Yes. On the order of 20%-35% of students get proficient and advanced on the ELA/Math CMA. That does, indeed, provide some way to estimate the amount that CMA expectations are diluted vs. CST, as the pool of participants is drawn exclusively from CST’s below basic and far below basic. On the other hand one needs to exercise caution as those low CST scores were earned in prior years and one could argue that at least some students may have improved. Further, CMA’s underlying student distribution is probably not “normal” (in the Gaussian sense) and hence comparing test difficulty based on equating population percentages is not the best measure. (For CMA results, go the the STAR site and select CMA as the “test”).
@Pamela Trounstine: I don’t think anyone is arguing against the use of CMA. What Doug McRae and John Fensterwald point out is that (a) what seems like excessively large fraction of SPED students get to use CMS, close to 5% vs. 2% federal expectations based on empirical study, and (b) the variability among counties of that percentage varies enormously from 20% to 70% of SPED students, indicating a probable misuse and certainly uneven selection criteria.
Report this comment for abusive language, hate speech and profanity
@ze’ev: your assumption that the fact that kids can jump to the proficient and advanced levels means that expectations are diluted is questionable. It’s not THAT much easier. Maybe one level. But 3-4 levels?
By going from 4 answers to 3, random guessing will be more likely to get a correct answer. And the questions are a bit easier. You can see kids slipping up from Far Below Basic to Below Basic on that alone. But not from those bottom levels up to Proficient or Advanced.
One might also point out that the additional whitespace, the change in font, and possibly (I didn’t read enough questions to know) a more straightforward answer set – ie, fewer trick questions – might allow kids to show that they really did learn something last year after all.
Report this comment for abusive language, hate speech and profanity
@el: I am not sure I understand your last comment. It seems to say two opposing things at the same time. Or I am misreading it.
In any case (assuming I understood you) I agree that jumping two or more levels is unreasonable for more than an occasional kid, so to me it does indicate that CMA is significantly easier, and not only because of differences from CST like 3 vs 4 choices per item. I simply wrote what I thought might be a counter argument by those that will claim they are more or less equivalent.
But I disagree with your calling CST distractors “trick questions.” They only serve as “tricks” for those that can’t do the straightforward work to solve the items and guess instead. CST items are intentionally not “tricky” and the purpose of distractors is to minimize the opportunity for students to succeed by guessing.
Report this comment for abusive language, hate speech and profanity
Testing data is always skewed no matter what school districts or states say. When my son was attending Marquez, one of the special day class teachers would “suggest” that parents let their kids “pass” on taking the tests due to stress and test anxiety. It allowed the school to post glowing scores since the kids who would bring down the averages had already been eliminated by “counseling out” tactics. Teachers who do their jobs properly with these kids know a student’s abilities and teach to what a child needs. Testing is out of control and is an industry (NCLB was created for Neil Bush to make a buck with his publishing company.) Testing does nothing to help a child and is only being used now to try and close “low performing” schools for charter conversion and to fire teachers. Kids with special needs are always lowest performers in any test, anytime – some will never “get it” or “improve”. To punish a school or teacher because a student has disabilities and will never process information as the “norm” sees fit – is insane. The LAUSD’s “100% graduation” goal is insane and unreasonable when factoring in our students. It just won’t happen. Teach them what they are able to learn and let them enjoy it. Give them opportunities to thrive in job training/living skills situations that are meaningful to them. Get off the testing bandwagon. It only stresses kids and we are not doing them a service by wasting so much time in the school year teaching to a test when critical thinking skills are so necessary. Teach them how to read contracts, rental agreements, what it means to open a credit card account with 21% interest rates. Teach them to be productive citizens by learning the governing process and their place in society. Civics funding was canceled during the Reagan era and we are seeing the effects. Arts! Music! Shop! Electronics! Physical Education! Those are the classes that kept kids in school who were not college bound in the past. We need plumbers, cooks, auto mechanics and others who are clever with their hands. Tests do nothing to “help” children, especially those with disabilities. They are always and consistently the lowest scoring group in any test across the board. And why should a “mandatory” CAHSEE exam determine whether a student graduates or not? How often do any of us use Algebra in our daily lives? A student’s entire K-12 experience boils down to a CAHSEE exam? Show a body of work to prove how a student has progressed over the years. That makes more sense for a student with disabilities. They should graduate with their peers and be allowed to participate in ceremonies, proms and other activities.
Report this comment for abusive language, hate speech and profanity
I think limited testing has value for parents, students and teachers. I think CAHSEE has value as a pro forma hurdle to be jumped in high school. There is value in a kid’s measuring up. But I’m entirely with Sonja on the way we should be dealing with special ed kids: to be worrying if the numbers of special ed students who opt out of the CST are skewing results upward? That is insane. What we need to be worrying about is a richer and more humane curriculum.
Report this comment for abusive language, hate speech and profanity
Frances: I found the issue that Doug raised to be significant, not insane. Since the state, districts, teachers and parents are relying on CSTs to measure student performance, there needs to be consistency from year to year. Shifting groups of students out affects the measurement and should factored in. That’s not casting a judgment on the need and benefit of the CMA.
However, there is a larger issue here. Depending on the students’ individual circumstances, advocates of special ed students often don’t favor having students take the CMA. They worry that once students start taking the easier test, they won’t be encouraged to reach their potential. That’s the potential problem with having more than 40 percent special ed students taking the CMA.
Of all the tests that the state administers, the high school exit exam is probably the least useful, since it fools students into thinking they are ready for a world requiring more skills than they have.
Report this comment for abusive language, hate speech and profanity
EducGuesser: Thanx for the defense of my sanity. But I have to comment on your last comment that CAHSEE is probably the least useful of the state tests. CAHSEE is a minimal skills test, in a sense not unlike a DMV drivers license test designed to keep incapable drivers off the road. If students are fooled into thinking CAHSEE addresses anything more than minimal skills, then someone isn’t communicating the purpose of CAHSEE or the level of skills it measures. By way of contrast, STAR CSTs are tests based on high expectations, skills we expect for readiness to deal with challenges the future may bring (the current buzzwords are “college and career readiness”). About ten years ago when CAHSEE was first on the drawing board, the State Board heard a report from one of the co-chairs of a CAHSEE advisory committee, a respected LEA Superintendent from the LA area. After his report, he fielded an extemporaneous question from a board member on STAR, which was then 3-4 years old. The question was — what kind of effects are the STAR program of high expectations having in the trenches? His answer — we see good movement to upgrade instruction and achievement at the Elementary School level, there are signs of progress at some Middle Schools, but at the High School level it is like “trying to move a cemetery.” My point is — STAR still isn’t a primary influencer for instructional improvement and higher achievement at the high school level, but CAHSEE’s minimal skills test has indeed “moved a cemetery” for many previously very low-achieving high school students. From this perspective, CAHSEE has been more valuable than STAR at the high school level. However, I will grant that for the full range of expected high school achievement, STAR is the more valuable instrument, although I’m very frustrated that policymakers refuse to address the obvious unneeded redundant test administations at the high school level, the refusal to coordinate the STAR and CAHSEE programs for efficiency and common sense reasons, and finally the ineptitude in addressing the issue of alternatives for CAHSEE for our Special Education kids.
Report this comment for abusive language, hate speech and profanity
Doug: As opposed to CSTs, the high school exit exam does have real consequences for those who take it. Therefore, it has motivated many low-performing students to work hard on their basic skills in English and math (you can pass he exam without knowing algebra). My point is that it needs to measure what students need to survive in a competitive work world or to go on to some form of higher education. I am pleased that the Superintendent’s Blueprint for Great Schools mentions the need to revisit the content of CAHSEE.
Report this comment for abusive language, hate speech and profanity
Here’s the next wave–California is one of two states, apparently, that developed modified tests for students with disabilities (SWD), the California Modified Test (CMT). These tests, by definition, fundamentally alter the California Standardized Tests. Simply stated, they lower the standard. As a result, students do better on the modified test than the standard test. And these scores have been added to the standard testing in California–the California Standardized Test (CST), raising the state’s results. There is the rub. Only 2% of those scores should have been added in, with the rest going against California’s requirment to test all students (actually, 95% of all students.) If states do not do that they cannot meet AYP.
While federal law allows states to test some (a small percentage of) SWD on modified tests, the limit on reporting these scores is 2%–that is, of all students tested on modified tests, only 2% can be counted in state reports that are geared to meeting No Child Left Behind’s AYP (adequate yearly progress) benchmarks. Apparently, in California the percent counted is upward of 4.4%. By doing this, the state scores have gone up and the federal standards are breached.
To fix this runaway use of the CMT, California can move to count only 2% of these results and inform schools that if they test more of these students on the CMT, the schools will become inelible for AYP results. That is, for all the rest of the students, they are marked ‘not participating’ which lowers the state’s AYP rating. Following that path would end the runaway use of CMTs, I suspect.
Report this comment for abusive language, hate speech and profanity
Mariam raises a very good point and potential constraint for overuse of CMAs in the CA STAR program. It had not occured to me that the federal 2% rule for CMAs applied to the participation rate requirements for AYP, thus penalizing perhaps not only the state as a whole but perhaps also districts and schools by reducing participation rates by any use of CMAs beyond 2% of total and/or subgroup enrollments. If such is the case, and districts/schools fail to make AYP because they fail the AYP’s participation rate requirement, then indeed Mariam’s suggestion to inform schools of this penalty for overuse of CMA will serve to dampen overuse of CMA. Anyone out there know more about how the federal 2% rule will affect district/school AYP participation rates than I do? It is probably one of those fine points in California’s NCLB accountability notebook that is approved by the feds each year.
Report this comment for abusive language, hate speech and profanity
Just to clarify, the law does NOT limit the number of students with disabilities who take the modified tests of either type, as it’s up to local IEP teams to decide how the student will be tested.
Since 2003, the ED has planned for two types of alternate tests.
For the first group, students with significatn cognitive disabilities, the law limits the number of scores a state or district can count as part of AYP. It’s 1% of students with significant cognitve disabilities (these are not those taking the CMT presumably).
For the second group, presumably those who take the CMT, the ED allows another 2%.
For both types of tests, it’s important to remember that the cap applies to the percent of proficient (or above) scores from alternate achievement assessments that the state can use for AYP designation.
So, even if many many students with disabilities take the CMT, only 2% of their scores can be counted toward AYP. In addition, these large numbers raise problems for the participation rate. The numbers of students over that 2% will be counted against the school’s participation requirements. Schools, after all, need to test 95% of their students for AYP. If many students take modified tests (not the state standard) neither their participation nor their scores not count. This is a serious double hit. That should end the practice, don’t you think?
I hope this clarifies it. If you need a citation to the NCLB, let me know.
Report this comment for abusive language, hate speech and profanity
One thought….I wrote an LRP special report about these requirements in 2005, Meeting NCLB’s Mandates: Your Quick-Reference Guide to Assessments and Accountability and was very much up on the details then. However, I have not carefully reviewed the law since then. While I am not aware of major changes, if there are any, my comments above don’t reflect them.
Report this comment for abusive language, hate speech and profanity
Finally! This is just what I was lokiong for.
Report this comment for abusive language, hate speech and profanity
KnmyBl alfxhatlttxn
Report this comment for abusive language, hate speech and profanity
The 1%/2% rules have not been changed as far as I know. Only made into routine requirements, while originally they were only piloted.
But the fact that California did not make sure that CMA enrollment is uniformly applied illustrates the general competence level of CDE’s Curriculum, Learning & Accountability Branch.
I heard that CDE stopped its cheating audits for STAR. If true, it goes hand in hand with the above.
Report this comment for abusive language, hate speech and profanity
Miriam: Your description of the 1%/2% rules for AYP comport with my understanding of the rules per the feds. But, I think states have some flexibility how they apply those rules at the district and school levels, and states can put that flexibility into their “accountability workbooks” that are then approved by the feds. I think California has some of this sort of flexibility in its accountability workbook, especially flexibility for schools with his concentrations of Spec Educ students (such as County Office schools) where the 1%/2% rules don’t make any sense at all. Also, the AYP participation rules do not apply to California’s statwide accountability system calculations, the API. And in California the API has higher profile for influencing district and school behavior than the AYP — that is because with our relatively rigorous content standards and relatively rigorous assessments [compared to other states], the AYP targets have become out-of-reasonable-sight for many schools and thus we have a relatively high percentage of schools in federal Program Improvement (determined by AYP, not API) with no reasonable hope for exit. Thus, schools focus more on their API numbers where they can see movement and gains — from this practical point of view, CA’s API system is far more influential for CA schools than the federal AYP system, in general. The bottom line is that if AYP participation rules are to serve as a constraint on overuse of STAR CMAs, somehow or the other the AYP rules have to be applied to CA’s API system as well as our version of the federal AYP system. That can be done via the governing body for the API rules which is the California State Board of Education with advice from a statewide advisory committee and recommendations from the SPI/CDE.
Report this comment for abusive language, hate speech and profanity
While researching one school’s one-year meteoric rise in API
score, it appears that a significant number of students weren’t tested this
time (I’m still trying to confirm this). As far as I can tell, the CDE doesn’t
report at the school level how many students are opting out, or taking the CMA.
Increased transparency would be helpful. Given the growing trend of parents
having their children opt-out, it would be helpful to have this information. Can
anyone can shed extra light on this?
Report this comment for abusive language, hate speech and profanity