High-stakes tests based on honor
State drops monitoring methods
Nancy Dianna Jones may be among a handful of California educators who have read and absorbed all 143 pages of the state manual that spells out the rules for administering the STAR test and other high-stakes exams. Jones is administrator of support services for the Encinitas Union School District in north San Diego County, where she’s coordinated the standardized testing program for about seven years. She is emphatic when discussing the measures taken to ensure the integrity of standardized test results in the district.
California's District and Test Site Coordinator Manual covers every possible scenario from how to set up the testing room to what to do in the event of power outages and natural disasters. (click to enlarge)
“I do probably a good two-hour training for each of our site coordinators,” she said. “Our teachers know exactly what they may or may not do and we follow that to the letter.” Sometimes even beyond the letter. Although the Educational Testing Service (ETS), which has the testing contract for the state, allows some documents to be thrown away, Jones has schools send every paper to her office to make sure there are no accidents. “If something needs to be discarded, I discard it. If something needs to be shredded, I shred it,” she said. Under her watch, Encinitas has never so much as lost a test or a test manual.
The state’s safeguards to prevent cheating seem so thorough to Jones that she was surprised to learn that California hasn’t conducted any statewide security checks since 2008.
When the STAR testing budget was cut by $6.5 million, the Department of Education ended the contract with a company that made random visits to schools during testing periods. A year later it ended the computerized screening that searched for unusual amounts of erasures and other telltale signs of cheating by teachers, like the sweeping scandal in Atlanta that implicated about 200 educators at nearly 50 schools and the superintendent’s office.
“We rely on the districts to report to us. We don’t do any sorts of audits or anything like that; we don’t have the resources,” said John Boivin, administrator of California’s STAR testing program.
From irregularities to extreme situations
Los Angeles Unified School District is still reeling from the recent cheating scandal that rocked six charter schools operated by Crescendo. As the Los Angeles Times has been reporting, a group of teachers alleged that the schools’ founder, John Allen, “ordered principals and teachers to break the seal on state standardized tests last spring and use actual test questions to prepare students.”
Crescendo is an extreme case, said Boivin, adding that the state may have one or two of those situations each year. More often than not, the problems are irregularities stemming from teachers not following procedures and leaving a test unsecured while they go to the restroom, or not checking to make sure that every student has written their name on the test booklet. His office receives about 200 of those reports annually.
Even though most of those reports are made by other teachers and sometimes by students, Boivin is uncomfortable doing away with the more rigorous screening. “Sure,” he said, when asked if he suspects that some teachers are getting away with changing incorrect answers now that erasures aren’t automatically spotted and spit out of the computer. But the alternative wasn’t acceptable to the state Department of Education; it would have meant cutting back on assessments.
States need clear expectations and consequences
Relying on other schools and teachers to report wrongdoing is a pretty good method, according to Stanley Rabinowitz, WestEd’s director of Assessments and Standards Development Services. Schools are competitive when it comes to test scores, so they don’t shy away from calling in with their suspicions. But states need more formal procedures in place as well, said Rabinowitz.
“When I became New Jersey’s test director, my first crisis was a security crisis,” he recalled. “We had a secure test booklet and not all the test booklets were properly returned, and since a lot of the same items were used every year, that’s a particular problem.”
Rabinowitz took a hard line and implemented a meticulous tracking system. Schools couldn’t use regular mail and every item had to be signed for. He wanted to make sure people understood that they were accountable. And he had a tough-love way of making that clear.
“I wanted two licenses a year because I wanted to be able to publicize that two teachers or principals lost or had their licenses suspended and that there really were consequences,” he explained. “I made sure they went out to the media; I made very clear what happened and what was wrong about it. To me, that’s an ideal state monitoring process.”
California isn’t that hard core and isn’t likely to implement anything more rigorous, at least for the time being. On a logistical level, both the STAR test program and the federal testing program under No Child Left Behind are up for reauthorization in the near future. On a practical level, the chance of the state Department of Education getting more money in the current financial climate is about as likely as the chance that an entire class of students will answer the same questions correctly and get the same questions wrong.
7 Comments
"Darn, I wish I had read that over again before I hit send.” Don’t let this be your lament. To promote a civil dialogue, please be considerate, respectful and mindful of your tone. We encourage you to use your real name, but if you must use a nom de plume, stick with it. Anonymous postings will be removed.
 |
Featured Video |
Did You Know?
Just the Facts
Other Sites to Watch
View More >>
SVEF is the education foundation advocating for the needs of school districts across Silicon Valley. SVEF seeks to improve public education through establishing partnerships with the private sector and the education community ..
The GRE test has been administered for years as an “adaptive” computer test so there is no chance of cheating since the answers are all entered directly into the computer. The scores on the multiple choice section are available immediately on completion leaving only the essay part (typed into the computer) to be graded later. The “adaptive” aspect means that if the test questions are answered correctly they get progressively harder, if they are answered incorrectly they get progressively easier until they begin being answered correctly. I know the immediate response will be “where’s the money for more computers?” The money is here but is spent on class size reduction and gold-plated facilities for some schools, while other schools do without.
All the schools around here have spent hundreds of millions of (bond money) dollars on solar panels which will pay for themselves in 40 years except that they have a 20-year lifespan (so they will never pay for themselves), and on astro-turf athletic fields made up of ground-up tires that will have to be replaced in 10 years and meanwhile are leaching toxic compounds into the drains which go to the bay. Some of that money could have been spent on computer aided instruction and testing.
One of the oldest and most effective computer aided instruction devices is called “Cognitive Tutor” from Carnegie-Mellon (http://en.wikipedia.org/wiki/Cognitive_tutor). The name is generic for adaptive computer teaching systems. It is not simply a computer program you stick kids in front of but must be used by a teacher with students. It has a great success story with poor and minorities and is in use in some districts in Cal. – San Jose Unified is one of them I believe, but I may be out of date. Accelerated Reading and Accelerated Math are two other programs in use in some areas that have helped poor and minorities (with teacher help).
An NEA research document
http://www.nea.org/assets/docs/HE/NEA_Rankings_and_Estimates010711.pdf
has the student teacher ratios for every state in the US. (And many other interesting things – something for everyone!). I ran an EXCEL spreadsheet of this data against NEAP results and found correlation coefficients (r) on the order of -0.25 to -0.17 which means that there is no correlation worth mentioning. The coefficient of determination is r^2 which comes out to something like 0.06 at most which means that 6% of the NAEP improvement is explained by class size reduction – which is meaningless. I have found no meaningful data indicating that class size reduction works except for the first 3 years after which it apparently doesn’t matter at all. A professor pointed that out here some time ago. Maybe someone else has other information? In fact those initial gains from class size reduction quickly dissipate in later years.
So why spend all that money reducing class size if it doesn’t work? The advantage to class size reduction is that districts get to hire more teachers, which means more members dues for the unions, and more administrators to supervise teachers, and more buildings for classrooms for those teachers which means more contracts for the local building unions and construction cos. who can be reliably counted on to contribute to the school board’s re-election campaign.
So if all you can figure out to do when you get more money is hire more teachers then all you can figure out to do when you lose some of that money is get rid of teachers, but they don’t want to do that so they cut something else like catching STAR cheating or the school year or Summer school.
Report this comment for abusive language, hate speech and profanity
Why worry about isolated instances of erasures and other irregularities when cheating by teaching-to-the-test is a widespread and an apparently-sanctioned practice in a vast number of our schools?
The State Board has a wimpy, toothless policy on academic preparation (www.cde.ca.gov/ta/tg/sa/documents/academicprepdec09.pdf). It seems that few take it seriously.
Report this comment for abusive language, hate speech and profanity
Without claiming to be an expert in school finance — far from it — it’s my understanding that bond money can only be spent on facilities — not computer aided instruction or testing. Also, I know there are grants floating around for just that kind of expenditure (and also not usable on computer aided instruction or testing).
This is a small point except that it shows how things that seem simple (stupid school districts, spending money on solar panels instead of instruction) really aren’t.
Report this comment for abusive language, hate speech and profanity
And Eric, I am definitely a non-fan of being forced to teach to a test, but I don’t see how it can be called cheating.
Report this comment for abusive language, hate speech and profanity
Just a short addendum… in terms of money for monitoring, three testing experts I spoke with said the cost ranges from less than 1 percent to about 2 percent of state education testing budgets. When I asked CDE’s John Boivin about that he agreed that it’s not a huge amount of money, but said the big cost is in staff time to investigate the irregularities.
Report this comment for abusive language, hate speech and profanity
In high school, my civics teacher had 5 classes of about 125 students each. He was a very good teacher and every Senior wanted him as their civics teacher. So he was given a lecture hall and two assistants to handle attendance, passing out and collecting tests, helping with audio/visual, etc. If you have a really good teacher, sometimes it is better to have a large class with assistants.
Unfortunately, the teachers for high school ‘general science’ are more often selected for their ability to keep control of the class then for their knowledge of science. There are far more errors in general science books then in books of physics, chemistry or biology that are used by teachers that usually have their Masters in the field they teach.
Report this comment for abusive language, hate speech and profanity
Caroline has a point that bond money must be spent on capital projects, but Michael G. has a point as well. Bonds must still be serviced and repaid out of property tax bills, and districts can choose how to use that precious resource. They can issue bonds, or they can emphasize parcel tax funding, which comes out of property tax bills as well but can be used to cover critical operating expenses. If Michael G. is referring to the Mt. Diablo district, the only district I know of to try to fund capital infrastructure (solar systems) having a 15-year life span using 42-year bonds, then that is a choice, and a very stupid one at that. In that case, it will require taxpayers to spend about $900 million in interest to build capital projects with a construction cost of about $300 million. And of course, property owners will be paying for them for decades after the facilities have become obsolete and been discarded. (There’s a reason why, in the private market, financing for solar projects rarely extends beyond 15 years.)
In 2009, voters in that district had voted 62% for a parcel tax, which of course was insufficient to pass. But that total was reached without ANY cooperation or support from the district administrators or unions. Not one rally, speech, email blast, or letter to the editor from any of the people whose support could have provided totally discretionary new dollars for instruction. The parcel tax campaign was solely parent-driven. Not even the superintendent would publicly endorse it! All the parent organizers got from the district was a big “We told you so” when it failed to pass by 4 points. Imagine if the district administrators and teachers had actually done something — anything — to support getting millions of dollars a year in new discretionary parcel tax funds that could be used right in their classrooms!
Instead, the following year the district heavily promoted a vast solar project financed with 42-year bonds (the longer term critical to keeping the annual costs low emough to pass). There were multiple glossy mailings, rallies, and a PR campaign, all bankrolled by the investment banks, law firms, bond consultants, solar companies, trade unions, and contractors who were salivating over the contracts for the $300 million dollar project. No surprise, it passed the 55% approval hurdle, because it was “for the students”. Now residents are watching, appalled, as their schools’ parking lots and playgrounds are being torn up for the massive solar collector structures. And, of course, construction costs are starting to mushroom. (To “save money”, but perhaps mainly to ensure that contracts get apportioned to the correct recipients, the district is trying to manage the project “internally”, even though the district has absolutely no experience constructing solar facilities, except for one small roof-top “test” solar structure that is currently “non-operational”.)
So this was a financing choice. Now the district will have ample solar structures and questionable energy savings (the calculations included multiple errors noted in the press, including incorrect PG&E rebates and ”savings” at campuses that are now closed). But without the parcel tax revenue that every other adjacent district has, Mt. Diablo is now at serious risk for a state takeover. In fact, if it gets a qualified rating as the issuer, it may not even be able to sell the rest of the solar bonds!! (A blessing in disguise, perhaps.) In the end, the students and teachers will face even more furlough days and layoffs, more elimination of classes, and fewer materials in the classroom. Oh, and one more thing: the local grand jury has apparently now become quite interested in the whole affair.
Report this comment for abusive language, hate speech and profanity