12 Comments

1. 

   Gary Ravani

   December 23, 2011 • 4:33 pm

   Readers, and those whose enthusiasms run toward using student test data for teachers’ evaluations, should note that the questions above from the 8th grade social studies/history CST are all questions based on the 7th grade standards. There are typically several questions on the 6th grade standards thrown in for good measure. The test scores though, “belong” to the 8th grade teacher. And then there are those subject matters for which there are no CST’s. The suggestion, from the legislature no less, is to just pull together some statistically “valid and reliable” test from thin air. Which tells me members of the legislature, as well as most pundits, never took basic statistics courses and have no idea what “valid and reliable” really mean. You can pull together a criterion referenced “test” of some kind locally with agreed scoring rubrics which may be much more informative to teachers than the CSTs, but statistically “valid and reliable?”  That’s a fantasy, but also a good “talking point” for those unfamiliar with assessment (and evaluation). This goes for much education policy made without input from those who do the educating. To oppose those kinds of policy train wrecks you are labeled “afraid of accountability” and/or “defenders of the status quo!” What the opposition to these fantasies are in reality is experienced and knowledgeable.

   Report this comment for abusive language, hate speech and profanity
   

   Reply
2. 

   Ze'ev Wurman

   December 23, 2011 • 11:42 pm

   Gary,
    
   You write that “the test scores though, “belong” to the 8th grade teacher.” I thought they belong to the student, as does the “valid and reliable” moniker. This test results are simply supposed to reflect the cumulative student knowledge in history and social studies in grade 8 (no previous SS test given in K-7)
    
   As far as I am aware, nobody in his right mind suggests to use  this “status” test for teacher evaluation. Everyone agrees that for use in evaluation you need to have at least some kind of value added test, which requires at a minimum a test at consecutive grades. The history test simply cannot be used to hold teachers accountable, and it is not. What have you been smoking?
    
   On a completely different note, the CAHSEE item 3 (stats) is a badly flawed item and somebody at CDE ought to get her (or his) nose rubbed in for allowing it. The prompt says:
    
   ” The Smithburg town library wanted to see what types of books were borrowed most often.”
    
   If the question is about what types of books are borrowed most often, the answer is simply “children’s books at 26%” which can be directly read off the chart. Unfortunately this answer is not present in the list of possible answers. A classic example of a bad item where unnecessary wordiness tripped the writers. I want to believe such item would never pass the Math CST review panel.

   Report this comment for abusive language, hate speech and profanity
   

   Reply
3. 

   Bruce William Smith

   December 24, 2011 • 10:37 am

   Immediately noticeable to me is California’s dependence on multiple choice questions, which is out of sync with PISA, with NAEP, with higher education, and with the world of work. Also interesting in this regard is the fact that Shanghai, recent winners in all categories on the most recent round of PISA testing, and unlike the rest of China, banned all multiple choice testing in 1985. This ban has the benefit of avoiding wasting school time on “test-taking strategies” such as eliminating unlikely answers and strategic guessing, with such time instead devoted to learning how to actually solve the problems or to write answers in sentences, since these are the only remaining options for getting credit for the problems. America needs to move beyond its current testing establishment if we want our students to develop the real intelligences necessary to be internationally competitive both on academic tests and in the world of work. Our belief in our exceptionalism, which has usually badly masked an incuriosity and sheer ignorance about what goes on beyond our shores, has held back our students for too long.

   Report this comment for abusive language, hate speech and profanity
   

   Reply
4. 

   Stephanie

   December 25, 2011 • 11:42 am

   Ze’ev,
   Several months ago Micheal Winerip of the New York Times wrote about the teacher evaluation system being rolled out  in Tennessee as a result of “race to the top”.  As I recall,  some teachers are being evaluated based on test scores from subjects they don’t teach.  The scores may be from a value added test from consecutive years of testing in a subject, but it certainly doesn’t seem to be a valid use of the scores.

   Report this comment for abusive language, hate speech and profanity
   

   Reply
5. 

   Ze'ev Wurman

   December 25, 2011 • 2:35 pm

   Stephanie,
    
   I didn’t follow that discussion but if Winerip is right about his facts, then this is a badly done evaluation system and will surely be challenged in courts. And lose. But I wouldn’t blindly take whatever people write on the NYT (and others opinion pages as facts. So do your own checking if you want to use his anecdote and be taken seriously.
    
   As to Bruce’s comment, well done both multiple choice and constructed response items tend to assess identical skills (with some limited exceptions). There has been a lot of research done on this over decades and there is little disagreement on that fact among testing experts. Note the testing experts, not educators or politicians, who think they “know better” than the experts. TIMSS is mostly multiple choice, as are the current Mass. tests (MCAS) and Massachusetts leads the nation in educational achievement. But one needs good and well posed items, and to select those one needs serious content area expertise not just testing expertise, and nationally many of our multiple tests severely lack such expertise. NAEP has a huge number of flawed items (25-35%) by various counts, and PISA has little connection to school curriculum. In fact, PISA explicitly says that success on PISA depends to a large degree on skills acquired outside school — at home, listening to news, etc. Would you like our students to be assessed at school on skills acquired at home and in the neighborhood? Can you imagine the correlation to SES in that case? Oh, and speaking of PISA, what do you think the Finns themselves think of their students who achieved so highly on PISA that the whole world beats the path to their door? Here is the opinion of a group of Finnish university mathematicians:
    
   “The results of the PISA survey … have brought about satisfaction and pride in Finland. Newspapers and media have advertised that Finnish compulsory school leavers are
   top experts in mathematics. However, mathematics teachers in universities and polytechnics are worried, as in fact the mathematical knowledge of new students has declined dramatically
   
   … This conflict can be explained by pointing out that the PISA survey measured only everyday mathematical knowledge … the kind of mathematics which is needed in high-school or vocational studies was not part of the survey.
   … The PISA survey provides us with useful information regarding the mathematical literacy needed in everyday life and the ability to solve simple problems. These skills are simply not enough in a world which uses and utilizes mathematics more and more.”
    
   We will have a real opportunity in the next few years to see how high ambitions for large-scale “intelligent” testing work out by the PARCC & SBAC assessment consortia. In all probability they will blow through half a billion federal tax dollars by 2015-6, and leave us with a mediocre test that measures nothing meaningful. Kind of like CLAS did in California in 1993-94, for those who remember that disaster.

   Report this comment for abusive language, hate speech and profanity
   

   Reply
6. 

   Cal

   December 26, 2011 • 8:12 am

   The CBEST is a formality. While all teachers do have to pass it, that’s merely an artifact of an older system.
    
   Elementary school teachers have to pass the Multiple Subjects CSET, which is a much harder test (each subject at about a 10th grade honors level) and middle/high school teachers have to pass the Single Subjects CSET, with a difficulty roughly on par with an AP test.
   NAEP is a worthless test, and I’m really tired of people holding it up as the “gold standard”.

   Report this comment for abusive language, hate speech and profanity
   

   Reply
7. 

   Gary Ravani

   December 29, 2011 • 3:17 pm

   Ze’ev;

   You wrote: “ As far as I am aware, nobody in his right mind suggests to use this “status” test for teacher evaluation. ”

   For once, we are pretty much on the same page; however, you are wrong about students “owning’” the test scores. There is no, and should not be, any kind of accountability for students based on their test scores. In HS those students going on to 4 year colleges can gain some advantage by doing well on the tests, but for the rest of the kids they can, and do, just bubble in the answer sheets with creative designs.

   As to “nobody in his right mind;” that’s exactily the point I’ve been trying to make about the testing fetish all along. There are all kinds of people out of their minds with the promise of  nailing a few teachers on something or anything. Just “making up tests” that are statistically “valid and reliable” for subjects currently without state tests have that mind-set.

   And there are people who think the 8th grade Social Studies “status” tests should be used in teacher evaluations. That it makes no sense makes no diference at all.

   Report this comment for abusive language, hate speech and profanity
   

   Reply
8. 

   Ze'ev Wurman

   December 30, 2011 • 12:23 am

   Gary,
    
   If you were trying to make some point other than that somebody was actually considering using the social studies 8th grade test for teacher evaluation, you were singularly unsuccessful — that is what clearly came through your original comment. If you are aware of such “somebody,” please let us know. And if it was just your vivid imagination …

   Report this comment for abusive language, hate speech and profanity
   

   Reply
9. 

   navigio

   December 30, 2011 • 9:10 am

   hate to be pedantic, but isnt the answer to the first one misleading? or is math convention nowadays that you get to choose whether a multiplier is part of the numerator or part of the denominator? if not, how does one write ‘one over seven x’?

   Report this comment for abusive language, hate speech and profanity
   

   Reply
10. 

    Ze'ev Wurman

    December 31, 2011 • 2:06 am

    Navigio,
     
    Excellent point. (1/7)*x would be better. x/7 would be much better. Using horizontal bar for 1/7 with x centered vertically on the bar would work. In general, if order of operations is equal there is a convention of processing left to right. So 1:7*x would be “kind of” OK, but it is ugly in the sense it relies on a confusing and not very important convention. The absence of an explicit operator between the 7 and the x just adds to this confusion. Some programming languages will process left-to-right in such case, some right-to-left. That is what parentheses are for — to avoid confusion. Again, CAHSEE seems not a high quality test.

    Report this comment for abusive language, hate speech and profanity
    

    Reply
11. 

    Paul

    January 3, 2012 • 4:35 pm

    Gary Ravani wrote, “…you are wrong about students ‘owning’ the test scores. There is no, and should not be, any kind of accountability for students based on their test scores.”
     
    Heretical question: Why not? What would happen if we took grading and promotion away from teachers (in subjects with CSTs), sped up the CST scoring process, and provided CST results instead of teacher-determined grades, on report cards?
     
    The CSTs aren’t perfect (though the criticisms here are nitpicking — the “1/7″ is rendered in HTML and was probably typeset properly on the paper test, and the intent of the book frequency item is to have students reason/make comparisons instead of simply reading a number from a graph), and they are not what I’d want for my own children, but the tests correspond [B][I]exactly[/I][/B] to taxpayers’ “back-to-basics” expectations and, in math, at least, to state policy. It is a matter of state policy that [I]not[/I] relying on a “timed multiple-choice test” represents a “serious misunderstanding of what mathematics is and what it means to understand mathematical concepts” (Mathematics Framework, pp. 224-225)!
     
    People who are interested in assessments that are authentic/carry intrinsic meaning and allow for different demonstrations of competency/can be accessed at multiple levels should look at MARS, offered in California by the Silicon Valley Mathematics Initiative. That’s what assessment and evaluation should be (but not, alas, what taxpayers and decision-makers say they want).

    Report this comment for abusive language, hate speech and profanity
    

    Reply

Trackbacks

1. Final exams : SCOE News Reader