Times’ ratings:solution or problem?
Divergence of views at UC Berkeley forum
Using students’ test results to evaluate teachers is a “solution in search of a problem” that diverts attention from larger inequities facing America’s poor children. It’s part of “a war on America’s teachers.”
To the contrary, it’s an effective way to wean out the least effective teachers and reward the best. And it beats the other forms of ineffective and subjective teacher evaluations that led to not one dismissal, out of 45,000 teachers in Los Angeles Unified, for reasons of poor teaching.
The views were emphatic, with little common ground, Monday at a forum sponsored by UC Berkeley’s Graduate School of Education on issues related to the Los Angeles Times’ controversial decision to publish the names and effectiveness ratings of 6,000 elementary school teachers in Los Angeles, based on their students’ scores on annual standardized tests in math and reading.
In defending the Times’ decision to publish, Jason Felch, one of three Times staffers who wrote the series Grading the Teachers, faced a generally tough and critical audience of teachers, grad school faculty, and fellow panelists. However, two researchers who spoke at least agreed that the technique the Times used to rate teachers could provide useful information, as long as it didn’t become the exclusive or predominant factor in judging teachers.
That technique, value-added analysis, is being promoted by the Obama administration in Race to the Top and in other federal competitive grants as a primary factor for evaluating teachers. It purports to measure a teacher’s impact on raising students’ test scores by controlling for other factors, including student demographics.
But it’s an imperfect tool, prone to imprecision and inconsistency, for reasons that two UC Berkeley Graduate School of Education professors with expertise in statistical analysis, Sophia Rabe-Hesketh and Mark Wilson, explained. It can’t control for all of the variables – student characteristics, parent education, effect of peers and disruptive students in a class, the impact on teachers of a school’s principal and resources. And the multiple-choice tests don’t measure important standards, such as an ability to problem solve or write coherent paragraphs. As a result, the results don’t measure teachers’ effectiveness, only their impact on raising test scores by narrow measures.
Compared with what?
Eric Hanushek, senior fellow at Stanford’s Hoover Institution, agreed that standardized tests should be improved and that value-added results must be used carefully. But despite its imperfections, it stands up well to the current system of subjective, perfunctory evaluations of teachers by principals, he said. What it can do is distinguish the best and the worst teachers, while acknowledging that there is uncertainty in the broad middle range of teachers. “If we can change the system to ensure that children are not taught by the very bottom, we would make enormous strides” to improve the outcomes for individual kids and affect the nation as a whole, he said. If we are interested in student achievement, then there is no substitute for paying attention to actual measurements of achievement.
While also warning against holding value-added data to a “standard of perfection,” Felch said that in every story, the Times made clear to readers that value-added test scores should not be the sole factor in evaluating teachers. The newspaper concluded that parents should have a right to the information, because thousands of children in Los Angeles were being assigned to ineffective teachers. The district has had value-added data for years, but done nothing with it. And the union remained deadlocked with the district over its use – another factor that led to the decision to publish.
But “nobody pays attention to the caveats,” retorted Richard Rothstein, research associate at the Economic Policy Institute in Washington and long-time critic of standardized testing. “We know well that most people will focus on value-added scores. … and this will feed the obsession with test scores while creating disincentives to pay attention to areas not covered by tests, like history, science and character development.”
Felch said that he had confidence that parents would put the value-added rating in perspective. Contrary to predictions that parents would flood their schools with requests to assign their children to teachers with the highest ratings, this has not happened, he said.
But on Monday came the news that Rigoberto Ruelas, 39, a popular and dedicated fifth grade teacher in an impoverished area of Los Angeles, committed suicide. Colleagues and relatives said that Ruelas was despondent because he was rated as “average” in raising his students to proficiency in English, and “less effective” in math, for a slightly less effective overall rating. United Teachers Los Angeles called for the Times to take down the data base.
Panelist Anthony Cody called Ruelas “the first casualty in America’s war on teachers.” Cody, a National Board certified science teacher from Oakland and writer of the blog “Living in Dialogue” in Teacher Magazine, said that the impact of teacher tenure has been exaggerated, and the claim that teachers don’t care about the quality of instruction of other teachers is false.
“Teacher evaluation is a red herring to divert attention from poverty and inequality that mar our society,” Cody said.
Gaming the system
Kyla Johnson-Trammell, principal of Sequoia Elementary School in Oakland, acknowledged that the current evaluation system is “inherently flawed,” because of its subjectivity and, in some cases, a principal’s limited knowledge of instruction. But using value-added scores won’t be an improvement, because standardized tests are flawed and subject to gaming by administrators.
She cited, as an example, the second grade California standardized math test, which doesn’t test for pre-algebra skills, even though those are critical to passing Algebra I later on. “So, as a principal, do I decide that even though kids need pre-algebraic skills, not to teach that and instead focus on superficial facts?” Her solution, she said, is a plan for test scores to rise but also a garden program and an arts integration program that are not tied to test scores.
United Teachers Los Angeles agreed to open negotiations on creating a new teacher evaluation system – something it had resisted until the Times published the teacher ratings. It may be a strategy to shut off further publishing of value-added ratings by formally incorporating them into the teacher evaluation process, thereby making the information confidential.
Foot-dragging carries a cost. David Plank, executive director of the non-partisan education research institute Policy Analysis for California Education (PACE), said that in states outside California, districts are working in partnership with unions on innovative approaches to teacher and administrator evaluations. California, he said, is falling behind the national conversation.
20 Comments
"Darn, I wish I had read that over again before I hit send.” Don’t let this be your lament. To promote a civil dialogue, please be considerate, respectful and mindful of your tone. We encourage you to use your real name, but if you must use a nom de plume, stick with it. Anonymous postings will be removed.
 |
Featured Video |
Did You Know?
Just the Facts
Other Sites to Watch
View More >>
SVEF is the education foundation advocating for the needs of school districts across Silicon Valley. SVEF seeks to improve public education through establishing partnerships with the private sector and the education community ..
Getting rid of ineffective teachers is only half of a solution. The other half of a solution is hiring effective teachers. So it remains to be seen how the second half of the solution can be accomplished. We didn’t end up where we are for no reason at all.
Report this comment for abusive language, hate speech and profanity
I had to leave after the second panel, but on the two I heard, I felt that there was almost entirely common ground. All the panelists (except of course Felch) were united in agreeing that value-added is a seriously flawed measure of teacher quality. All agreed that publishing the teachers’ names was wrong, except of course for Felch and for Eric Hanushek of the Hoover Institution – and Hanushek damned it with surprisingly faint praise. Hanushek said that value-added should be used only to identify the “very bottom” teachers, but also agreed that administrators would already know who the “very bottom” teachers were. He didn’t really state whether the names of the “very bottom” teachers should be published, while everyone else agreed that teachers’ names should not be published – and that included Susan Rasky of the UC-Berkeley Grad School of Journalism, Jason Felch’s former professor.
Rabe-Hesketh’s and Mark Wilson’s views were much strong than merely saying value-added is an “imperfect tool.” Both were emphatic (in their mild wonkish way) in stating that it absolutely should not be used.
I thought the most important part of Cody’s talk was when he described being part of the PAR process (in Oakland Unified), which works with struggling/problem teachers and sometimes counsels them out of the teaching profession. That rebutted Felch’s indication that there’s no process to deal with problem teachers.
Cody also rebutted the widespread inaccurate claims that teachers automatically get lifetime job security after two years, as repeatedly and falsely stated by Davis “Waiting for Superman” Guggenheim, including on Oprah’s show (and she picked up and amplified the misstatement, reportedly).
More revealing was when Felch turned to the mild, ultra-wonkish Rothstein and accused him of being content with the current state of public education and “defending the status quo.” This is a line from the education-reformer playbook – a scripted response when you have no actual facts to rebut someone’s point – pure PR, not a sincere comment under any circumstances. What is a supposedly impartial journalist doing parroting a scripted propaganda line? Unlike in the public arena (and apparently in the newsroom), Felch’s comment drew cries of disagreement from everyone at the panelist table and all over the room, and he actually retracted it.
Rasky did complain at one point that those objecting to the Times project were shooting the messenger. My personal response would be that the Times has gone beyond being the messenger in this case to positioning itself as the judge, which is not the proper role of the press. She also made a mild objection to a speaker (Cody, I think) lumping the Times in with Oprah and MSNBC, a comment that drew no response, though the obvious “if the shoe fits…” retort hung there in the room unsaid.
The heart of the story here is that every statistician/wonk type on the panels agreed that value-added is a badly flawed way to measure teacher effectiveness, and the only one who didn’t view it as too fatally flawed to use at all is Hanushek – and he represents an organization that promotes the privatization and anti-teacher-union viewpoints and thus whose agenda would be to support the Times project. And again, even he presented a tepid defense.
Felch said there was much debate among Times editors and reporters about printing the teachers’ names. I have to wonder if besides just discussing it among themselves they consulted with statisticians or other experts – beyond the one they hired, who of course had financial motivation for agreeing to take on the project – about the validity of value-added as a gauge of teacher quality. It seems unimaginable that they could have heard those talks by Wilson and Rabe-Hesketh (I won’t even mention Rothstein, as he is a nationally known critic of “education reform”) and decided it was valid to put resources into the project.
It’s also interesting that the Times intends to continue updating the teacher database year by year – in perpetuity? It’s an intriguing view of the future of journalism.
Felch repeatedly said that currently, teacher evaluation is based on “drive-bys” — a tiny amount of time spent on rare occasions by principals observing in classrooms. I’m not sure that it’s true that competent principals have little idea of what’s going on in their schools’ classrooms. But I would note that every urban principal I know is chronically overwhelmed with a contant barrage of work and demands, and that school site administrations are too stretched too thinly to do much more observation. Of course, in a time of chronic budget shortages, taking more resources from the classroom to shift them to site administration is problematic, to say the least. But Felch’s observation does raise the question of whether sufficient funding would allow for more thorough teacher evaluation by their administrators.
I’ll conclude by describing my son’s high school French teacher. The class mainly spent its time watching subtitled movies – there were a few worksheets, not much more work. The teacher would leave the class and come back smelling of pot. (The students were generally fine with this whole program!) There was a parade of parents complaining to the principal, and some parents made arrangements for their kids to take private French classes, at their expense but for school credit. The principal told them he sympathized, but he couldn’t fire the teacher. The implication was that the teacher was classified as disabled and that that made it even harder to get rid of him. Then the principal left for a new job, a new principal arrived, and the French teacher was just quietly gone. I don’t know; that tells me a lot about the “impossibility” of getting rid of bad teachers.
(I hope that if you respond to my comments here, you will omit personal comments about me, John. That’s really an unproductive waste of everyone’s time.)
Report this comment for abusive language, hate speech and profanity
Caroline: This is actually my second try. I deleted the first accidentally; even I have can trouble posting on my blog. I respond when I can.
As edfundwonk observed, no one responsible is calling for the exclusive use of value-added analysis . And everyone on the panel acknowledged the flaws. But you must have zoned out when Mark Wilson, even after his extensive critique, acknowledged the information from valued-added scores would be useful as one factor in teachers’ evaluations. And Eric Hanushek told me afterward that he wasn’t troubled by printing the names and scores of all of the teachers; there would have been more uproar, had the newspaper chosen only to publish the worst and the best.
The point that Jason Felch, Hanushek and David Plank continued to return to was, Compared with what? The current system of perfunctory evaluations by principals isn’t working; principals’ evaluations are even more subjective and prone to errors, as the Oakland principal pointed out. Value-added analysis, for all its flaws, can serve as a check on the subjectivity and bring to attention what’s been overlooked, especially for outstanding teachers.
Yes, Hanushek acknowledged that principals probably know who are their worst performing teachers — a comment that evoked some sneers from the audience — but the point is what are principals doing with that knowledge? Apparently not much in many cases. Value-added info can give them hard data they may need to support taking action, which can often be difficult and expensive in the jungle of due process rights that teachers have won.
I’m no more enamored with value-added analysis than many of the other panelists, but disagreed with your characterization of Hanushek’s defense as tepid. (Maybe if you hadn’t gone in with your mind made up, you would have seen it differently. This is not a personal attack, just a helpful observation.)
Anthony Cody and I have talked about Peer Assistance and Review (PAR), and I don’t doubt he has counseled teachers out of the profession. But you need effective evaluations to get to that point. In June, I wrote about the recommendations for a new evaluation system proposed by Accomplished California Teachers, of which Cody is a member. Even that report does not dismiss the use of standardized test results, although they would play a minimal role. I hope that the report gains traction. As you correctly observed, with cuts in school administration, principals are struggling to do teacher reviews, let alone extensive ones. As David Plank pointed out, evaluations by portfolio, as done for a National Board certification, with video observations, lesson plans, and student surveys, demand a huge time commitment. I would think this opens an opportunity for teachers to play a new role in creating an effective evaluation system, in which peer observations and analysis would be a major component. Instead of fighting a change in the system, I would think teacher unions should embrace it.
I thought it was poor judgment to publish the scores of all of the teachers for the reasons cited in this and past posts. And I could be wrong, but I doubt the Times will expend a lot of resources repeating this in future years. The paper won’t be able to get the value-added data and perhaps the raw data on individual teachers, if the union agrees to incorporate the data into an evaluation system. A court will have to decide. But, as Jason Felch said, there’s a lot of useful data at a school level that the district hasn’t used or publicized. I hope the Times pursues that avenue in the future.
Report this comment for abusive language, hate speech and profanity
In all this talk on evaluations, I don’t see any mention of charter schools. If public value added evaluations have any promise, why not experiment first with charter schools ? For one, there is less red tape to get things started.
Report this comment for abusive language, hate speech and profanity
In all the talk yesterday, I don’t remember any of the panelists mentioning upper grade student attention to test taking as an issue of teenage achievement accuracy. Students know which tests personally count for them and which don’t.
Report this comment for abusive language, hate speech and profanity
CarolineSF,
If John won’t respond to your comments, allow me to do so. First, I am aware of no one who has ever proposed that value-added analysis be used as the sole measure of a teacher’s effectiveness. To the contrary, all of the proponents whom I know of have clearly stated that value-added analysis should be but one among many measures, including direct observation. Second, the benefit of value-added analysis is that it eliminates the problems of having to control for statistically estimated variables (e.g. race, parental income) that marginally (but usually not statistically significantly) have impacts on the classroom average test score. This method uses regression analysis (an imperfect measure itself) rather that value-added analysis (another imperfect measure).
The strength of value-added analysis is that you don’t have to control for all of these background variables, because Johnny and Janey at the beginning of the school year are the same Johnny and Janey at the end of the school year — in other words, differences are automatically controlled for, because there are no differences. Moreover, with value-added analysis, increases in average classroom achievement are measured as the average of these individual differences, rather than changes in which the unit of analysis is the entire classroom. Value-added analysis can answer the question ,”How many grade levels did Johnny (a single student) gain for a year of instruction?” A classroom-based analysis cannot. Admittedly, value-added analysis has problems of its own. The question is, is as a measure of teacher effectiveness, is it legitimate to include in an overall evaluation process? I believe that it is.
I can sympathize with the teachers whose value-added scores are published. (I’m a state employee and the Sacramento Bee published all of our salaries — by name.) On the other hand, this is public information; It is not protected by the state and federal Freedom of Information Acts. All that the Times is doing is publicizing the same information that school districts are required to provide to anyone who requests it. Does the Times have an ethical obligation to disclose all of the caveats applicable to these measures? Probably. does it have a legal obligation to do so? Probably not.
Let me now turn to your arguments in order. Regarding Rabe-Hesketh and Mark Wilson’s views, I have stipulated that value-added analyis is imperfect and should be just one factor of several that are used to evaluate teacher performance.
I only know PAR’s program model and its goals and thus am unqualified to comment on its effectiveness.
I adamantly disagree with your assertion that the”[c]laims that teachers automatically get job security after two years” is widespread and inaccurate. Call it what you will (completion of “probationary period”), once a teacher is permanent, it is exceedingly difficult to dismiss such a teacher for poor job performance. Look at the miniscule number of teachers who get dismissed from LAUSD annually for poor performance. Do you think this handful truly represents all of the ineffective teachers in LAUSD? The real reason why so few teachers are dismissed for poor performance is that it virtually takes an Act of God to do so. The reams of evidence that must be accumulated over many years just to bring a case to an administrative law judge is ridiculous. Most districts can’t afford to devote their time to meeting such onerous requirements and therefore focus their efforts on only the most egregious offenses, e.g. sexual abuse.
By the way, I’m always suspicious when a debater must resort to ad hominem attacks (Rabe-Hesketh’s and Mark Wilson’s “mild wonkish ways” and “mild, ultra-wonkish Rothstein”) or using negative caricatures (”education-reformer playbook,” “scripted response,” “pure PR,” “scripted propaganda”) This is usually a sure sign that the author has run out of factual arguments, sort of like “playing the race card.”
As to the accusation, I agree wholeheartedly with the statement that the CTA’s raison d’etre has been to protect the status quo or, in those years when the district administration is unable to offer money, to increase its power and influence.
I find it interesting that you agree with Felch’s statement that teacher observation is based on “drive-bys.” In other words, the current system is inadequate. I agree. Something needs to be done. Your solution is more money. I’m sorry to inform you, CarolineSF, but if money were the answer, the schools of Washington, DC would be the highest-performing in the nation. And, in case you haven’t noticed, we’re in the worst recession since the Great Depression. The answer isn’t more money; it’s a question of priorities. If you think too little is being spent on principals, negotiate the issue. Collective bargaining wasn’t created to give unions all of the benefits and none of the responsibilities. You’re not going to get any more money. You’re both in this together. If you want to add vice principals, work with the administration to identify what should be cut. This is collective bargaining.
Your experience with your child ’s French teacher demonstrates more eloquently than could I (1)the need for more information, rather than less, in the evaluation process and (2) the inability of principals to get rid of underperforming teachers. First, if the school district had adopted, as LAUSD has proposed, the inclusion of value-added measures as one factor in a teacher’s evaluation, this would by just another piece of objective (albeit imperfect) information to include in the teacher’s evaluation. And given the way he conducted his class, I have no doubt that the value-added analysis would confirm his incompetence.
Second, you state, “[t]he principal told them he sympathized, but he couldn’t fire the teacher.” I think you just made my point. It is a tragedy that these students had to endure such a teacher for an entire year. These students’ opportunity to learn French in fourth grade is gone forever. I know very few, if any, schools that have departmentalized fourth and fifth grades. If this is the case, then not only will these students have missed out on fourth-grade French, they would be expected to study French at the fifth-grade level the next year. In other words, they either spend their summer making up fourth-grade French or begin fifth grade one year behind.
In the private sector, this person would have been given notice and if he didn’t shape up by a date certain, he’d be gone. Instead, he remains employed until the end of the school year when, in all likelihood, he’s not gotten rid of; he’s transferred to another school. Don’t tell me you haven’t heard of the “dance of the lemons” Even we dummies in Sacramento know about that one.
If nothing else, this debate has had the salutary effect of bringing UTLA to the bargaining table to seriously discuss the inclusion of performance-based measures in teacher evaluations. I truly believe that this has the potential to increase the quality of education for all LAUSD students — and that’s what it’s about.
Report this comment for abusive language, hate speech and profanity
Edfundwonk, this is just an exchange of opinions rather than a debate where one viewpoint must prevail, I’m just addressing a few of your points.
While my own opinions obviously strongly color my account of yesterday’s forum at UC-Berkeley, what I was doing was reporting on the presentations. Three presenters whom I heard said that value-added should not be used at all in the manner in which the Los Angeles Times used it. They said that flatly and forcefully. They also said that it IS necessarily to control for background variables, and that that is impossible to do accurately.
So your argument is with Wilson, Rabe-Hesketh and Rothstein, not with me.
It is Anthony Cody’s assertions, as described by me, with which you’re arguing, in regards to the PAR program and tenure.
You are commenting on me when you characterize my description of “wonkish ways” as an ad hominem attack. You’re missing a point. Since I clearly agree with the comments made by Rabe-Hesketh, Wilson and Rothstein, so why would I be launching ad hominem attacks against them? I don’t view the word “wonk” as a negative at all. Whatever it is you’re “suspicious” about, I don’t think your suspicion is warranted. My point in characterizing them as wonkish was to emphasize that their reports were scholarly, not heated or partisan. (The only speakers whose views I was inclined to disagree with were Jason Felch and Eric Hanushek.)
I see that you disagree with my characterization of Felch’s comment. Sorry you dislike the way I framed it. Regardless of the manner in which I framed my description, Felch retracted the comment (in what I interpret as embarrassment) after vocalizations indicating disagreement and objection from all the other panelists and from around the room.
As a parent volunteer, I’m not versed enough in what goes on in schools to agree in an informed way with Felch’s statement that teacher observation is based on “drive-bys.” What I was doing was reporting and commenting on his statement. I also observed that our administrators are stretched thin due to lack of funds, and yes, I do think our schools need more money. Clearly we disagree. The comment that D.C.’s schools do poorly with a high per-student expenditure uses one example to attempt to disprove that. I would respond that D.C. students are overwhelmingly high-poverty and that the school system also seems to have been particularly afflicted with decades of dysfunction. If more money weren’t essential to improving school quality, private schools wouldn’t spend vastly more per student than public despite enrolling students with a fraction of the needs, and high-functioning schools in wealthy suburbs would aggressively solicit parent donations and do further large-scale fundraising to increase their available resources.
What my experience with my child’s French teacher illustrates is that an effective manager was immediately able to move him out of the school, while a passive manager did not do so. That’s my point in relating the anecdote. As he’s a high school French teacher, it would not be possible to rate him based on value-added measures. (It wasn’t fourth grade; it was high school.)
Obviously we have differing viewpoints on education policy in general. My comments were my observations of yesterday’s UC-Berkeley event in particular.
I didn’t urge John Fensterwald not to respond to me; I hope he does. I was just voicing my preference that he not pick on me personally.
Report this comment for abusive language, hate speech and profanity
Sorry; I meant that “If more money weren’t essential to improving school quality … would NOT aggressively solicit parent donations and do further large-scale fundraising to increase their available resources.” My apologies for all typos.
Report this comment for abusive language, hate speech and profanity
The reality is that the presntations of the two testing experts condemned VAM. One described VAM (as I recall) as an attempt at regression analysis and regression analysis requires that you control for variables. She went on to list around half a dozen critical variables that you (or they) can’t control for. She said very specifically that VAM should not, in the LA Times sense, “blame and shame teachers.” VAM is a bad methodology in scientific terms.
The two experts were more compromising when sitting at the table. No doubt in the interest of commity. That’s the problem with reasonable people, they will do that and leave the discussion open to people like Hanushek with his vague references to “studies.” That is, not peer reviewed academic work released in professional journals or prensented at professional seminars, but proaganda hit pieces released by right-wing think tanks at press conferences.
Basically, aftere the two testing/statistical experts sat down, the panels went on as if they had never spoken.
Rothstein did what he does, and does well. He refuted the entire “schools in crisis” myth, without doing much to confront the Times. He noted that NAEP scores for all subgroups had increased dramatically over the course of twenty years. Though, of course, they flattened after the imposition of NCLB. The “achievement gap had not closed because, though minorites were higher achieveing, so were middle-class white kids. Why wouldn’t they?
Hanushek saved his most appalling moment for the end when he brought up Finland and its international test scores. When it was pointed out that Finland had a tiny percentage of kids living in poverty compared to the US, he said we had to just ignore that. Schools were the only “policy lever” we had to work with. Right. Universal healthcare, extended paid parental child-care leave, and seamless social servies all out of the reach of the richest nation in the history of mankind. Finland can do it, but we can’t! Don’t even think of it. We want those Finnish test scores, but forget that pesky socialism! Hanushek is a credit to his institute. Herbert Hoover couldn’t have said it better himself.
Report this comment for abusive language, hate speech and profanity
Thank you, John…
I think you’re misreading what evoked sneers from the audience in this comment: “Hanushek acknowledged that principals probably know who are their worst performing teachers — a comment that evoked some sneers from the audience…”
It was the question that evoked chuckles (or perhaps they were sneers) from the audience. I know that for certain, because it was my question, so I was acutely aware that there were chuckles around the room as soon as Louis Freedberg read it and before Hanushek responded — and Hanushek answered in a spirit of some amusement. I don’t think the audience was disagreeing that principals would know who their worst performing teachers are.
When you say that if I hadn’t gone in with my mind made up I might have seen Hanushek’s defense as other than tepid, that could be true, but on the other hand, the same could be said about you, or anyone else with a strong opinion (which is to say anyone who is already informed and has a pulse — certainly anyone who was willing to crowd into that small room on a 95-degree day). I was surprised, though — I certainly didn’t expect to agree with him, since I’m very familiar with the Hoover Institution (I did a large freelance writing job for them in the ’90s), but I expected his defense to be more vigorous.
Report this comment for abusive language, hate speech and profanity
That was an interesting question, Caroline. “Sneer” was probably the wrong word. I read the, “well, duh” reaction as teachers and principals dismissing the idea that think-tank researchers would be telling them what is obvious to those who work daily in schools. Perhaps so, but my point was that the data on performance can help build a case for action — not necessarily punitive — and shouldn’t be readily discounted.
Report this comment for abusive language, hate speech and profanity
How much “value” is enough value?
If you have a class of 30 4th graders, and half of them are:
1. sleeping on the kitchen floor (16 people in a 2 bedroom apartment - hard to do homework) , or
2. being molested/raped by a family member/neighbor, or
3. witnessing dad beat mom and all the emotional trauma that includes, or
4. up until midnight roaming the streets with older siblings with baseball bats getting into fights, or
5. watched an uncle murdered at point blank range in gang-related violence, or
6. witness daily drug/alcohol abuse in the family, or
7. come home to an empty house, sleep, get up and go to school with no one at home (parent working two jobs to survive) and thus have NO parenting…
…and you ‘mysteriously’ find that your 4th graders are not focused, their behavior is disrupting the whole class, and their academic achievement is not on pace with the state average, who is responsible?
The teacher?? Why not? Who else can we blame it on? Problem solved!
No wonder 50% of new teachers don’t stay in the profession. Those of us who do stay, out of the belief that a good education is the ONLY thing that will turn life around for some of these innocent kids, cannot do so without any kind of public support.
Report this comment for abusive language, hate speech and profanity
Not having been able to be there, I appreciate hearing all of this.
I know if I was in LA Unified now, I’d be insisting on a move to 1st grade or 6th or above lightning quick, given that 3rd, 4th and 5th grade teachers are the ones being personally skewered in the media. That is, if I didn’t leave the profession altogether. I agree that publishing the data makes the LA Times the judge.
And I thought the LA Times said they hired someone to come up with these calculations… How can they say that LAUSD had this info for years and never did anything with it? Unless LAUSD administration actually did this very same set of calculations, having year to year test data for each teacher’s students is NOT AT ALL the same thing as this particular algorithm and it’s results, as far as it’s useful to an individual teacher, a teacher team, a leadership team, a school or beyond. So unless someone can prove LAUSD had this same data before, and not just the 2 weeks they gave teachers to “review it and make comments,” I think it’s an unfair accusation of Felch and the LA Times.
Despite the discussion that this VAM analysis controlled for all variables, the teacher’s comments printed in the paper are proof that the data is still flawed, as it didn’t address team teaching arrangements and language arts block coaches that had math scores attached to them. That is perhaps a failure of LAUSD to give the LA Times those details, but there’s several possible, reasonable reasons for that.
Report this comment for abusive language, hate speech and profanity
Data on performance can help build a case, but the data can be made available to principals without being published in one of the nation’s largest (albeit shrinking like all of them) newspapers, in a project with the obvious outcome of publicly blaming, shaming and vilifying teachers.
Similarly, there’s a lot of distance between teachers’ getting nothing but brief and rare drive-by evaluations and the mass public dissemination of their names and ratings.
As I say, the Times went beyond being the messenger to being the judge. That’s outside the role of a newspaper.
Report this comment for abusive language, hate speech and profanity
I don’t disagree with you about publishing the scores and names of 6,000 teachers, Caroline.
But there were equal numbers of unheralded teachers who raised scores significantly and were praised in publishing the names. You focused only on those who were “vilified.”
On the larger point: The newspaper’s job is to inform. The Times info showed that year-in, year-out, some students regressed in some teachers’ classes. Real harm was probably done to those children and their potential to succeed in school. Put aside the issue of the flaws in the data, which we have discussed at length. For those at the extreme end of the scale, the data was probably true in indicating ineffective teachers. The district had the data and did nothing for years — at least that is what Felch and the LA Times assert. The teachers union and the district had been deadlocked over creating new, effective evaluations for teachers. Should a newspaper not publish what it found in some form? What would have been a better way?
Report this comment for abusive language, hate speech and profanity
Let’s hope the conversation on teacher evaluation and “effectiveness” has begun in earnest!
Report this comment for abusive language, hate speech and profanity
It wouldn’t be so hard to dismiss a teacher if principals were trained in how to do it. Look to fixing administrator preparation programs if you think this is an important skill for a principal to have.
One must also remember that most teacher dismissals occur without a challenge the employer, just like in the private sector. These unchallenged dismissals are not captured in any data set. A hearing is conducted only in cases where the employee feels that the dismissal was done for improper reasons as described by law. Among the reasons for firing a teacher: immoral or unprofessional conduct, dishonesty, incompetency, evident unfitness for service. You have to ask why it is that some districts regularly dismiss teachers without any trouble, while some blame due process rights as an excuse for not acting.
Also, remember that the Commission on Teacher Credentialing (CTC) exercises its strict authority over removing teachers from the profession if they are involved in crimes including sex offenses and controlled substance offenses. They regularly suspend and revoke credentials, making it impossible for the teacher to teach, even before the teacher is found guilty and sometimes even if a teacher is exonerated. This is an important augmentation to due process law that should be recognized.
Report this comment for abusive language, hate speech and profanity
Well, truth and accuracy suffer as much when some teachers are inappropriately overpraised as effective based on an unsound gauge as when other teachers are inappropriately disparaged as ineffective. But other than that, it’s evident to me that the harm is done mostly to the teachers at the “ineffective” end of the scale.
It also concerns me that publishing long lists of names of teachers designated “ineffective” (whether or not there are “we don’t really necessarily mean they’re ineffective even though that’s what it says on the graphic” disclaimers scattered hither and yon) contributes to a climate of disrespect for educators and education that is counterproductive when you’re trying to improve education. As Diane Ravitch said, you can’t win a war when you’re firing on your own troops.
The newspaper’s job is to inform, not to judge. Labeling teachers “ineffective” – and basing that on a measure that is widely viewed as unsound (and overwhelmingly so by the most knowledgeable experts in the field) – puts it in the position of judging. The Times also made the decision to ignore the many learned opinions cautioning that the measure is unsound – another judgment.
Given the high degree of disagreement about whether that gauge is a valid measure of teacher effectiveness, the Times should not have published the teachers’ names.
You haven’t asked me the question that leaps out to me, which is: If we absolutely knew that value-added was a 100% accurate way to assess teacher effectiveness, then would I support publishing the names? I have to say, probably. But then, I think that if it were that simple and clear to assess teacher effectiveness, that would change the whole nature of administrator evaluations of teachers and the issue of whether the Times “needed” to step in would be moot.
The other obvious looming question, which I’ve asked before but I think not here, is: The newspaper business is collapsing and something needs to be done to save it. Shouldn’t we be getting rid of ineffective newspaper journalists? Wouldn’t it be clear and simple to rate them based on shifts in the newspapers’ circulation and profits? And for good measure, we can measure based on the hits on their work as posted on the newspapers’ websites. Oh, but circulation and profits go up and down (or these days, only down) based on factors beyond reporters’ and editors’ control. And everyone knows that sports and celebrity gossip get all the hits, while education stories and in-depth coverage of the state budget crisis are important but attract far fewer readers. So if reporters were rated (and paid) based on hits, no one would want to tackle the tough but important issues. I see a few parallels here.
At the UC-Berkeley forum, Richard Rothstein related how Medicare at one point released rankings of cardiac surgeons based on their survival rates, at which point cardiac surgeons started shunning the sickest and most high-risk cases. Sounds fine until you need someone to be a hero and try to save the life of your own loved one in a desperate situation.
Report this comment for abusive language, hate speech and profanity
Gary Ravani: I’m not a psychometrician nor do I claim to be. I have worked with enough of them, however, to know that scholarly opinions about the pros and cons of the value-added methodology (VAM) are far from unanimous. I’m frankly astounded that a “statistical expert” described VAM as “an attempt at regression analysis,” when the two methodologies come at the problem of measuring a student’s year-over-year increase in achievement from completely different angles. Allow me to describe the two approaches and I think you’ll see what I mean.
Regression Analysis
A regression analysis uses the actual values of a dependent variable (the amount you want to estimate) and one or more independent variables (the amounts you want to use to estimate the dependent variable) drawn from a sample of data, in order to estimate the “true” relationship among these variables for the entire population from which the sample is drawn. Why draw a sample? Why not use the entire population? There are three reasons. First, it may cost too much to gather all of this data. Second, we need this relationship in order to predict the value of dependent variable not in the sample, based on the values of the independent variables. Third, there is random error – the actual values (a dependent variable and one or more independent variables) for each “data point” in the sample will not fall exactly on the line describing the “true” relationship.
For example, suppose that we want to use family income (x) to predict test scores (y)for all a district’s fifth graders. We don’t have enough money to collect data for every fifth grader, so we draw a random sample of data from the population of all fifth graders. Each piece of data has the student’s family size and his or her test score. At this point, we could throw all of this data into a “regression machine” and out the other end would come an estimate of the line that “best fits” the sample data.
Suppose the regression equation is y = .001x. Because we want to estimate Johnny’s test score given his family’s income, and we know his family income is $65,000, we put $65,000 for x in the regression equation, multiply $65,000 by .001, and we get our estimate of Johnny’s test score. Johnny’s predicted test score is 65 – even if his actual test score was 60. We can also estimate what his score would have been if his family’s income was $70,000. It would be 70.
A problem with regression analysis is that there may be other important variables, like class size (z) that also affect the value of y. If we don’t take account of (“control for”) all of these, the estimated relationship between y and x will be inaccurate. Suppose Johnny’s class size is 32. So, we add z to the data collected in our sample and throw these into the regression machine. Now we’re estimating the relationship between y, on the one hand, and x and z, on the other. Suppose that now, the estimated relationship among the variables is y = .0015x – 0.6z. Or, in Johnny’s case, y = (.0015)(55,000) – (0.6)(32), or 63.3.
The test scores produced by the “regression machine” are estimated – not actual. They have to be, because it’s the only way to control for the influence of class size on test score.
So, how do we estimate year-over-year increase in a classroom’s average test score? Well, you could do it using each student’s actual values, but that wouldn’t account for the influences of family income and class size on test scores. (This is analogous to the unadjusted decile rankings in the Academic Performance Index.) If we want to adjust for these, we go through the whole process again, but using data that is a year newer. (This is analogous to the adjusted API rankings.) Of course, the values of each student’s y, x, and z will be different, as will the estimated relationship among them. You put each student’s actual values for y, x, and z into the regression machine and out the other end comes the estimated relationship between y and x, controlling for z but using the newer year’s data. You then get an estimated value of y (test score) for each student by plugging into the regression equation his or her actual x and z.
Finally, to get a student’s growth in academic achievement, we would subtract his estimated prior –year achievement level from his estimated current-year’s. Lots of estimates there … with lots of potential for error.
Wow! This really gets complicated. But wait… it gets worse. What if the relationship between y and x is a curve rather than a line? Suppose that this year’s test score is correlated with last year’s? And, worst of all… suppose you left out an important variable that needs to be controlled for? You’re totally up a creek.
Value-Added Methodology
VAM eliminates or minimizes all of these problems. It is based on a simple insight: a student who is tested today is the same student, with the same characteristics, as he or she was when tested last. Clearly, this assumption is a simplification. But it merits serious consideration vis a vis the extremely complex regression methodology.
Remember all of the problems due to year-to-year correlation, inaccurate specification of relationship between y and x, estimated rather than actual values, omission of important control variable? They don’t exist with VAM. You don’t have to control for them and produce estimated values because the student tested in one year is the same student as the one tested the following year. None of his “control variables” have changed!
As with my description of regression analysis, I omitted some of the more obscure issues in the interest of brevity and (I hope) clarity.
My points are simple:
VAM is not at all like regression analysis
Neither VAM nor regression is perfect.
VAM’s methodology is straightforward, logical, and easily understood
VAM merits serious consideration as one part of an overall evaluation system.
Report this comment for abusive language, hate speech and profanity
I’d be curious whether the organizers of the UC-Berkeley forum were able to find any experts willing to be on a panel and make these statements:
VAM’s methodology is straightforward, logical, and easily understood
VAM merits serious consideration as one part of an overall evaluation system.
Even Eric Hanushek did not make the former statement. He did basically make the latter one, though in my opinion in a manner displaying a surprising lack of enthusiasm. (John disagrees, as we’ve seen.)
And all that is colored by the fact that Hanushek is paid to speak for an organization that strongly promotes the pro-charter, pro-privatization, pro-free-market viewpoint — which would tend to be aligned with the view that teachers should be subject to more critical evaluation – and thus walked in with a strong bias. (Or at least he was paid to represent that agenda — with paid spokespeople, who knows what they really believe?)
Report this comment for abusive language, hate speech and profanity