API has served its purpose

A court decision this week involving Los Angeles Unified has raised again the contentious issue of evaluating teachers using standardized test scores. But a recent report for the think tank Education Sector recommends adopting the same method developed by Los Angeles Unified to replace the Academic Performance Index as a statewide way of measuring schools’ progress.

Called Academic Growth over Time, AGT is a value-added model that compares students’ actual performance on state tests to their predicted performance based on demographic characteristics – family income, language, and ethnicity – as well as past test scores. The intent is to distinguish factors of learning that schools can control from those they can’t.

The use of AGT to evaluate individual teachers has sharply divided teachers in Los Angeles Unified. United Teachers Los Angeles opposes using AGT in any manner, while teachers affiliated with Teach Plus Los Angeles and Students Matter support using it as one of several measures, counting for no more than a third of an evaluation. But less controversial is the district’s use of AGT as a tool to evaluate schools, in part because it involves a larger number of student test scores and doesn’t call for high-stakes decisions affecting individual teachers’ careers. To the contrary, a schoolwide AGT can encourage collaboration and team-teaching

This is a page from the Academic Growth over Time report for Taft Senior High School in Los Angeles Unified, cited in the report for its underwhelming achievement. Scores in green indicate a performance that exceeded the district averge for the popularion of students served (Algebra II over three years); gray is close to the dsitrict average (geometry); yellow is below the predicted AGT (English language arts last year; and red (Algebra I last year) is far below the predicted AGT. Taft's overall API was 744 last year; for whites, who comprise 40 percent of the student body, it was the state's target of 800; Source: Los Angeles Unified

This is a page from the Academic Growth over Time scorecard for Taft Senior High School in Los Angeles Unified, cited in the Education Sector report for its underwhelming achievement. The score, on a 1 to 5 scale, in green indicates a performance that exceeded the district average for the population of students served (Algebra II over three years); gray is close to the district average (geometry); yellow is below the predicted AGT (English language arts last year); and red (Algebra I last year) is far below the predicted AGT. Taft's overall API was 744 last year; for whites, who comprise 40 percent of the student body, it was the state's target of 800; for Hispanics, it was 695. (Source: Los Angeles Unified)

Last fall, for the first time, Los Angeles Unified released AGT report cards for all schools, breaking down every subject or grade taught on a scale of one to five, with students’ actual scores compared with where they should have been, given student populations, for a one-year and a three-year average. The AGT’s advantage is that it can highlight improvements in high-minority, high-poverty schools that may flunk under the federal and state accountability criteria, while pointing to mediocre performances in high-wealth schools that can glide by the targets of No Child Left Behind and the state’s API.

Here is the AGT report card for Audubon Middle School for 2010-11. All of he subject and grade level scores are in green and blue, indicated progress that exceeded and far exceeded the district averages. Its API score remains relatively low at 733. (Source: Los Angeles Unified.)

Here is the AGT report card for Audubon Middle School for 2010-11. All of the subject and grade level scores are in green and blue, indicating progress that exceeded and far exceeded the district averages. Its API score remains relatively low at 733. (Source: Los Angeles Unified.)

The Education Sector report pointed to Audubon Middle School that, under a new principal and re-energized staff, had a 12 percent gain in the API score in one year. But it was still in the bottom 20 percent and failed to meet the proficiency target under NCLB for the 10th straight year.

The state’s three-digit API number, on a scale of 200 to 1,000, is “a crude proxy for student achievement and allowed schools to be ranked,” writes Richard Lee Colvin, former executive director of Education Sector and author of “Measures That Matter: Why California Should Scrap the Academic Performance Index.” “But it was not designed to give educators much help in analyzing school performance, and it told the public more about who attended each school than how well they were being taught.”

The API’s shortcomings have been known for a long time, and Colvin  lists them:

  • It’s an indicator of students’ wealth rather than of a school’s educational quality;
  • It places too much emphasis on math and reading scores, so that schools end up giving short shrift to science, social studies, and the arts ­– subjects that don’t factor much or at all in the API number;
  • More than 40 percent of schools are above the arbitrary target of 800 and so are no longer held accountable for helping students who are struggling academically;
  • It doesn’t track individual students’ academic growth over time; progress is measured by comparing  how students in a particular grade or subject do one year, compared with different  students the previous year.

Narrow measure of school success

The Legislature had intended that the API be a wider index when it created the index in 1999, but nothing has changed. Now, for the second year, Senate President pro Tem Darrell Steinberg has proposed SB 1458 to broaden the API to include possible factors as graduation, dropout rates and college acceptances, and Advanced Placement scores, along with giving science and other subjects more weight. In a nod to Gov. Jerry Brown, who suggested the idea, Steinberg’s bill could include the results of school inspections measuring non-quantifiable but important factors like school climate and parent evaluations.

There’s no reason why a new index that emerges – whatever it’s called – couldn’t also incorporate AGT as a measure of student progress in combination with proficiency rates on state tests. Colvin said that the costs for districts to compute the AGT scores for its students need not be significant; Colorado has developed an open-source model that districts or the state could buy for $250,000.

State Board of Education President Michael Kirst said he was open to innovative accountability models, but that now is not time to switch to value-added method. The state will begin using Common Core assessments in 2014-15, and at least two or three years of new data would be needed, bringing the adoption of a new system to 2018-19 at the earliest. The State Board will be reviewing the state’s accountability methods over the next year. Colvin called for making a commitment to AGT now and preparing for a transition. The State Board could grant waivers from the use of API to districts like Los Angeles Unified in the meantime.

But Los Angeles Unified Superintendent John Deasy told me the district was interested in a federal waiver from No Child Left Behind, not a state waiver, so that it get out from federal sanctions for school failures as the feds defined it and also gain more control over federal Title I money. After months of delay, the state has requested an NCLB waiver, but not on terms requested by the Department of Education; getting the waiver would appear problematic.

This entry was posted in 10.1Assessments, Achievement Gap, Common Core standards, Tests on by .

About John Fensterwald - Educated Guess

John Fensterwald, a journalist at the Silicon Valley Education Foundation, edits and co-writes "Thoughts on Public Education in California" (www.TOPed.org), one of the leading sources of California education policy reporting and opinion, which he founded in 2009. For 11 years before that, John wrote editorials for the Mercury News in San Jose, with a focus on education. He worked as a reporter, news editor and opinion editor for three newspapers in New Hampshire for two decades before receiving a Knight Fellowship at Stanford University in 1997 and heading West shortly thereafter. His wife is an elementary school teacher and his daughter attends the University California at Davis.

19 thoughts on “API has served its purpose

  1. Navigio

    Ironically, Audubon has met all its API targets for the past 2 years and it’s school-wide target for the past 4 or 5 years. So that may not be the best example for a school where API is failing it. :-)

    Regardless, there is something mentioned in this article that has always bothered me about these kinds of metrics. Namely, that the students being measured are different from year to year. I’m not so much concerned that this can’t be ‘adjusted’ for by trying to follow cohort or individual progress, rather I think it begs the question of why we should expect something like API to improve over years when the ‘quality of the input to the system’ is staying the same, or even declining from year to year.

    It seems like there are only two possible answers: either we are getting better at teaching to the test (ie narrowing our curriculum) or teachers were and are not doing their jobs and having these metrics shine a light on that ‘incentivizes’ them to do better.

    The absurd thing is that it is not possible to separate those two things, and worse, even for teachers who were/are doing their jobs, ‘progress’ will still be possible to equate with narrowing the curriculum.

    To be honest, I think narrowing the curriculum was actually the goal. This has been driven by policies that were created to address the lack of English and math proficiency rates to the exclusion of most everything else, and worse, ignoring the impact that would have on students who were already proficient. And I think it’s pretty much impossible to argue that narrowing the curriculum is not what we’ve gotten. This may not be surprising given we’ve begun to prioritize policy based on optimizing performance over dollars. Even our most influential ‘education’ reformers are economists.

    I look at the over 400% improvement in 7th grade African American math proficiency rates in LAUSD over the past decade or so and wonder what that is a reflection of. This has happened in spite of a 500% to 600% increase in the number of students segregating to charters and an increase in poverty rates. (ironically I often get the argument that charters are valuable because they have the ‘freedom’ to not narrow the curriculum by not teaching to the test–sigh). And after all that, we are attacking teachers more than ever, because now, and especially now, our education system has finally failed. Double sigh.

    I look at the changes in our educational priorities over the past decade or so and wonder whether we can afford another decade on that same path. Perhaps it’s time for us to  lift our heads out of the water and stop for a moment to take a breath and decide whether we are still heading where we actually want to be.

    In the end, the sad part is we are not even able to differentiate our lack of desire to properly use ‘accountability’ methods from the ‘accuracy’ of those methods themselves. The problem with API is not so much API, but it’s that we think it means (and want it to mean) something other than it does. It kind of reminds me of the ongoing debate over teacher evaluations based on test scores. People (including those who designed them) keep trying to make the point that the tests were not designed to measure teachers. But we ignore that fact and continue to focus on new and ingenious ways to use them to do just that. Even the wrong method will surely work if we can only apply it more forcefully.

    Perhaps the best thing we can do to improve our accountability tools is teach our citizens what they actually mean, regardless of how they are defined.

    Report this comment for abusive language, hate speech and profanity

    Reply
    1. John Fensterwald - Educated Guess Post author

      Navigio: API and AGT scores are both based on the results from the California Standardized Tests (CST); if CST scores rise significantly in any year, the API score will rise. So Audubon will make its targets, as you point out. However, its API score of 733 is still low as is its similar schools ranking of 3 out of 10, and it remains in School Improvement status under NCLB. However, its AGT report card looks terrific because of the progress it has made. (I have added it to the post.) So, parents and others who focus on the API number will now see that Audubon is a turnaround star. That’s the value of AGT as a school metric.

      I share your concern over teaching to the CST. The Education Sector report suggested replacing API with AGT, which won’t address that issue. I imply that it would be better to include AGT as one factor in whatever accountability tool the State Board and the Department of Education settle on. Similarly, the AGT should be one of several factors – though with even less weight – in a teacher’s evaluation.

      Report this comment for abusive language, hate speech and profanity

      Reply
  2. Kim Kenne

    I agree that API/AYP (CST results) should not be the only way that a school is judged.  But it is a handy, quick indicator.  I do prefer looking at actual CST proficiency results in all subjects – too much can be hidden in one API number.  And 800 is not the goal we should be striving for.  Having said that, I have two points:

    - Audubon made significant growth.  So that progress should be measured and lauded.  But the school still has fewer than half of its students at a proficient level in ELA and Math (45.1% and 40.4% respectively for 2011).  Our accountablity system needs to capture that absolute standard as well.  Otherwise we can get caught up in incremental improvements and not notice that schools (and students) are not really ever reaching grade level standards.
    - I have a concern about AGT – who is setting  “where they should have been”?  If we compare the actual scores to this idea of what the students could have/should have achieved, how does the public know what this standard was based on?  If low expectations are set for high-poverty students, would we fool ourselves into thinking those students were fine even when they weren’t?

    Report this comment for abusive language, hate speech and profanity

    Reply
  3. Paul Muench

    I’m trying to figure out the word growth in AGT.  The reports I’ve found on the web describe a scale where the district average is set to 3.  So if all schools in a district grow equally it seems as if no growth is shown in the numbers.  For all practical purposes maybe the average stays stable from year to year so any school that grows will have growing numbers as well.  It seems a more optimistic approach to peg growth to an absolute scale such as the proficiency levels in the CST.

    Report this comment for abusive language, hate speech and profanity

    Reply
  4. Pingback: June 15, 2012 « tigersteach

  5. navigio

    Hi. Out of curiosity, I took a look at some of Audubon’s characteristics. Here are a few points.

    First off, the size of the school itself has changed significantly. Although its performance has only recently improved, the enrollment trend is consistent over the long term. In 02-03 it had over 2,300 students. In 10-11 it had just over 800. Thats quite a difference.

    Its average class size was hovering around 28 for many years then in 08-09 dropped to about 22 and in 10-11, to 19. In comparison, LAUSD overall hovered around 27 for those same many years, then dropped to about 25. That is a significant difference. (the student to teacher ratio was 13 for the school vs 20 for the district in 10-11; prior to 07-08 the school’s ratio was higher than the district).

    Similarly, student to pupil services employee ratios were much higher than the districts for many years, then in 06-07, dropped to much lower than the district’s.

    Its ratio of computers to students is about 30% better than LAUSD overall (though admittedly, this metric is highly inaccurate since computers that are over a certain age are not counted as valid computers).

    Perhaps most surprising was that the percentage of credentialed teachers at Audubon was only around 70% for many years then suddenly jumped to over 94% in 08-09. In contrast, LAUSD had rates that were 10 percentage points higher for almost all that time, and then in 08-09 went up to 95%.

    The percentage of english learners was near 15% for a number of years, then dropped to closer to 10% in the most recent years. That is obviously a very large difference from the district as a whole, which has closer to 30% (though that is also down from over 40% 5 or more years ago). Although it would not explain the extreme increases in scores in the past few years, the hispanic population (as a percentage of total enrollment) has almost doubled over the past decade, while african american rates dropped from 80% to just over 60% in that same time.

    Anyway, I dont know this school personally, but it seems some things changed quite significantly there in the past handful of years. The most recent years are the ones in which performance metrics dramatically improved. Its hard to really say what kinds of changes occurred without being there, but it appears that they made a point of providing the school with qualified teachers (for a change) and support staff and reducing class sizes dramatically. If this is what actually happened, maybe the ‘turnaround’ is not so surprising?  Maybe we can take some hints for our state-wide policy.. ?   :-)

    Report this comment for abusive language, hate speech and profanity

    Reply
  6. Cal

    “It’s an indicator of students’ wealth rather than of a school’s educational quality;”

    Incorrect. It is an indicator of the students’ cognitive ability. If it were an indicator of student wealth, then poor white students wouldn’t outscore non-poor black students and tie with non-poor Hispanics. Cognitive ability is loosely linked to income, but not perfectly., Race is a more accurate predictor. Two groups of students that have identical percentages of each income quintile should have equal achievement outcomes, if the wealth indicator were accurate. But if group A is entirely black and group B entirely white, group B’s scores will trounce group A.
    But more importantly, taxpayers don’t want to dump the API, I suspect.. Because taxpayers want to know what the actual student achievement level is, not how well the teachers in super low-income, high crime School X are doing at raising barely literate freshman reading scores to something approaching a sixth grade level.
    Taxpayers, specifically home buyers and sellers, want the API. Take it away from them and pretend that Taft High School is a better school than Monta Vista High School because hey, Monta Vista students stay stuck at 97% advanced  while Taft students moved from 10% below basic to 30% below basic? Yeah. That’s going to fly.
    But go ahead, policy makers, and devalue California housing in a down market.

    http://bayarearealestatetrends.com/2012/01/19/comparing-school-api-scores-along-the-680-corridor/

    Report this comment for abusive language, hate speech and profanity

    Reply
  7. Manuel

    Do any of you know the origins of the API?
     
    Do any of you know why “800″ is the magical number?
     
    Do any of you actually know what is in the CSTs? (No, trusting them to actually cover all California Standards in a meaningful way is not enough. Pineapples don’t have sleeves, you know.)
     
    Do any of you actually trust the CST not to be a bell curve, which, of course, renders any value-added model useless?
     
    It is amazing to me that none of the contributors of this blog that have dismissed my above queries in the past has ever looked at them without prejudice and actually dug into them for themselves. As they say in the old country: “there is no worse blind man than the one who doesn’t want to see.” (Yes, I know, it is one-third of the sanzaru.) And so it goes…

    Report this comment for abusive language, hate speech and profanity

    Reply
  8. Ze'ev Wurman

    Some background info.
     
    API is made up of weighted average of a series of STAR tests. Details change slightly over time but it is mostly a mix of math & ELA in K-8, and of math, ELA, science and history in 9-12. Details are here.
     
    The 800 threshold for an OK school/district is somewhat arbitrary and it reflects what was considered reasonable for most schools. I agree that there is no particular reason to expect the API to grow forever. The expectation (hope?) was that it will flatten for all schools somewhere between 800 and 100.
     
    As a side comment, current CST generally cover the overwhelming number of content standards. Depends on the grade and subject to a degree but typically in the 80-95% range. Blueprints of the standards coverage by CST can be found here.
     
    The idea of showing API trend as a measure (which is what AGT seems to do) is a nice one, and showing the trends for each of the components — and in comparison to similar demographics — is fine too. But let’s not confuse it with value-added measures — it is not. The CSTs that lie at the root of AGT are not vertically calibrated and tracking individual student data in California is limited to within a district, both seriously limiting the value-added value (pun intended) of AGT.
     

    Report this comment for abusive language, hate speech and profanity

    Reply
  9. Ze'ev Wurman

    Navigio, I wanted to spend some time on this paragraph of yours
     
    I look at the over 400% improvement in 7th grade African American math proficiency rates in LAUSD over the past decade or so and wonder what that is a reflection of. This has happened in spite of a 500% to 600% increase in the number of students segregating to charters and an increase in poverty rates. (ironically I often get the argument that charters are valuable because they have the ‘freedom’ to not narrow the curriculum by not teaching to the test–sigh). And after all that, we are attacking teachers more than ever, because now, and especially now, our education system has finally failed. Double sigh.

    The 400% improvement in LAUSD black 7th graders’ math doesn’t surprise me. It is similar to the 4.6X increase in successful Algebra 1 taking by grade 8 since 2003 by black students (as compared to only 2.6x increase for the cohort). I attribute it to the much clearer and stronger curriculum since we’ve got our standards in 1997. Before then, blacks were frequently fed with diluted curriculum. Consequently, even with their rapid improvement, blacks have not yet caught with the cohort as they started 3X behind.
     
    Regarding your “500% to 600%” increase in charter population, this would be a proper use of stats if the focus was on charter growth. When the focus is supposed to be on the potential impact on LAUSD public student body, the actual fraction must be supplied otherwise it is misleading. In this case this growth represents a change from about 2% to about 10% — significant but not overwhelming. For comparison, New Orleans charter enrollment is at roughly 60% and DC at about 40%. And, as you yourself observe, this charter enrollment does not seem to indicate cherry picking as the regular public black achievement increase remains strong.
     
    Finally, whether you get the argument that charters are valuable because they “have the ‘freedom’ of not teaching to the test” or not, it does not change the fact that every school has the freedom of not teaching to the test. Indeed, every school has the obligation of not teaching to the test. Both the regular public schools and the public charters still have to test their students on state tests, but if charters are smarter and understand that the road to better achievement (on that same test!) is not through teaching to the test, so much the better. This in itself seems to justify their existence.

    Report this comment for abusive language, hate speech and profanity

    Reply
  10. navigio

    Hi Ze’ev, I did intend to focus on charter growth. LAUSD’s total enrollment is falling, while the number of kids in charters rises. In 03-04, charter enrollment was about 3% of the total. In 10-11 it was 13%. 11-12 numbers arent all out yet, but total enrollment is down again, and the number of charters schools in LAUSD increased by 19 last year. Charter enrollment statewide went up 13% last year alone. My guess is that the percentage of LAUSD kids in charters will be between 15% and 18% for 11-12. If the number happens to be on the higher end, that puts almost 1 in 5 LAUSD kids in charter schools. In the second largest school district in the country. I’d say thats pretty significant, and maybe even overwhelming. Note also that during that same span, the number of classified employees in LAUSD has decreased by only 5%, but the certificated (including teachers) by 9%.
     
    NOLA is a very different situation.
     
    The issue of cherry picking is pretty independent of some absolute measure of kids that are left behind (ie you cant say what might have happened had they stayed). My comment about teaching to the test was not intended to make a claim about the teaching methods of charter schools or traditional public schools, rather it was intended to highlight a contradiction amongst the reform, accountability and charter movements.  An interesting situation: I have seen a charter elementary school created that immediately attracted all the families of ‘the higher performing ilk’. In fact, the first year it had API scores, it was the highest scoring school in the district (funny, think about where all those kids came from–yep, straight from the public schools). As the years went on, the scores fell. And fell. And fell. In fact, the longer kids were in the school, the worse they did on the tests. As a response to this phenomenon, parents from the school said, ‘well, we dont focus on teaching to the tests. Its clear our scores will be lower.’  While it is clearly a valid strategy to not teach to the test (and Im not even saying traditional public schools necessarily do this, nor am I saying charters do not), my point was that the charter movement and test-related accountability go hand in hand. Its disingenuous to lambast traditional public schools based on their test scores, then completely ignore test scores once you get the kids. To be honest, I dont believe the tests were actually ignored, I think that was an excuse for some other problem. I do agree with you that success can be had without ‘teaching to the test’ per se. Though I still think it is ironic that we get both pressure to perform (on tests) and criticism for trying to make that happen.
     
    Overall, I dont think charters are justified by either teaching or not teaching to the test (in fact, mostly I dont think they are justified by anything, but thats a separate discussion..  ;-) ). They still need to perform or they get shut down. I know of charters that focus highly on tests. I know of ones that claim not to. I know of ones that only focus enough to stay above 800. etc, etc..
     
    Perhaps the greatest irony, and I apologize for continuing down the tangent, but one of the arguments for charter schools was that they were to ‘pressure’ traditional public schools into performance. I think its pretty clear that was not really the intent. Charters are replacing traditional public schools. Even as the ‘left-behinds’ continue to improve, charter enrollment is increasing (not clear yet whether it is exponentially, but the increase in CA charter enrollment has been larger than the previous year in each of the past 8 years.. in math-speak, I think this corresponds to a positive second derivative on the function that is CA charter school enrollment by year..  :-) ).
     
    El, I wrote a couple posts about similar schools, but they were trapped in the toped moderation filter..   lets hope at least one is let free..  :-)

    Report this comment for abusive language, hate speech and profanity

    Reply
  11. el

    I keep coming back around to the idea that one of the reasons charters are very attractive in LAUSD is that LA Unified schools effectively have no local governance. If you have an issue with one of their 700 schools that isn’t satisfied by a discussion with the principal, you don’t really have anywhere to go. Even if say a dozen parents get together and decide to attend a board meeting, most likely the board members have never even visited their school and will have limited ability (or interest) in solving the issue. Part of the appeal of charters is that governance, for better or worse, is more likely to be concentrated at the individual school site, and that meetings and the like will also be at that site.
     
    LAUSD could be ten districts and they would still be large districts.
     

    Report this comment for abusive language, hate speech and profanity

    Reply
  12. navigio

    Having single schools of governance does not seem like the proper solution to the large district, though I agree with you that it does not seem appropriate to have such a large district. That said, there are subdistricts within LAUSD, each with its own super (http://tinyurl.com/7sp9t79) but I am not sure how much power they have over local school issues (I expect they would have to, otherwise there would be no point), nor whether they have their own local ‘board’.

    I think one reason charters are so appealing in LAUSD because there is such a diversity in population. The city (and region) of LA encompasses areas that are borderline 3rd world, as well as some of the most exclusive residences in the world. Even away from the extremes, the differences can be stark and geographically immediate (eg across a freeway or street). Plus, charter schools are an easy way out of the need to think and understand. Something that is particularly problematic in this area, imho (I get to say that because I’m from here. :-) ).

    Report this comment for abusive language, hate speech and profanity

    Reply
  13. Pingback: More Thoughts on LAUSD, Standards and Evaluation « InterACT

  14. Pingback: Another report urges changing API | Thoughts on Public Education

  15. Pingback: After AB5: Trust, Consensus, Mutual Responsibility « InterACT

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>