CALPADS ‘in danger of failure’

Consultant lists defects and urges action
By

The state Department of Education has received but not yet publicly released IBM’s response to its demand to fix the troubled statewide student data system, known as CALPADS, that the company is building. The problems are serious enough that consultants who did a systems analysis earlier this month concluded that CALPADS was “in critical danger of project/system failure.”

“The system should be stabilized before any further development is conducted,” wrote Sabot Technologies of Folsom in a 15-page report. Even at the risk of further delays to a system that is already a year behind schedule, “the instability and performance problems with the system warrant decisive focused corrective action,” the report said.

Sabot has done several analyses of CALPADS over the past year. In a report in June, Sabot was encouraged by progress that IBM had made. But everything changed in November, when IBM updated software to accommodate the next round of reporting requirements by school districts. The system encountered new instability that, if not fixed, would be compounded when districts file two additional reports later this year.

The problem fit a pattern, Sabot said. “There have been delays in all major releases as a result of a variety of IBM shortcomings. IBM continually causes delays for reasons including quality issues, underestimation of scope, and poor coordination of testing.”

The biggest problem was IBM’s inability to process large quantities of data overnight that districts have uploaded. It’s supposed to be done in eight hours, but IBM had regularly missed the deadline for reasons that aren’t clear. “It is apparent that IBM does not know the root cause of the performance issues,” Sabot said.

IBM is supposed to design a system that will function effectively three years out, which would mean that CALPADS soon should be able to process the overnight data in four hours. IBM has resisted creating tests that would simulate conditions that could be expected under a heavier workload, the report said.

A year ago, Sabot criticized the quality of the work and the experience of the team that IBM assigned to CALPADS. The latest report was equally critical. “Despite these improvements (in communications and coordination), there remain critical issues with quality control in the IBM engineering process. In analyzing the results, the IBM team has demonstrated that they do not possess the technical skills to design and develop the system to contractual specifications.” (How about sending in the team that worked on Watson?)

Noting a “longstanding lack of proactive attention” to problem areas, Sabot recommends that IBM send in a strike team to do a thorough systemwide analysis.

In a letter to IBM earlier this month, Richard Zeiger, the new Chief Deputy Superintendent of Public Instruction, wrote that the Department had determined that IBM’s “substandard work” constituted a material breach of the CALPADS contract.

The Sabot report supported that conclusion: “IBM is grossly out of compliance with their contract schedule and as a result the project is grossly out of compliance with its planned schedule for completion and realization of value as stated in the project approval documents.”

In a statement yesterday, IBM disputed the report’s conclusion and insisted that it remained on track to deliver the completed system by the end of June.

“We disagree with the conclusions of the Sabot Report and it is important to note that CALPADS currently is operational in California school districts who are successfully using it to submit information vital to tracking the progress of the State’s students. IBM continues to work in partnership with the California Department of Education in order to meet the ambitious scope of this project,” it said.

Paul Hefner, director of communications for the Department of Education, declined to speak about IBM’s response to Zeiger’s letter, but he did say that the Deparmtent has noticed “a stepped up level of activity and efforts to improve the performance of the system” by IBM since the letter was written, which the Department views as “hopeful signs at this point.”

Tagged as:

9 Comments

  1. Thank you for continuing to provide information on the CALPADS system. As an educator, I resent the fact that I am held accountable for student issues beyond my control but NO taxpayers are calling for the head of IBM for its inability to get this multi-million dollar system up and functioning.  This is a fiasco at best and taxpayers ought to focus some attention on IBM and hold it accountable as well.

    Report this comment for abusive language, hate speech and profanity

  2. I would also like to thank the author and site for providing this information.
    Even when CALPADS “works” it doesn’t. The simplest of tasks are ridiculously difficult. Difficult issues are a nightmare. We know this is important and necessary. All across California people are working nights and weekends to meet deadlines. Give us a system that is easier to use and up most of the time.
    I hope the people in charge talk to the end users. I for one am fed up and will no longer be a team player supporting CALPADS. I will tell anyone who will listen how bad it is.

    Report this comment for abusive language, hate speech and profanity

  3. It should be handy for IBM to put Watson on the job now that its power has been demonstrated. CALPADS should be child’s play …

    Report this comment for abusive language, hate speech and profanity

  4. Even if CALPADS “works,” and even if it “works” in Bob B’s use of the term, one still needs to question the system’s utility.  Since California’s assessment data are of such poor technical quality and are not scaled across grade levels, even a “working” CALPADS system won’t be able to make much practical use of the assessment data.
    In essence, California has put the CALPADS “horse” ahead of the testing “cart.”  Instead of beating a dead horse and spending millions on it, perhaps the state should drop CALPADS and instead spend the funds on building a working standards and testing “cart.”

    Report this comment for abusive language, hate speech and profanity

  5. John, thanks so much for your continuing your coverage of CALPADS.  I often wonder where things went astray.  One thing I do know.  SB 1453 which originally created CALPADS required the California Department of Education to form an advisory committee “to establish privacy and access protocols, provide general guidance, and make recommendations relative to data elements.”  I was a member of that committee.  I’m not sure why, but as I testified before the Senate Education Committee in September 2009, the department last convened a committee meeting in August 2004.  In June 2007, the department notified committee members it would reorganize the committee to address both CALPADS and CALTIDES issues concurrently.  In 2008, the committee was simply dropped off the department’s website.  I also know that during its short existence, various committee members with very substantial data base experience, including Margaret “Macke” Raymond of Stanford’s Hoover Institution, offered what I thought was insightful feedback and advice, most of which has been ignored.  In hindsight, one can only wonder what would have happened had the department decided to follow the law, continue with the committee and incorporated at least some of the advice in developing CALPADS.

    Report this comment for abusive language, hate speech and profanity

  6. Eric, thanks again for pointing out the CA Dept of Ed’s disappointing failure to vertically scale STAR results (principally the CST) either across grades in a given year or across years.  Ginger Hovenic, Ed.D and I in a meeting once asked Deputy SSPI Deb Sigman (who now leads the Curriculum, Learning, and Accountability Branch) why this was so.  Deb advised the standards were/are not vertically aligned, which prevented vertical scaling of the results.  That same day Ginger and I visited with Scott Hill, the former Chief Deputy SSPI who was also the Executive Director of the California Academic Standards Commission and Curriculum Commission.   Interestingly, Scott told us the commission believed the standards were vertically aligned and therefore scaleable.   Either I misunderstood or there’s a disconnect somewhere.  (FYI, a number of education analytics organizations, including SAS in School and UCSD’s Julian Betts, have developed statistical models to vertically scale STAR results.)

    Report this comment for abusive language, hate speech and profanity

Trackbacks

  1. Tweets that mention CALPADS ‘in critical danger of failure’ | Thoughts on Public Education -- Topsy.com
  2. CALPADS ‘in critical danger of failure’ (TOP-Ed/SVEF) « Education Advocacy

"Darn, I wish I had read that over again before I hit send.” Don’t let this be your lament. To promote a civil dialogue, please be considerate, respectful and mindful of your tone. We encourage you to use your real name, but if you must use a nom de plume, stick with it. Anonymous postings will be removed.

10.1Assessments(37)
2010 elections(16)
2012 election(16)
A to G Curriculum(27)
Achievement Gap(38)
Adequacy suit(19)
Adult education(1)
Advocacy organizations(20)
Blog info(5)
CALPADS(32)
Career academies(20)
CELDT(2)
Character education(2)
Charters(82)
Common Core standards(71)
Community Colleges(66)
Data(25)
Did You Know(16)
Disabilities education(3)
Dropout prevention(11)
© Thoughts on Public Education 2014 | Home | Terms of Use | Site Map | Contact Us