Is CALPADS unfixable? No answer yet
State education officials expressed deep disappointment last week on learning that California was out of the running for money to expand the statewide student data system.
They haven’t heard yet why the state placed 26th out of 50th in a grant competition that funded only the top 20 states. But they shouldn’t be surprised if the feds’ answer is, “Are you kidding? Why would you expect taxpayers to enlarge a data system when you have yet to get it to work right?”
Nearly one year into its operation, CALPADS, the California Longitudinal Pupil Achievement Data System, is still struggling. Five months after a consultant warned of an imminent system collapse and urged a top-to-bottom review, the student data system is still being fixed. It will take at least a month before it becomes clear whether the processes work, and the system can perform as designed. Still to be determined is whether management problems – a big factor behind the poor operation – have been straightened out.
Meanwhile, the collection of key data elements – courses students are taking, their grades, credits, and their teachers, as well as information on attendance and English learners – has been pushed back a year. Only enrollment and dropout data will have to be uploaded by summer’s end.
The Department of Finance and the Schwarzenegger administration have grown frustrated with the delays, to the point of giving an ultimatum of sorts to the Department of Education, which oversees CALPADS and a related data base for teachers, CALTIDES. A one-paragraph reference in Schwarzenegger’s revised budget in May, called “Contingency Proposal for Longitudinal Data Systems,” warns that if the system can’t reliably receive and transfer data by the end of the year, the administration will “seek to contract these projects out to a consortium of local school districts” and higher education institutions to meet federal reporting requirements.
It’s not clear what that means, but Keric Ashley, director of the Department of Education’s Division of Data Management, is confident it won’t come down to that. CALPADS is in the middle of a two-month test of a software release that includes 200 repairs to stabilize the system. Ashley said that IBM, the system vendor, has assigned additional employees, at its expense, and the department has added oversight personnel. This was partly in response to the consultant’s conclusion in January that IBM had assigned its B team, under-experienced and understaffed, to the project, and that the Education Department’s poor monitoring and the “distinct lack of technical leadership and engineering resources” compounded the problems.
In an interim report on May 20, the same consultant, Sabot Technologies of Folsom, found some encouraging signs. Complaints and service calls were down; there had been no major outages and slow-downs in operation; there had been no system-threatening defects since March. However, Sabot also said that notable governance and operations issues remain unresolved, and the system hadn’t been up and running long enough, or under high-capacity use, to declare success.
A statewide data system that tracks students from preschool through college is the linchpin for school improvement. For more than a decade, every important study on school reform has highlighted the need for it to help determine what programs and practices work best, which teacher training programs produce the best teachers, which forms of spending are the most effective. Other states, such as Florida, are years ahead of California. In scoring Race to the Top applications, the Obama administration made data system a top priority (It’s one of the areas in which California was docked points.). In the latest round of federal funding, which California lost, the Department of Education allocated $245 million. California had sought $20 million to expand the K-12 data base to include higher education institutions and the workplace.
CALPADS would be the nation’s largest student data system. But it’s been beset by problems since going on line last August. Districts found the system would freeze, especially during peak times, when they tried to upload data. They complained about slowness.
Execution, not architecture, to blame
In its January report, Sabot concluded the architecture of the system, built on top of the California School Information Services system, was sound. But the execution and management were seriously flawed.
So far, 173 districts, out of nearly 1,000, have successfully uploaded the first phase of data, on enrollments and student dropouts. One of these, San Jose Unified, reported that CALPADS has already proven useful. By tracking where students go once they leave the district, San Jose Unified will be able to lower its dropout rate this year.
For years, there’s been a running argument between budget hawks at the Department of Finance and CALPADS advocates and the Department of Education over whether the state has invested enough in the creation of CALPADS and CALTIDES. Finance cites the $155 million, including $41 million for CALPADS and CALTIDES, that the state has spent on education data systems since 1997. Every year, Superintendent of Public Instruction Jack O’Connell calls for more investment in CALPADS and money to train district personnel how to use it. This year’s version, AB 2265, sponsored by Assemblywoman Mary Salas of Chula Vista, would appropriate $32 million to local districts for training.
The Sabot report recommended adding state personnel to monitor CALPADS, and the Department of Education requested three more technical positions. But Finance fought this on the grounds that the Education Department has yet to show it has competent managers in place. For now, the department has shifted personnel from other duties to work on the project.
The next critical date to judge CALPADS operation will be in July, when the last of this round of fixes to CALPADS is installed.
The assertion that “a statewide data system that tracks students from preschool through college is the linchpin for school improvement” is quite the overstatement–or at least getting the data cart ahead of the assessment horse.
California’s current assessments are not vertically scaled and thus one cannot, for example, compare the scores of a second grader in Year 1 with the same student’s third grade scores in Year 2. Thus, having a student data system that can track assessment data longitudinally is a component, but far from a linchpin. Having a longitudinal data system seems pointless if we lack an assessment system that can generate assessment data that is comparable across grade levels.
California currently has no practical plans to dump its Stone Age assessment system to generate vertically scaled data. In fact, the CDE is proposing to roll-over its contract with the Educational Testing Service for the third time and without going out to bid.
If we want longitudinal student assessment data, we probably need to start with revising the academic content standards to facilitate the process. Then we can put out bids to create an assessment system. After this groundwork, it may make sense to evaluate whether it is feasible to create a statewide longitudinal data system and whether it’s worth the cost. Doing the reverse order is classic cart before horse.
Just because “every important study” says something doesn’t make it so–we count on you John to cut through the BS.
Report this comment for abusive language, hate speech and profanity
OK. Point well taken, Eric. But notwithstanding the weaknesses of an assessment that currently lacks vertical scaling, a data system will prove its worth. Having a student identifier that tracks students as they move from district to district is one example. Seeing which students take algebra in what grade, who has a-g credits, which teachers got their training where, etc.
The renewal of the ETS contract is a separate issue worth exploring – particularly with the Obama administration willing to spend $380 million or so developing new assessments.
Report this comment for abusive language, hate speech and profanity
John hate to say it but Eric is partially correct. john
Report this comment for abusive language, hate speech and profanity
Eric is partially right, and I don’t hate saying that at all. However, I would add his assessment that any data system is pretty meaningless if we don’t extend greater educational choice. Knowing your school is failing doesn’t help if you are unable to take advantage of a better alternative.
Report this comment for abusive language, hate speech and profanity
Eric correctly points out that Calif. can’t do growth model without both data system and vertically aligned assessment, but I disagree with his conclusion that doing data system first is putting the cart before the horse. We have been mucking with the data system “forever” and it is still not there, so clearly it is the horse and not the cart — it only took 2-3 years to put STAR together in a reasonable shape and another couple of years before we had longitudinally comparable scaled scores. Which brings me to Eric’s second statement, that STAR is “stone age.” It is not, and as a year-end summative assessment it holds its own with any other state. Having few extra open-ended items thrown it, as many states do, may provide some good vibes to politicians but doesn’t provide much additional insight into student achievement. And we can reasonably easily make STAR vertically aligned once we decide we need to do it. Vertical alignment has little to do with whether we stay with California Standards, or move to Common Core Standards.
Report this comment for abusive language, hate speech and profanity
Well, guys, the inconvienent truth is that there are 3 elements needed to execute a reasonable “growth model” accountability system: (1) a data tracking system [not necessarily as fancy a CALPADS unless one wants a fancy growth model like value added), (2)vertically articulated assessment data (not necessarily vertical scales, tho less accurate methods such as recommended by ETS will yield less accurate growth model data -- Ze've is right on this element), and (3) a schema to deal with the mobility issue [growth models require multi-year data, preferably more than 2 years, which makes them extremely problematic for high mobility schools such as schools with high concentrations of socioeconomically disadvantaged students and/or English Learners; accountability systems based only on students continuously enrolled for multiple years will have credibility problems]. So, CALPADS is only one issue involved in solving the desire for accountability growth models. And on the individual shots fired below, Eric is a tad over-the-top with the Stone Age reference for STAR; STAR was conceived in a previous century (98) but not born until this century (03) but now suffering from lack of progress needed for better accountability data due to lack of leadership at both the CDE and SBE; and Ze’ve is mathematically deficient claiming only 2-3 years to develop STAR; E/LA was ready in 4 years, but the boys advising on Math were slow learners and that portion took 5 years to develop. On the main topic of John’s post, i.e., CALPADS flexibility, my main comment is that its development seems to be taking place in slow motion and patience is running out. Doug McRae, Retired Test Publisher, Monterey, CA
Report this comment for abusive language, hate speech and profanity
Your article is right on point, but misses a huge demographic of what will make CALPADS work, or fail – The End Users.
I work for Southern California LEA; I have a Bachelor in Computer Science, and a Masters in MIS, and can tell you the user interface for CALPADS is terrible. John you stated IBM’s B string coded CALPADS. I worked R & D for IBM in Tucson in the early 2000s; I’d say this was more like their D, or F string coders.
CALPADS has no continuity of format.
One of the simplest coding errors I found is that dates on two different screens are coded in two different formats. One will pre-fill the “/” between the numbers, while another won’t. Other areas want m/dd/yyyy while other screens want mm/dd/yyyy. This type of error is juvenile; I learned to clean up code issues like this in high school. This is minor to some of the other functionality issues CALPADS has.
Your article stated that service calls are down. Did you factor in how many End Users aren’t using CALPADS anymore. Most other LEAs I talk to are completely fed up with the system, and make only the most cursory attempts to use it.
Secondly not including the pre-mentioned lag or crashing of the system, CALPADS takes more steps, more screens, and exponentially longer complete a task vs. CSIS. A task that took me four or five steps in CSIS, now takes eight or nine in CALPADS. It now takes hours or days longer to process my LEA’s students through CALPADS. This doesn’t even take into account the functionality that hasn’t even been implemented in CALPADS yet. For some bigger LEAs, CALPADS will need to be worked as a full time position. Is there a single LEA in the state that could afford that right now?
If the changes aren’t made to make CALPADS more user friendly, and shorten the step necessary to complete tasks, LEAs will not use it. Usability is just as important as functionality.
Report this comment for abusive language, hate speech and profanity
The essential problem with CALPADS does not lie with its numerous technical flaws, and “fixing” them is simply bailing out a leaky boat at a slower rate than the incoming water. The true problem with CALPADS is that the project managers simply don’t understand how information systems of this type work. And that’s why it should (and likely will) be scrapped altogether.
CDE started CALPADS by determining what information they wanted, and proceeded to blithely require that information from school districts and county offices of education. They spent most of their energies focusing on code sets and resolving data anomalies, with no appreciation of how varied things are in the real world of student information. While this might seem like an intuitive and obvious way to approach the project, it is in fact, not at all how you develop a new information system that will rely on data from a motley assortment of student information systems that have already been in use for years or decades. They needed to assess the existing information environment first, and design things based first on what data are available (not just what they want), and they needed to talk to their customers. They assumed that all users of a particular student information software product would collect information in the same way, that school districts were not different from county offices of education, and that everyone could learn their impossibly complex set of rules if they could only schedule enough four-hour trainings and provide enough piles of technical documentation. CDE was wrong on all of these accounts.
A perfect example of the CDE’s cluelessness is their requirement that alternative education programs specify which school in California a student was “expected to attend” when they left. The only people who would like to know this more than CDE are the alternative education programs themselves, but they don’t know and can’t find out. County offices of education that operate schools in the juvenile detention facilities in California rarely know what school a student has come from, and generally don’t know that the child will be leaving until they are released by the probation department. It is especially infuriating to be required to provide information about where you “think” a student is going, when information about where they actually went is already available from within CALPADS itself – it’s the school that submits the next enrollment record for the student!
CALPADS needs to be scrapped and be re-built from the ground up. The only thing that comes close to working is the process for issuing student ID’s. No amount of money for training will bring this dying beast to life.
Report this comment for abusive language, hate speech and profanity
Currently I am the Director of Technology for a group of Charter Schools in the Fresno area and to be honest, I believe that CALPADS is a disaster. Its evolution from CSIS to CALPADS has not been an easy one and yet it is still crawling into LEAs throughout the state.
The amount of manpower that is being put into CALPADS at each LEA is removing good staff from their positions to babysit this beast. My Tech-coordinator spent the majority of his summer vacation attempting to meet a deadline that was inevitably pushed back. I myself spent a great amount of time away from family attempting to meet deadlines that are (thank God) pushed back. Spending late hours into early mornings to only get data returned with encrypted error messages that I have to locate in some 600 row spreadsheet to fix. Having to locate a CALPADS rep at another school to add the correct code in order for my data to be correct seems to be a little unfair at times. What about that LEA who has no one working on CALPADS or maybe someone that is incompetent and not dropping students correctly? To have one school dependent upon the work of another is a design for potential failure.
I have developed many databases and worked with some very good computer coding experts, and we all agree when we say that IBM has no idea about how schoolastic data flows. Data will not flow correctly, end users will input erroneous and at times false data just to meet a deadline. Some end users aren’t properly trained and are finding themselves “flung” into the world of CALPADS without so much as a memo to explain what they are doing. I foresee some schools entertaining the possibility of becoming private schools just so they do not have to report to CALPADs.
I myself am still working on my certification, and as a Technology Director could be working on more interactive things that increase test scores for our students or implementing plans that improve the quality of education for our schools. I honestly feel for larger LEAs that have a few thousand students to contend with. CALPADS requires a full time person to do it. I do not believe that any LEA has the funds to pay for that. If CALPADS died today I am sure that it would not be grieved by many if any.
Please correct me if I’m wrong, but being that everyone has a SSN, wouldn’t that work as a primary key for all data? Maybe allowing for encrypted usage of the Social Security number in the data system for transmission purposes would be the way to go. SSNs could be typed for searching; however they would only show as stars on screen. I understand the privacy issues; however, every school that a student attends will have that student’s SSN anyway. Students who do not have an SSN will need to have a SSID at that time. Those students without SSNs could possibly be the ones that need to be identified for migrant purposes or other reasons that go into political issues. Maybe I’m thinking to simple.
All in all, we have a lot of people working a lot of hours on CALPADS and a lot of children not being taught. So how are we NOT leaving children behind when all we are doing is data mining? I look forward to your responses. Please disregard any grammatical errors or spelling because I am very cross-eyed from looking at ERD’s, MID’s, and CCE’s … lol… Thanks for letting me vent.
Report this comment for abusive language, hate speech and profanity
Thanks for writing, Verles. I don’t know what I can add, other than it is disturbing to hear that CALPADS problems persist. I welcome more observations from those with experience using the system.
Report this comment for abusive language, hate speech and profanity
You are welcome John. After reading the others I realized that I am not alone in my observation of CALPADS. I am finding that as an LEA we are over stepping our boundries with the Hispanic population by telling many that they have to pick a race and an ethnicity to identify with. Many are wondering why being Hispanic is the only ethnicity that has to have additional data collected on them. The state of Arizona has provided fear of personal family issues for many California Hispanics. If we are not careful, we may see more and more Hispanic students being pulled out of public schools for fear of deportation of other family members based on suspicion of state and federal usage of CALPADS. I have had this very uncomfortable discussion with Hispanic parents in the past explaining that they must choose an “Ethnic group” if they are Latino. Within this group is anything that many want to choose so we as an LEA have to choose one for them. What is the educational world coming to when we have the right to tell someone what their ethnic group is? This is posing a major problem for LEA’s all over the state. Hispanics make up 47% of the population in California. If they begin pulling their students out of our schools then ADA (the driving financial force within any school) will drop substantially. Schools will close and the education system will topple. If this happens then what use would the data in CALPADS be for?
I believe in data driven schools. I think that data is essential however, it needs to be done in a more diverse manner. If each LEA is responsible for creating their own data collection with a guided format from the state then we may have more acurate information being distributed. One last point, if CBEDS has been working well (and from what I have seen in my 12+ years in education) why are we not attempting to improve the data collection through there instead of duplicating data collection in a new system? Of course I am a Jarhead and most Marines arent paid to think, just follow orders. lol…
Report this comment for abusive language, hate speech and profanity
As of December, 2010, Calpads is still a burden. It takes an average of 10 minutes per student to update records manually on the Calpads server. That includes time to search for the student by demographic record, navigate to the correct record set, navigate to the correct element within the record set, enter new data, post, validate, and search for the next student. Every page change involves delay, sometimes for minutes. Often times we are kicked off the system for “inactivity” while waiting for the server to churn to the next page. Of course, all data is lost then. We do submit our student records as a mass upload, but have any of you out there not had to go back and correct a significant number of anomalies and fatal file uploads? We are a small, rural school district. We do not have the time, money, or people to dedicate to this project without jeopardizing our mission–actually teaching students.
Report this comment for abusive language, hate speech and profanity