To read test-maker NCS Pearson Inc.’s contract with New York State, the company all but bullet-proofed its new tests for grades 3 to 8. Pearson promised each question would seamlessly align with the Common Core Learning Standards and be subject to multiple reviews by batteries of experts before it ever appeared before students.

Yet Pearson’s assurances don’t square with reports from hundreds of teachers and administrators that this year’s questions, especially on the ELA test, emphasized irrelevant details, were at the wrong level for the grade, and had ambiguous answers. And how did last year’s nonsensical “Hare and the Pineapple” passage get through such a vetting process?

“We could not believe how bad the tests were this year,” Principal Elizabeth Phillips of PS 321 in Brooklyn told a panel at New York University shortly after the tests were administered. “If anyone were to teach to this test, we’d wind up with illiterate children.”

 

A Rube Goldberg process

In its five-year, $32.8 million contract with New York State to write new Common Core tests, Pearson agreed to create more than 2,500 “items” (questions) using independent professional item writers, supervised by content experts and psychometricians — the statisticians who manage the technical details.

Under the contract terms, each item is reviewed by grade and subject specialists for factual accuracy, clarity and support of the standards. Then the items are checked for potential bias or “sensitivity” (insulting or favoring one particular group of students).

Statistical tests verify that the items cover a range of difficulty and track previous tests. Next, the items are submitted to a State Education Department review committee; then they are field-tested; next they go back to Pearson; then Pearson hires teams of teachers to review them; and finally Pearson reviews them again.

Scott Marion, the associate director of the National Center for Improvement of Educational Assessment, said a diagram he made of an item’s development “looked like a Rube Goldberg flow chart.” But he said that Pearson, which has about 39 percent of the K-12 testing market, has been constructing tests for a long time and knows what it is doing.

The various review committees are tough, Marion said. “They reject a lot of items.” He has heard the complaints, he said, but any problems will be worked out in time. Everyone is facing new demands with the Common Core, but “we sort of know how to do this.”

 

Don’t assume it’s hunky-dory

So were Phillips and her teachers just flummoxed by a harder test? Phillips leads a school of high performers in middle-class Park Slope whose students will probably score high. Test protesters went beyond just ELA teachers. They included parents, science teachers, librarians and literacy coaches, to name a few, from many parts of the city. The State Education Department forbade teachers from revealing specific questions, but staffers who proctored the tests in high- and low-performing schools agreed they were poor. A month later, teachers said the math tests given in late April were “fair” and “friendlier to students,” so this was not a general anti-testing crusade.

Board of Regents member Kathleen Cashin, a former city superintendent, took the complaints seriously. “I think it’s very important not to just assume everything is hunky-dory with the tests,” she told Capital Pro on April 28. “The professionals in the field last year and again this year objected very vigorously to the tests.”

 

Building the airplane while flying it

So what went wrong? The first thing to consider is the enormity of the task. Pearson is essentially creating tests out of whole cloth.

The Common Core standards themselves are still sketchy. As former State Education Commissioner David Steiner wrote recently on an education think tank blog, “The challenge is that the [ELA] Standards themselves do not require specific content beyond classical mythology, one (any) play by Shakespeare, and a selection of founding American documents.”

Steiner is a supporter of the Common Core, but called the exams “patronizing” for their elimination of rich, authentic passages. In fact, Pearson says only half of its ELA passages are drawn from literature. The other half the item writers make up.

New York insisted on rolling out Common Core tests early, before almost any other state. It could have waited and used tests developed by a larger consortium, but instead it contracted with Pearson for state-specific exams.

The state spent $32 million on these exams, which sounds like a lot of money, but evidence suggests it wasn’t enough. The reason teachers cannot reveal the questions is that items are expensive and have to be reused. The tests were so long this year in part because fully one-third of the questions were actually being field-tested for next year’s test. The Common Core item bank is thin because the standards are new. In building their new tests, which will be marketed in several states, Pearson was using our students’ and teachers’ time.

The United States spends just one-quarter of 1 percent of its annual K-12 budget on test development, a very low $20 to $30 per student considering how much the tests matter, according to Brookings Institution scholar Matthew Chingos. New York’s spending actually comes in dead last among the states. Did New York State invest the money needed to develop excellent tests?

“Really bad tests can meet psychometric qualifications,” says Chingos, and that may be part of the answer to New York City’s ELA debacle. There was not a lack of process in developing the tests so much as a profound lack of oversight. The State Education Department rushed something that needed care and time. It has not fully articulated the standards, and there isn’t yet rich curriculum.

The failure of the tests could be a tail wagging a much bigger dog: the state’s failure to manage the transition to the Common Core standards. The ones who are paying the price are the students and teachers.