The Reading First Study - Problems
The May 2008 Interim Report on Reading First Has Numerous Problems
It may not matter what conclusions are in the recently publish Reading First Impact Study. A number of flaws have been pointed out in the study's design.
The recently publish Reading First Impact Study has attracted a lot of attention and been used to show that Reading First is ineffective. Whether or not the study itself, the Reading First Impact Study: Interim Report, supports those conclusions may not actually matter for reasons that are both scientific and political.
The Schools in the Study
The first reason that the study's conclusions may not mean much is that the sample of schools in the study was not chosen randomly. That lack of randomization makes it difficult to say that the 240 or so schools that got looked at in this Impact Study are actually representative of the 5,880 schools that have been awards reading First funds. If the schools in the study are not representative of the larger body of reading First schools, the study's findings are useless. The scientific community will look at what the study uncovered about these particular 240 schools and basically say, "So what?"
Measuring Teacher Behavior
The Reading First Impact Study looked at two main things. One of those things was teacher behavior in the classroom. Specifically, the study wanted to see if teachers at Reading First schools spent more time on the five components of reading defined in the No Child Left Behind law - phonemic awareness, phonics, fluency, vocabulary, and comprehension. To do this the study team observed teachers in the classroom. They used an interval method of recording teacher behavior, breaking class periods down into three minute intervals and recording whether or not a teacher touched on one of the five components of reading during each three minute interval.
The problem with this method is twofold. First, three minutes is a long time. If you don't believe that, try getting a first grader to be completely quiet for three minutes. Because the interval is fairly long, some teachers touched on one (or more) of the five components of reading several times in a three minute interval while other teachers may have only touched on one of the five components a single time. But those two teachers received the same score in the observations.
In short, many in the academic community are concerned that the method used for observing teacher behavior is flawed because it doesn't distinguish between teachers who spend a lot of instructional time on the five components of reading and teachers who manage to just touch on some component periodically.
Measuring Student Response to Instruction
The second main thing that the Reading First Impact Study looked at was reading comprehension in students at reading First schools. And the study has been criticized partly because that was the only one of the five components of reading that the study reported on in students. Especially in first and second grade, some assessment of phonics or fluency in reading would have been valuable. But the study limited itself to reading comprehension. Many in the academic community were disappointed with that and they question whether the study fulfills the Congressional mandate that funded it.
Political Considerations
The final reason that the study's conclusions may not matter is that reading First's funding has already been cut considerably - from about $1 billion in the current fiscal year to $393 million for the coming fiscal year. With the controversy surrounding this study, Congress may cut Reading First funding even further - in which case the program's impact could become irrelevant.
