A few weeks ago I published an op-ed in the NY Daily News about how purchasing decisions are made in educational technology. In this blog I want to elaborate on a couple of points I made.
First, regarding the overall point of the op-ed: most people got it, but a few took it to be a criticism of the use of technology in schools, and wrote me irate emails, tweets, or blog posts. The op-ed was a criticism of how tech purchasing decisions are made, not a blanket criticism of ed tech. "Ed tech" is such a broad category (like "explicit instruction" or "progressive pedagogy") it seems improbable you're going to be able to draw conclusions about effectiveness that will apply in most cases. Sure, you can do meta-analyses (Hattie has the effect size of tech interventions at around 0.3, I believe) but there will be so many exceptions and caveats, it may not be worth it.
My point in the op-ed was that data to guide purchasing decisions seldom exist. Decisions must be based on intuition, and intuition has in the past been a poor guide.
Second, op-eds do not provide space to get much into evidence, so I provide some citations here, specifically for the three reasonable-sounding intuitions that I said were wrong.
Intuition 1: reading on screen and paper will be equivalent. I claimed that reading comprehension consistently takes a small hit on screen vs. paper. I just published a piece over at Larry Felazzo's blog that cites lots of the data on this point.
Intuition 2: information is more readily accessible, so students don't need to have as much information in their memory. I claimed that the Internet has not changed the need to have information in ones head. There's a great deal to say about this claim, and other reasons I listed ended up edited out of the piece; I've described these reasons in magazine pieces, books (here and here), blog posts, and a video. One claim that remained in the op-ed was that people can look stuff up, but they don't. Once the number of words you don't know hits two percent, the odds go way up that readers find the text difficult and will quit.
The 2% figure comes from Carver (1994), who called texts with 0% unknown words "easy," 1% unknown words "appropriate" and 2% or more "difficult." Schmitt et al (2011), studying second language learners, argued that there is (unsurprisingly) a linear relationship between known vocabulary and comprehension, and that 98% is a reasonable middle ground for what constitutes appropriate challenge.
I touched only briefly on the fact that "looking things up" is trickier than it sounds, and usually requires a fair bit of knowledge to get right, a point made forcefully by George Miller in 1987. As he points out, meaning comes from context, and dictionaries can't provide much. So children look up a word like "relegate," take away an abbreviated, context-free understanding of the definition, like "send away," and so misuse the word, e.g., "I relegated the letter to my pen pal."
It's not that "looking things up" isn't useful. It's that people not doing the looking-up underestimate the extent to which lookers-up will view it as mental work, and overestimate the likelihood that an accurate definition will be learned.
Intuition 3: kids don't need to worry about handwriting, because keyboarding is so important. Finally, I mentioned evidence that handwriting may improve cognition: recent papers include Cameron et al 2012, Grissmer et al 2010; Dinehart & Manfra, 2013. These are correlational findings, but the effect size is pretty big, and the weird thing is that what seems to be the best predictor of academic ability along these lines is how well children can copy a figure; look at a complex model, and then try to reproduce it on paper. That's pretty much exactly what handwriting practice is. The effect may not be causal, but before we jettison a year of practicing something that might aid cognition substantially, we should be sure it's disposable.
My real motivation in writing this op-ed was frequent encounters with teachers, who feel that ed-tech interventions are held to a lower standard of promise when administrators evaluate them. Teachers who are doubtful about a new intervention are brushed off as fuddy-duddies, or frightened of change.
I think there's something to this. I don't think it's that ed tech enthusiasts are dazzled by the shiny and new. But I do think this business about intuition is a problem. It often seems obvious how the human cognitive system will interact with a new situation, but these intuitions can easily mislead.
First, regarding the overall point of the op-ed: most people got it, but a few took it to be a criticism of the use of technology in schools, and wrote me irate emails, tweets, or blog posts. The op-ed was a criticism of how tech purchasing decisions are made, not a blanket criticism of ed tech. "Ed tech" is such a broad category (like "explicit instruction" or "progressive pedagogy") it seems improbable you're going to be able to draw conclusions about effectiveness that will apply in most cases. Sure, you can do meta-analyses (Hattie has the effect size of tech interventions at around 0.3, I believe) but there will be so many exceptions and caveats, it may not be worth it.
My point in the op-ed was that data to guide purchasing decisions seldom exist. Decisions must be based on intuition, and intuition has in the past been a poor guide.
Second, op-eds do not provide space to get much into evidence, so I provide some citations here, specifically for the three reasonable-sounding intuitions that I said were wrong.
Intuition 1: reading on screen and paper will be equivalent. I claimed that reading comprehension consistently takes a small hit on screen vs. paper. I just published a piece over at Larry Felazzo's blog that cites lots of the data on this point.
Intuition 2: information is more readily accessible, so students don't need to have as much information in their memory. I claimed that the Internet has not changed the need to have information in ones head. There's a great deal to say about this claim, and other reasons I listed ended up edited out of the piece; I've described these reasons in magazine pieces, books (here and here), blog posts, and a video. One claim that remained in the op-ed was that people can look stuff up, but they don't. Once the number of words you don't know hits two percent, the odds go way up that readers find the text difficult and will quit.
The 2% figure comes from Carver (1994), who called texts with 0% unknown words "easy," 1% unknown words "appropriate" and 2% or more "difficult." Schmitt et al (2011), studying second language learners, argued that there is (unsurprisingly) a linear relationship between known vocabulary and comprehension, and that 98% is a reasonable middle ground for what constitutes appropriate challenge.
I touched only briefly on the fact that "looking things up" is trickier than it sounds, and usually requires a fair bit of knowledge to get right, a point made forcefully by George Miller in 1987. As he points out, meaning comes from context, and dictionaries can't provide much. So children look up a word like "relegate," take away an abbreviated, context-free understanding of the definition, like "send away," and so misuse the word, e.g., "I relegated the letter to my pen pal."
It's not that "looking things up" isn't useful. It's that people not doing the looking-up underestimate the extent to which lookers-up will view it as mental work, and overestimate the likelihood that an accurate definition will be learned.
Intuition 3: kids don't need to worry about handwriting, because keyboarding is so important. Finally, I mentioned evidence that handwriting may improve cognition: recent papers include Cameron et al 2012, Grissmer et al 2010; Dinehart & Manfra, 2013. These are correlational findings, but the effect size is pretty big, and the weird thing is that what seems to be the best predictor of academic ability along these lines is how well children can copy a figure; look at a complex model, and then try to reproduce it on paper. That's pretty much exactly what handwriting practice is. The effect may not be causal, but before we jettison a year of practicing something that might aid cognition substantially, we should be sure it's disposable.
My real motivation in writing this op-ed was frequent encounters with teachers, who feel that ed-tech interventions are held to a lower standard of promise when administrators evaluate them. Teachers who are doubtful about a new intervention are brushed off as fuddy-duddies, or frightened of change.
I think there's something to this. I don't think it's that ed tech enthusiasts are dazzled by the shiny and new. But I do think this business about intuition is a problem. It often seems obvious how the human cognitive system will interact with a new situation, but these intuitions can easily mislead.
RSS Feed