The insidious thing about tests is that they seem so straightforward. I write a bunch of questions. My students try to answer them. And so I find out who knows more and who knows less.

But if you have even a minimal knowledge of the field of psychometrics, you know that things are not so simple.

And if you lack that minimal knowledge, Howard Wainer would like a word with you.

Wainer is a psychometrician who spent many years at the Educational Testing Service and now works at the National Board of Medical Examiners. He describes himself as the kind of guy who shouts back at the television when he sees something to do with standardized testing that he regards as foolish. These one-way shouting matches occur with some regularity, and Wainer decided to record his thoughts more formally.

The result is an accessible book, Uneducated Guesses, explaining the source of his ire on 10 current topics in testing. They make for an interesting read for anyone with even minimal interest in the topic.

For example, consider the making of a standardized test like the SAT or ACT optional for college applicants, a practice that seems egalitarian and surely harmless. Officials at Bowdoin College have made the SAT optional since 1969. Wainer points out the drawback--useful information about the likelihood that students will succeed at Bowdoin is omitted.

Here's the analysis. Students who didn't submit SAT scores with their application nevertheless took the test. They just didn't submit their scores. Wainer finds that, not surprisingly, students who chose not to submit their scores did worse than those who did, by about 120 points.

Figure taken from Wainer's blog.

Wainer also finds that those who didn't submit their scores had worse GPAs in their freshman year, and by about the amount that one would predict, based on the lower scores.

So although one might reject the use of a standardized admissions tests out of some conviction, if the job of admissions officers at Bowdoin is to predict how students will fare there, they are leaving useful information on the table.

The practice does bring a different sort of advantage to Bowdoin, however. The apparent average SAT score of their students increases, and average SAT score is one factor in the quality rankings offered by US News and World Report.

In another fascinating chapter, Wainer offers a for-dummies guide to equating tests. In a nutshell, the problem is that one sometimes wants to compare scores on tests that use different items—for example, different versions of the SAT. As Wainer points out, if the tests have some identical items, you can use performance on those items as “anchors” for the comparison. Even so, the solution is not straightforward, and Wainer deftly takes the reader through some of the issues.

But what if there is very little overlap on the tests?

Wainer offers this analogy. In 1998, the Princeton High School football team was undefeated. In the same year, the Philadelphia Eagles won just three games. If we imagine each as a test-taker, the high school team got a perfect score, whereas the Eagles got just three items right. But the “tests” each faced contained very different questions and so they are not  comparable. If the two teams competed, there's not much doubt as to who would win.

The problem seems obvious when spelled out, yet one often hears calls for uses of tests that would entail such comparisons—for example, comparing how much kids learn in college, given that some major in music, some in civil engineering, and some in French.

And yes, the problem is the same when one contemplates comparing student learning in a high school science class and a high school English class as a way of evaluating their teachers. Wainer devotes a chapter to value-added measures. I won't go through his argument, but will merely telegraph it: he's not a fan.

In all, Uneducated Guesses is a fun read for policy wonks. The issues Wainer takes on are technical and controversial—they represent the intersection of an abstruse field of study and public policy. For that reason, the book can't be read as a definitive guide. But as a thoughtful starting point, the book is rare in its clarity and wisdom.

I get so many questions about learning styles that I added an FAQ to my website. You can find it here.

David Daniel and I have a letter in latest issue of Science. It's behind a paywall, so I thought I'd provide a summary of the substance.

David and I note that there is, in some quarters, a rush to replace conventional paper textbooks with electronic textbooks. It is especially noteworthy that members of the Obama administration are eager to speed this transition. (See this report.)

On the face of it, this transition is obvious: most people seem to like reading on their Nook, Kindle or iPad--certainly sales of the devices and of ebooks are booming. And electronic textbooks offer obvious advantages that traditional textbooks don't, most notably easy updates, and embedded features such as hyperlinks, video, and collaboration software.

But David and I urged more caution.

We should note that there are not many studies out there regarding the use of electronic textbooks, but those that exist show mixed results. A consistent finding is that, given the choice, students prefer traditional textbooks. That's true regardless of their experience with ebooks, so it's not because students are unfamiliar with them (Woody, Daniel & Baker, 2010). Further, some data indicate that reading electronic textbooks, although it leads to comparable comprehension, takes longer (e.g., Dillon, 1992; Woody et al, 2010).

Why don't students like electronic textbooks if they like ebooks? The two differ. Ebooks typically often have a narrative structure,  they are usually pretty easy to read, and we read them for pleasure. Textbooks in contrast, have a hierarchical structure, the material is difficult and unfamiliar, and we read them for learning and retention. Students likely interact with textbooks differently than books they read for pleasure.

That may be why the data for electronic books are more promising for early grades. Elementary reading books tend of have a narrative structure, and students are not asked to study from the books as older kids are.

Further, many publishers are not showing a lot of foresight in how they integrate video and other features in the electronic textbooks. A decade of research (much of it by Rich Mayer and his collaborators and students) show that multimedia learning is more complex than one would think. Videos, illustrative simulations, hyperlinked definitions--all these can aid comprehension OR hurt comprehension, depending on sometimes subtle differences in how they are placed in the text, the specifics of the visuals, the individual abilities of readers, and so on.

None of this is to say that electronic textbooks are a bad thing, or indeed to deny that they ought to replace traditional textbooks. But two points ought to be kept in mind.
(1) The great success of ebooks as simply the porting over of traditional books into another format may not translate to electronic textbooks. Textbooks have different content, different structure, and they are read for different purposes.
(2) Electronic textbooks stand a much higher chance of success if publishers will exploit the rich research literature on multimedia learning, but most are not doing so.

For these two reasons, it's too early to pick the flag and shout "Hurrah!" on electronic textbooks.


A. Dillon, Ergonomics 35, 1297 (1992).
W. D. Woody, D. B. Daniel, C. Baker , Comput. Educ. 55, 945 (2010)