This column first appeared at RealClearEducation.com on June 5, 2014.

We normally expect that a kindergarten classroom will be decorated with colorful posters, maps, and student artwork. But a recent study indicated that trimming out the classroom may distract students, and compromise learning.

Headlines in the press have been not equivocated. The Telegraph, Tampa Bay Times, and Smithsonian all ran stories along the lines of “wall displays distract from learning,” while the Daily Mail asked whether they should be banned altogether. My hunch is that most kindergarten teachers and parents of kindergarteners would not agree, but it’s worth looking at why this conclusion would be hasty.

The study confirms something you likely find intuitive; if there are bright, colorful things on the wall, kids will look at them, and will do so at times that the teacher hopes their attention will be focused on something else. Hence, kids might learn less about the lesson plan in such an environment.

The study (Fisher et al, 2014) took place in a laboratory reconfigured to look like a classroom. Kids first visited on five occasions to get to know the “teacher” (i.e., researcher) and to acquaint them with the paper-and-pencil measures that would be used in the experiment. Over the following two weeks, kids listened to six read-alouds on science topics (e.g., volcanoes, the solar system) in groups. Afterwards, children answered six questions about the lesson.

During the familiarization sessions, there were a moderate number of decorations on the walls. During the experimental sessions, the walls were either bare, or quite covered.

There were two outcomes of interest: first, how kids scored on the test in each environment, and second, the frequency with which kids were distracted. Children were videotaped during the story, and they were deemed to be thinking about something else if their eyes were not on the teacher.

The results were as you would expect. Kids spent a greater percentage of time looking away from the teacher in the decorated classroom (38% of time vs. 28% of time). And kids in the decorated classroom scored lower on the assessment (42% vs. 55%).

I can see two important reasons to be cautious when interpreting these results.

First, it seems likely to me that kids would habituate to the wall decorations in time. The authors address that possibility in their paper, citing data presented at a conference (but not yet published) showing that kids distraction was above baseline levels two weeks after the introduction of a distracting classroom environment. But the abstract of the talk also says that there was significantly less distraction in week 2 than in week 1. In other words, kids were getting used to the busy environment, and might have returned to baseline soon.

Second, even if we accept that classroom decoration brings a cost to learning, we should remember that teachers have other reasons for brightening their rooms; they want the classroom to be inviting, to feel like a social environment. Would it be more difficult to build a sense of classroom community in the sterile environment? That’s at least plausible. In other words, there may be a cost to outcomes that were not evaluated in this study.

Or to put it another way, the sterile environment is the wrong control condition. How about comparing the busy environment with something that looks homier, but less stimulating:  a serene painting or two on each wall, perhaps a vase of dried flowers on a table, that sort of thing.

I think there’s a useful point for lower elementary teachers to consider here. As the lead researcher mentions in a video about the study, it seems ironic that classrooms become less decorated as kids get older, given that it’s the younger kids who have more difficulty regulating attention. But I don’t think bare walls are the way to go.

Fisher, A. V., Godwin, K. E., & Seltman, H. (2014). Visual Environment, Attention Allocation, and Learning in Young Children When Too Much of a Good Thing May Be Bad. Psychological science, 0956797614533801.

This column originally appeared at RealClearEducation on May 27, 2014.
Is college worth the high cost? After all, everyone knows a college degree doesn’t guarantee a good job—English majors all work at McDonalds, right? This lore is gaining acceptance (see discussion of the “college bubble” here and here, for example) but a new review in Science (Autor, 2014) marshals data suggesting that it’s dead wrong. 

College graduates not only continue to make more than non-graduates, the education-wage premium is increasing. It has risen in most advanced economies, and nowhere more than the US. In fact the difference between the median incomes of high school graduates and college graduates in 1979 was about $17,400 (in 2012 dollars). In 2012 it had doubled to around $35,000. (Autor doesn’t cite these data, but a 2013 OECD report also suggested that unemployment rose much less for college graduates during the recent recession.)

The income difference is not an artifact of the wild increases among the top 1% of earners. As Autor emphasizes, wage inequality has increased across the spectrum of incomes and is well represented among the “other 99%.” He cites various studies suggesting that 60% or so of the increasing difference in US wages is due to the increase in the education-wage premium. Other factors contributing to the difference include the decline (in real dollars) of income for high school graduates, the declining influence of trade unions, the rise in automation and subsequent loss of jobs calling for minimal cognitive skills, and competition for those jobs in the developing world.

Autor makes the case that these seemingly disparate factors can come together into a relatively simple model of supply and demand. Since the 1980’s the number of people getting a college education has increased, but the rate of increase has slowed, relative to earlier decades. Hence, the supply of people with the cognitive skill set afforded by education was expanding, but at a slow pace. Meanwhile, the percentage of jobs requiring that skill set was increasing at a much faster pace, as low skill jobs were exported or automated. The market responded to supply and demand: wages for college graduates skyrocketed because there were not enough of them; wages for high school graduates declined (in real dollars) because there was a glut of such workers, relative to the need.

As Richard Murnane (who often collaborates with Autor) has argued for decades, there is every reason to think that these employment trends will continue; jobs that require modest cognitive skill yet pay living wages will not return to the United States economy.

So yes, college is worth it—or better, obtaining cognitive skill that is demonstrable to employers is worth it. The promise of obtaining them outside of the usual educational route (via MOOCs, for example) is still a work in progress.

For educators and education researchers, this conclusion doesn’t change much of what we’ve been trying to do, as we already believed in the importance of education for individual economic promise. Autor’s analysis offers data should we care to take on skeptics, and adds urgency (if any was needed) to the work.

This column was originally published on RealClearEducation.com on May 13, 2014.

Last week I discussed the case Carl Wieman made that education research and physics have more in common than people commonly appreciate. I mentioned in passing Wieman’s suggestion that random control trials—often referred to as the “gold standard” of evidence—are not the be-all and end-all of research, and that qualitative research contributes too. But I didn’t say how it contributes, and neither did Wieman (at least in the paper I discussed). Here, I offer a suggestion.

Qualitative research is usually contrasted with quantitative research. In quantitative research the dependent variable—that is, the outcome—is measured as a quantity, usually using a measurement that is purportedly objective. In qualitative research, the outcome is not measured as a quantity, but as a quality.

For example, suppose I’m curious to know what students think about their school’s use of technology. In a quantitative study, I might develop a questionnaire that solicits student’s opinions about, say, the math and reading software they’ve been using, and also asks students to compare them to the paper versions they used last year. I’d probably try to get most or all of the students in the school to respond. I would end up with ratings which I can treat as quantitative data.  

Or I could do a qualitative study, in which I use focus groups to solicit their opinions. The result of this research is not numeric ratings, but transcripts of what people said. I’d try to find common themes in what was said. Because focus groups are time-consuming, I’d probably talk to just a handful of students.

People who criticize qualitative research treat it as a weak version of quantitative methods. For example, a critic would point out that you can’t trust the results of the focus groups, because I might have, by chance, interviewed students with idiosyncratic views. Another problem is that focus groups are not very objective; responses are a product of the dynamic between the interviewer and the subjects, and among the subjects themselves.

These complaints are valid, or would be if we hoped to treat the focus group results the way we treat the outcomes of other experiments. I suggest we should not.

Here’s a simple graphic I used in a book to describe how science works. We begin with observations about the world. These can come from our casual, everyday observations or from previous experiments. We then try to synthesize those observations into a theory; we try to find simplifying rules that describe the many observations we’ve made. This theory can be used to generate a prediction, something we haven’t yet observed, but ought to. Then we test the prediction, and that test leads to a new observation, a new fact about the world. And we continue around the circle, always seeking to refine the theory.

The criticism of qualitative research is that it does not provide a very reliable test of a theory. But I think it’s better viewed as providing a new observation of the world.

An advantage of qualitative over quantitative data is the flexibility in how its collected. If I want to know what students think of the new technology instruction and I create a quantitative scale to measure it, I’m really guessing that I know the important dimensions of student opinion. I may try to capture their views on effectiveness and ease-of-use, for example, but what students really care about is the fact that so many websites are blocked.

Qualitative studies allow the subject to tell you what he thinks is important. For example in this qualitative study, students suggested that the schools policy on blocking websites affected their relationships with teachers—they felt untrusted. Maybe that’s the kind of thing a researcher would have been looking for, but I doubt it.

In addition, qualitative data tend to be richer. I’m letting the subject describe in his or her own words what she thinks, rather than asking her to select from reactions that I’ve framed.

Naturally, either type of research—qualitative or quantitative--can be poorly conducted. Each has its own rules of the road. It’s important to judge the quality of research by its own set of best-practice rules, and to judge it by how well it fulfils the purpose to which it is suited.

That’s why I believe qualitative research has an undeserved bad reputation. It is judged by standards of quantitative research, and deemed unable to serve purposes that ought to be served by quantitative research. But I agree with Wieman that qualitative research, well-conducted, makes a valuable contribution, one to which quantitative research is ill-suited.

I have a new article (with Sian Beilock) on math anxiety in the new American Educator. Free download here:
http://www.aft.org/pdfs/americaneducator/summer2014/Beilock.pdf

This post first appeared at RealClearEducation.com on May 6, 2014

As someone who spends most of their time thinking about the application of scientific findings to education, I encounter plenty of opinions about the scientific status of such efforts. Almost always, a comparison is made between the rigor of “hard” sciences and education. What varies is the accompanying judgment: derision for education researchers or despondency about the difficulty of the enterprise. In a recent article, physicist Carl Wieman (2014) offers a different perspective on the issue, suggesting that the difference between research in education and in physics is smaller than you might guess.

In education, Wieman is best known for refining and popularizing techniques to have college students better engage in large lecture courses. He started his career as a physicist, producing work that culminated in a Nobel prize. So he has some credentials in talking about the work of “hard” scientists.

Wieman begins by making clear what he takes to be the outcome of good science: predictive power. Can you use the results of your research to predict with some accuracy what will happen in a new situation? A common mistake is to believe that in education one ought to be able to predict outcomes for individual students; not necessarily so, any more than a physicist must be able to predict the behavior of each atom. Prediction in aggregate—a liter of gas or a school of children—is still an advance.

Wieman’s other points follow from his strong emphasis on prediction.

First, it follows that “rigorous” methods are any that contribute to better prediction. You don’t state in the absolute that randomized controlled trials are better than qualitative research. They provide different types of information in a larger effort to allow prediction.

Second, the emphasis on prediction frames the way one thinks about the messiness of research. Education research is often portrayed as inherently messy because there are so many variables at play. Physics research, in contrast, is portrayed as better controlled and more precise because there are many fewer variables that matter. Wieman argues this view is a misconception.

Physics seems tidy because you’ve probably only studied physics in textbooks, where everything is worked out: in other words, where no extraneous variables are discussed as possibly mattering. When the work that you study was first being conducted, it was plenty messy: false leads were pursued, ideas that (in retrospect) are self-contradictory were taken seriously, and so on. The same is true today: the frontiers of physics research is messy. “Messy” means that you don’t have a very good idea of which variables are important in gaining the predictive power that characterizes good science.

Third, Wieman suggests that bad research is the same in physics and education. Research is bad when the researcher has failed to account for factors that, based on prior research, he or she should have known to include. There is plenty of bad research in the hard sciences. People aren’t stupid; it’s just that science is hard.

I agree with Wieman that differences in “hardness” are mostly illusory. (That’s why I’ve been putting the term in quotation marks.) The fundamentals of the scientific method don’t differ much wherever they are applied. I also agree that people (usually people uninvolved in research) are too quick to conclude that, compared to other fields, a higher proportion of education research is low-quality. Come to a meeting of the Society for Neuroscience and I’ll show you plenty of studies that were poorly conceived, poorly controlled, or were simply wheel-spinning and will be ignored.

Wieman does ignore a difference between physics and education that I take to have important consequences: physics (and other basic sciences) strive to describe the world as it is, and so strive to be value-neutral. Education is an applied science; it is in the business of changing the world, making it more like it ought to be. As such, education is inevitably saturated with values.

Education policy would, I think, benefit, with a greater focus on the true differences between education and other varieties or research, and a reduced focus on the phantom differences in rigor.

Reference:

Wieman, C. E. (2014). The similarities between research in education and research in the hard sciences. Educational Researcher, 43, 12-14.  

This article first appeared at RealClearEducation.com on April 29, 2014.


Do children learn to read by translating letters into sound, or by perceiving the spelling of the word? The answer has an indirect bearing on teaching; it would presumably be best to instruct kids in a way consonant with how most perform the task. The last fifteen years has seen an increasing consensus among researchers: children initially learn via the letter-sound translation mechanism. As they gain reading practice, they acquire the spelling mechanism as well, although the letter-sound translation method continues to make a contribution to reading. Now a new study of 284 French children in grades 1 through 5 offers support to this model (Ziegler et al, 2014).

From the child’s perspective, the experimental task was simple. They sat before a computer screen. An asterisk appeared at the center for one second, and then a string of letters replaced the asterisk (I’ll call this the “response letter string.”) Children were to push one button if the response letter string formed a word, and another if it did not.

What the kids were not told was that another letter string actually appeared between the asterisk’s disappearance and the appearance of the response letter string. This letter string (called the prime) appeared for just .07 seconds, so the children didn’t consciously see it—if anything, they might have thought the screen flickered.

Even though you’re unaware of it, the prime can influence your response. If the prime is “MOP” and then the response letter string is also “MOP,” you’re faster to verify that “MOP” is indeed a word, compared to how fast you respond if the prime were a non-matching word, say, “DOG.” Even though you were unaware of the prime, you read it, and so you’re a bit faster to read it a second time.

But what if the prime were “MAWP?” Would you still be faster to verify that “MOP?” is a word when it appears? If you think that people read via letter-sound translation, the answer ought to be yes. When “MAWP” appears, you read it and generate the right sound, and if sound is the basis of reading, you should get the advantage when “MOP” appears.

Using a comparable method, the researchers tested whether kids read via spelling. They used a prime with nearly the same spelling as the response letter string: for example “TALBE” followed by “TABLE.” Other work has shown the readers are pretty resistant to spelling errors like this one, where letters are off by just one position (McCusker et al, 1981). So if you’re using the spelling of a word to identify it, we can expect that you’ll be faster to verify that “TABLE” is a word if the prime was “TALBE,” compared to a prime like “CAIRH.” 

So take a moment and guess. Do first graders read mostly by sound or by spelling? How about fifth graders?

The data indicated that first graders read by sound. With each successive year, kids showed more and more evidence of using the spelling of words in their reading. BUT there was no diminution of the influence of sound. Experienced readers use both the sound and the spelling mechanisms.

This result fits with the following view of reading: most kids will learn to read by learning to sound out words. With practice over the course months and years, they develop an increasing number (and increasingly robust) mental representations that allow them to identify words by their appearance, i.e., by their spelling. These representations form as a consequence of reading practice and don’t require any special instruction. This general view accords with other behavioral data showing that methods of reading instruction that emphasize phonics have an edge over other methods.

References:

McCusker, L. X., Gough, P. B. & Bias, R. G. (1981). Word recognition inside out and outside in. Journal of Experimental Psychology: Human Perception and Performance, 7, 538-551.

Ziegler, J. C., Bertrand, D., Lété, B., & Grainger, J. (2014). Orthographic and phonological contributions to reading development: Tracking developmental trajectories using masked priming. Developmental Psychology, 50, 1026-1036.

This article first appeared on the RealClearEducation website on April 22, 2014.

Please note: A few people have told me that they have trouble finding my column on RealClearEducation.com. If you would like an email notification when I post a new column, please just email me, and I'll ping you when I post.


You often hear the phrase that small children are sponges, that they constantly learn. This sentiment is sometimes expressed in a way that makes it sound like the particulars don’t matter that much; as long as there is a lot to be learned in the environment, the child will learn it. A new study shows that for one core type of learning, it’s more complicated. Kids don’t learn important information that’s right in front of them, unless an adult is actively teaching them. 

The core type of learning is categorization. Understanding that objects can be categorized is essential for kids’ thinking. Kids constantly encounter novel objects; for example, each apple they see is an apple they’ve never encountered before. The child cannot experiment with each new object to figure out its properties; she must benefit from her prior experience with other apples, so that she can know, for example, that this object, since it’s an apple, must be edible.

But how can a child tell which properties of the apple are incidental (e.g., it has a long stem) and which properties are true of all apples (it’s edible)?  The child must ignore many incidental properties, and hold on to the important properties that are true of all apples.

Previous research shows that by age three or four, children are sensitive to linguistic cues about this matter. They appropriately attach significance to the difference between “This apple has a long stem” and “Apples are good for eating.” “This apple” signifies that the information provided applies only to this particular apple; “Apples” indicates a generalization about all apples.

A recent study (Butler & Markman, 2014) examined whether there are cues outside of language that guide children in resolving this problem. The researchers tested the possibility that children are sensitive to adults teaching them; if an adult deliberately highlights a property for the child’s benefit, that presumably is a property of some importance, and one that is characteristic  of all objects of this sort.

Children (aged 4-5) were shown a novel object and were told that it was a “spoodle.” There was a brief test to be sure that the child got the name right, and then another, irrelevant task. The experimenter began to clean the materials from this other task, and this is when the special property of the spoodle came into play; spoodles are magnetic.

In the pedagogical condition, the experimenter said “Look, watch this” and used the spoodle to pick up paperclips. In the intentional condition, the experimenter used the spoodle to pick up paperclips, but did not request the child’s attention or make eye contact. In the accidental condition, the experimenter feigned accidentally dropping the spoodle on the clips. In all of the conditions, the experimenter held the spoodle with the paper clips clinging to it and said “wow!”

Next, the child was presented 16 objects and was asked to say which were spoodles. Half were identical to the original spoodle, and half were another color. In addition, half of each color were magnetic and half were not.

So the question is which property kids think makes an object a spoodle: appearance (i.e., color) or function (i.e., magnetism). The data are shown here:

These children were clearly quite sensitive to the non-verbal teaching. Recall that in the Intentional condition, the adult used the spoodle’s function purposefully. The child could easily infer that magnetism is an important property of spoodles. But the children didn’t. Appearance made a spoodle a spoodle for these kids.

Yet when the adult did the exact same thing, but also made plain to the child her actions were for the child’s benefit, that she was teaching, then the child understood that magnetism held special significance for spoodle-hood.

I think this study has an interesting implication for differences in kids’ preparedness for schooling, associated with their home environment. We tend to focus on differences in the richness of experiences available to kids. That’s important, but this experiment provides a concrete example of small differences in parenting may have important consequences for children’s learning. “Little sponges” don’t learn certain types of information, even in a rich environment. They have to be taught.

Reference:

Butler, L. P., & Markman, E. M. (2014). Preschoolers use pedagogical cues to guide radical reorganization of category knowledge. Cognition, 130,  116-127.

This piece first appeared on RealClearEducation.com on April 16, 2014.
If you would like an email notification when I post a new column at RealClear, send me an email, and I'll add you to my list.

A recent article in the Washington Post sounds a warning klaxon for our ability to read deeply. You’ve probably heard this argument elsewhere, made most forcefully by Nick Carr in the The Shallows: frequent users of the Web (i.e., most of us) are so in the habit of skittering from page to page, scanning for juicy bits of information but not really reading, that they have lost the ability to sit down and read prose from start to finish. I think the suggestion is probably wrong.

The first thing to make clear is that anyone who comments on this issue (including me) is guessing. There are simply not any data that address it directly. We might predict, for example, that scores on standardized reading tests would have dropped in the last fifteen years or so (they haven’t) but such data are hardly definitive, as reading comprehension test scores are a product of many factors.

The Post article cites studies comparing reading on paper versus reading on screens, but that won’t address the issue, which concerns the long-term consequences of a particular type of reading. The Post also incorrectly says that paper is superior. Most studies indicate no difference between screens and paper for pleasure reading. For textbook reading, students take longer to read on screens, although comprehension is about the same. (Daniel & Willingham, 2012).

The article, like all the pieces I’ve seen on this topic, is short on data and long on individual’s impressions. For example, teachers aver that students can no longer read long novels. Well, if we’re swapping stories, I (and most of my classmates) had a hard time with Faulkner and Joyce back in the early ‘80’s, when I was an English major.

The truth is probably that the brain is simply not adaptable enough for such a radical change. Yes, the brain changes as a consequence of experience, but there are likely limits to this change, a point made by both Steve Pinker and Roger Schank when commenting on this issue. If our ability to deploy attention or to comprehend language processes were to undergo substantial change, the consequences would cascade through the entire cognitive system, and so the brain is probably too conservative for large-scale change.

For example, there’s a lot of overlap in the processes of reading and the processes used for understanding spoken speech—processes that assign syntactic roles to words. Do we see any evidence that people are having a harder time understanding spoken language? Or does the problem lie in the mental processes that build understanding of larger blocks of language, as when we’re comprehending a story? If so, habitual Web users should have a hard time understanding complex narratives not just when they read, but in television and movies. No one should have watched The Sopranos, with its complicated, interweaving plotlines.

A more plausible possibility is that we’re not less capable of reading complex prose, but less willing to put in the work. Our criterion for concluding “this is boring. This is not paying off” has been lowered because the Web makes it so easy to find something else to read, watch, or listen to. (I explore the possibility in some detail in my upcoming book, Raising a Reader in an Age of Distraction.) If I’m right, there’s good news and bad news. The good news is that our brains are not being deep-fried by the Web; we can still read deeply and think carefully. The bad news is that we don’t want to.

Reference

Daniel, D. B. & Willingham, D. T. (2012). Electronic textbooks: Why the rush? Science, 335, 1569-1571.

This blog posting first appeared on RealClearEducation on April 8, 2014.

The 2012 results for the brand-new PISA problem-solving test were released last week. News in various countries predictably focused on how well local students fared, whether they were American, British, Israeli, or Malaysian. The topic that should have been of greater interest  was what the test actually measures.

How do we know that a test measures what it purports to measure? There are a few ways to approach this problem.

One is when the content of the test seems, on the face of it, to represent what you’re trying to test. For example, math tests should require the solution of mathematical problems. History tests should require that test-takers display knowledge of history and the ability to use that knowledge as historians do. 

Things get trickier when you’re trying to measure a more abstract cognitive ability like intelligence. In contrast to math, where we can at least hope to specify what constitutes the body of knowledge and skills of the field, intelligence is not domain-specific. So we must devise other ways to validate the test. For example, we might say that people who score well on our test show their intelligence in other commonly accepted ways, like doing well in school and on the job.

Another strategy  is to define what the construct means—“here’s my definition of intelligence”—and then make a case for why your test items measure that construct as you’ve defined it.

So what approach does PISA take to problem-solving? It uses a combined strategy that ought to prompt serious reflection in education policymakers.

There is not any attempt to tie performance on the test to everyday measures of problem-solving. (At least, none have been offered so far, but there is more detail on the construction of the test to come, in an as-yet-unpublished technical report.)

From the scores report, it appears that the problem-solving test was motivated by a combination of the other two methods.

First, the OECD describes a conception of problem solving—what they think the mental processes look like. That includes the following processes:

·         Exploring and understanding

·         Representing and formulating

·         Planning and executing

·         Monitoring and reflecting

So we are to trust that the test measures problem solving ability because these are the constituent processes of problem solving, and we are to take it that the test authors could devise test items that tap these cognitive processes.

Now, this candidate taxonomy of processes that go into problem-solving seems reasonable at a glance, but I wouldn’t say that scientists are certain it’s right, or even that it’s the consensus best guess. Other researchers have suggested that different dimensions of problem solving are important—for example, well-defined problems vs. ill-defined problems. So pinning the validity of the PISA test on this particular taxonomy reflects a particular view of problem-solving.

But the OECD uses a second argument as well. They take an abstract cognitive process—problem solving—and vastly restrict its sweep by essentially saying “sure, it’s broad, but there is a limited way that we really care about how it’s implemented. So we just test those.”

That’s the strategy adopted by the National Adult Assessment of Literacy. Reading comprehension, like problem solving, is a cognitive process, and, like problem solving, it is intimately intertwined with domain knowledge. We’re better at reading about topics we already know something about. Likewise, we’re better at solving problems in domains we know something about. So in addition to (as best they could) requiring very little background knowledge for the test items, the designers of the NAAL wrote questions that they could argue reflect the kind reading people must do for basic citizenship. Things like reading a government-issued pamphlet about how to vote, and reading a bus schedule, and reading the instructions on prescription medicine.

The PISA problem solving test does something similar. They authors sought to present problems that students might really encounter, like figuring out how to work a new MP3 player, or finding the quickest route on a map, and or figuring out how to buy a subway ticket from an automated kiosk.

So with this justification, we don’t need to make a strong case that we really understand problem-solving at a psychological level at all. We just say “this is the kind of problem solving that people do, so we measured how well students do it.” 

This justification makes me nervous because the universe of possible activities we might agree represent “problem solving” seems so broad, much broader than what we would call activities for “citizenship reading.” A “problem” is usually defined as a situation in which you have a goal and you lack a ready process in memory that you’ve used before to solve the problem or one similar to it. That covers a lot of territory. So how do we know that the test fairly represents this territory?

The taxonomy is supposed to help with that problem. “Here’s the type of stuff that goes into problem solving, and look, we’ve got some problems for each type of stuff.” But I’ve already said that psychologists don’t have a firm enough grasp of problem-solving to advance a taxonomy with much confidence.

So the PISA 2012 is surely measuring something, and what it’s measuring is probably close to something I’d comfortably call “problem-solving.” But beyond that, I’m not sure what to say about it.

I probably shouldn’t get overwrought just yet—as I’ve mentioned, there is a technical report yet to come that will, I hope, leave all of us with a better idea of just what a score on this test means. Gaining that better idea will entail some hard work for education policymakers. The authors of the test have adopted a particular view of problem solving—that’s the taxonomy—and they have adopted a particular type of assessment—novel problems couched in everyday experiences. Education policymakers in each country must determine whether that view of problem solving syncs with theirs, and whether the type of assessment is suitable for their educational goals.

The way that people conceive of the other PISA subjects (math, science and reading) is almost surely more uniform than the way they conceive of problem-solving. Likewise, the goals for assessing those subjects is also more uniform. Thus, the problem of interpreting the problem-solving PISA scores is formidable compared to interpreting other scores. So no one should despair or rejoice over their country’s performance just yet.