Readers of this blog are probably aware of the research, pioneered by Carol Dweck, showing that certain types of praise--especially praise that focuses on who the child is, rather than what the child has done--can have counter-intuitive effects.

A new report (Brummelman et al, 2013) shows that the consequences of certain praise for kids with low self-esteem can be particularly destructive.

In experiment 1, 357 Dutch-speaking parents (87% mothers) read brief descriptions of children, some with high self-esteem ("Lisa usually likes the kind of person she is") and some with low ("Sarah is often unhappy with herself.") Parents were asked to describe what they would say in response to something the child was described as having done (e.g., "she has just made a beautiful drawing.")

Responses were coded as praising the child's personal qualities (e.g., "You're such a good drawer") or the child's behaviors (""you did a good job drawing!"), other praise (e.g., "beautiful!") or no praise.

The figure shows an interaction--children with high self-esteem were less likely to receive person praise than children with low self-esteem. Children with high self-esteem were more likely to receive process praise.

Experiment 2 examined whether children with high or low self-esteem respond differently to person praise.

313 children (mean age about 10.5 yrs) completed a standard measure of self-esteem. Several days later at their school, they performed a computer task that (they were told) pitted them against an opponent from another school to see who had faster reactions. They were told that a webmaster would monitor the competitors' performance. (In fact, there was no competitor or webmaster; everything was controlled by the computer.)

After a practice round the webmaster gave either process praise ("wow, you did a great job!"), person praise ("wow, you're great!") or no praise to the subject.

Next, the subject played against their "opponent" and were told that he or she won or lost.

Finally, subjects were asked to rate "how you feel, right now" by agreeing or disagreeing with adjectives like "ashamed," and "humiliated." (They had made similar ratings before the game.)

The graph shows difference scores, based on the two measures of shame (taken before and after the reaction time game).

As you would expect, the students who were told they won (the open figures in the graph) didn't feel much shame. Students told they lost (closed figures) felt more.

In addition, the students receiving person praise feel more shame, overall, but crucially, all of this effect is due to the students with low self esteem.  They are represented by that highest point on the graph at the upper left.

The effect size is pretty substantial--around d = .5

So it seems that the person praise makes the children with low-self-esteem feel more invested in the game, more like they have something at stake. So when they lose, they feel more shame. The high-self esteem students, in contrast, shrug off the loss, even after the person praise because they generally feel more secure about their abilities.

The message, coupled with the result from Experiment 1, is that adults are biased to do exactly the wrong thing--try to "buck up" kids with low-self esteem by offering person praise ("you're a great kid!") when these children will actually suffer more after a failure if they have received this praise.

The interpretation hangs together, to my mind, but I'd like to see this effect replicated. In particular, the measure of shame seemed heavy-handed. As far as I can tell, students were not asked about other feelings, just those related to shame, so there is a really chance that demand characteristics played a role. (That is, that students were reacting as they thought the experimenter expected them to, not necessarily as they felt.)

Still, an interesting, possibly important experiment.

Reference: Brummelman, E., Thomaes, S., Overbeek, G., Orobio de Castro, B., van den Hout, M. A., & Bushman, B. J. (2013). On Feeding Those Hungry for Praise: Person Praise Backfires in Children With Low Self-Esteem. Journal of Experimental Psychology: General. Advance online publication: doi: 10.1037/a0031917

Daphne Bavelier and Richard Davidson have a Comment in Nature today on the potential for video games to "do you good."

The authors note that video gaming has been linked to obesity, aggressiveness, and antisocial behavior, but there is a burgeoning literature showing some cognitive benefits accrue from gaming. Even though the data on these benefits is not 100% consistent (as I noted here) I'm with Bavelier & Davidson in their general orientation: so many people spend so much time gaming, we would be fools not to consider ways that games might be turned to purposes of personal and societal benefit.

Could games help to make people smarter, or more empathic, or more cooperative?

The authors suggest three developments are necessary.

1. Game designers and neuroscientists must collaborate to determine which game components "foster brain plasticity." (I believe they really mean "changes behavior.")
2. Neuroscientists ought to collaborate more closely with game designers. Presumably, the first step will not get off the ground if this doesn't happen.
3. There needs to translational game research, and a path to market. We expect that some research advances (and clinical trials) of the positive effects of gaming will be made in academic circles. This work must get to market if it is to have an impact, and there is not a blazed trial by which this travel can take place.

This is all fine, as far as it goes, but it ignores two glaring problems, both subsets of their first point.

We have to bear in mind that Bavelier & Davidson's enthusiasm for the impact of gaming is coming from experiments with people who already liked gaming; you compare gamers with non-gamers and find some cognitive edge for the former. Getting people to play games is no easy matter, because designing good games is hard.

This idea of harnessing interest in gaming for personal benefit is old stuff in education. Researchers have been at it for twenty years, and one of the key lessons they've learned is that it's hard build a game that students really like and from which they also learn (as I've noted in reviews here and here.)

Second, Bavelier & Davidson are also a bit too quick to assume that measured improvements to basic cognitive processes will transfer to more complex processes. They cite a study in which playing a game improved mental rotation performance. Then they point out that mental rotation is important in fields like navigation and research chemistry.

But one of the great puzzles (and frustrations) of attempts to improve working memory has been the lack of transfer; even when  working memory is improved by training, you don't see a corresponding improvement in tasks that are highly correlated with working memory (e.g., reasoning).

In sum, I'm with Bavelier & Davidson in that I think this line of research is well worth pursuing. But I'm less sanguine than they are, because I think their point #1--getting the games to work--is going to be a lot tougher than they seem to anticipate.

Bavelier, D, & Davidson,  R. J. (2013). Brain training: Games to do you good. Nature, 494,  425-426.

A math teacher and Twitter friend from Scotland asked me about about this figure.

I'm sure you've seen a figure like this. It is variously called the "learning pyramid," the "cone of learning," "the cone of experience," and others. It's often attributed to the National Training Laboratory, or to educator Edgar Dale.

You won't be surprised to learn that there are different versions out there with different percentages and some minor variations in the ordering of ac

Certainly, some mental activities are better for learning than others. And the ordering offered here doesn't seem crazy. Most people who have taught agree that long-term contemplation of how to help others understand complicated ideas is a marvelous way to improve one's own understanding of those ideas--certainly better than just reading them--although the estimate of 10% retention of what one reads seems kind of low, doesn't it?

If you enter "cone of experience" in Google scholar the first page offers a few papers that critique the idea, e.g., this one and this one, but you'll also see papers that cite it as if it's reliable.

It's not.

So many variables affect memory retrieval, that you can't assign specific percentages of recall without specifying many more of them:

• what material is recalled (gazing out the window of a car is an audiovisual experience just like watching an action movie, but your memory for these two audiovisual experiences will not be equivalent)
  
• the age of the subjects
  
• the delay between study and test (obviously, the percent recalled usually drops with delay)
  
• what were subjects instructed to do as they read, demonstrated, taught, etc. (you can boost memory considerably for a reading task by asking subjects to summarize as they read)
  
• how was memory tested (percent recalled is almost always much higher for recognition tests than recall).
• what subjects know about the to-be-remembered material (if you already know something about the subject, memory will be much better.
  

This is just an off-the-top-of-my-head list of factors that affect memory retrieval. They not only make it clear that the percentages suggested by the cone can't be counted on, but that the ordering of the activities could shift, depending on the specifics.



The cone of learning may not be reliable, but that doesn't mean that memory researchers have nothing to offer educators. For example, monograph published in January offers an extensive review of the experimental research on different study techniques. If you prefer something briefer, I'm ready to stand by the one-sentence summary I suggested in Why Don't Students Like School?: It's usually a good bet to try to think about material at study in the same way that you anticipate that you will need to think about it later.

And while I'm flacking my books I'll mention that When Can you Trust the Experts was written to help you evaluate the research basis of educational claims, cone-shaped or otherwise.

A new bill just passed the Education committee in the Oklahoma House of representatives, as reported in the Oklahoman. Titled "The Scientific Education and Academic Freedom Act," the bill purports to protect the rights of students, teachers and administrators to explore fully scientific controversies.

The bill supposes that some people currently feel inhibited in their pursuit of truth regarding "biological evolution, the chemical origins of life, global warming, and human cloning" and so the bill forbids school administrators and boards of education from disallowing such "exploration."

According to opinion pieces in the Daily Beast, The Week, and Mother Jones, the bill is a fairly transparent attempt to allow intelligent design into science classrooms, one that is being pursued in other states as well.

Yeah, that's what it sounds like to me too.

But even if we take the purported motive of the bill at face value, it's still a terrible idea.

Why shouldn't science teachers "teach the controversy?" Isn't it the job of teachers to sharpen students critical thinking skills? Isn't it part of the scientific method to evaluate evidence? If evolution proponents are so sure their theory is right, why are they afraid of students scrutinizing the ideas?

Imagine this logic applied in other subjects. Why shouldn't students study and evaluate the version of US history offered by white supremacists? Rather than just reading Shakespeare and assuming he's a great playwright, why not ask students to read Shakespeare and the screenplay to Battlefield Earth, and let students decide? And hey, why is such deference offered to Euclid? My uncle Leon has an alternative version of plane geometry and it shows Euclid was all wrong. I think that theory deserves a hearing.

You get the point. Not every theory merits the limited time students have in school. There is a minimum bar of quality that has to be met in order to compete. I'm not allowed to show up at the Olympics, hoping to jump in the pool and swim the 100 m butterfly against Michael Phelps.

Indeed, the very inclusion of a theory in a school discussion signals to students that it must have some validity--why else would the teacher discuss it?

The obvious retort from supporters of the bill is that intelligent design is actually a good theory, much better than the comparisons I've drawn.

That belief may be sincere, but it's due, I think, to a lack of understanding of scientific theory.  So here are a few of the important features of how scientists think about theories, and how they bear on this debates.

1) It's not telling that legitimate scientists point out unanswered questions, problems, or lacunae in the theory of evolution. Every theory, even the best theories, have problems. People who make this point may be thinking about the status of scientific laws as scientists did until the early part of the 20th century--as immutable laws. Scientists today think of all theories as provisional, and open to emendation and improvement.

2) A vital aspect of a good scientific theory is that it be open to falsification. It's not obvious what sort of data would falsify intelligent design theories, especially young-earth theories, which make predictions that are already disconfirmed by geology, astrophysics, etc., and yet are maintained by their adherents. Evolution, in contrast, has survived tests and challenges for 100 years--indeed, the theory has changed and improved in response to those challenges.

3) In the case of old-earth intelligent design theories, the focus is much more on the putative beginnings of the universe of or life on Earth, and these don't have the feel of a scientific theory at all. They seem much more like philosophical queries because they focus on large-scale questions and how these questions ought to formulated--they never get to detailed questions that might be answerable by experiment, the meat-and-potatoes of science.

4) Good scientific theories are not static. They not only change in the face of new evidence, they continue to spawn new and interesting hypotheses. Evolution has been remarkably successful on this score for over 100 years. Intelligent design has been static and unfruitful.

These are some of the reasons that scientists think that intelligent design does not qualify as a good scientific theory, and therefore does not merit close attention in K-12 science classes, and more than my uncle's theory of geometry does.

If you're going to write bills about what happens in science class, it's useful to know a little science.

EDIT: 2/22/13 1:20 p.m. EST: typos

What aspects of background, personality, or achievement predict success in college--at least, "success" as measured by GPA?

A recent meta-analysis (Richardson, Abraham, & Bond, 2012) gathered articles published between 1997 and 2010, the products of 241 data sets. These articles had investigated these categories of predictors:

• three demographic factors (age, sex, socio-economic status)
  
• five traditional measures of cognitive ability or prior academic achievement (intelligence measures, high school GPA, SAT or ACT, A level points)
  
• No fewer than forty-two non-intellectual measures of personality, motivation, or the like, summarized into the categories shown in the figure below (click for larger image).
  

Make this fun. Try to predict which of the factors correlate with college GPA.

Let's start with simple correlations.

41 out of the 50 variables examined showed statistically significant correlations. But statistical significance is a product of the magnitude of the effect AND the size of the sample--and the samples are so big that relatively puny effects end up being statistically significant. So in what follows I'll mention correlations of .20 or greater.

Among the demographic factors, none of the three were strong predictors. It seems odd that socio-economic status would not be important, but bear in mind that we are talking about college students, so this is a pretty select group, and SES likely played a significant role in that selection. Most low-income kids didn't make it, and those who did likely have a lot of other strengths.

The best class of predictors (by far) are the traditional correlates, all of which correlate at least r = .20 (intelligence measures) up to r = .40 (high school GPA; ACT scores were also correlated r = .40).

Personality traits were mostly a bust, with the exception of consientiousness (r = .19), need for cognition (r = .19), and tendency to procrastinate (r = -.22). (Procrastination has a pretty tight inverse relationship to conscientiousness, so it strikes me as a little odd to include it.)

Motivation measures were also mostly a bust but there were strong correlations with academic self-efficacy (r = .31) and performance self-efficacy (r = .59). You should note, however, that the former is pretty much like asking students "are you good at school?" and the latter is like asking "what kind of grades do you usually get?" Somewhat more interesting is "grade goal" (r = .35) which measures whether the student is in the habit of setting a specific goal for test scores and course grades, based on prior feedback.

Self-regulatory learning strategies likewise showed only a few factors that provided reliable predictors, including time/study management (r = .22) and effort regulation (r = .32), a measure of persistence in the face of academic challenges.

Not much happened in the Approach to learning category nor in psychosocial contextual influences.

We would, of course, expect that many of these variables would themselves be correlated, and that's the case, as shown in this matrix.

So the really interesting analyses are regressions that try to sort out which matter more.

The researchers first conducted five hierarchical linear regressions, in each case beginning with SAT/ACT, then adding high school GPA, and then investigating whether each of the five non-intellective predictors would add some predictive power. The variables were conscientiousness, effort regulation, test anxiety, academic self efficacy, and grade goal, and each did, indeed, add power in predicting college GPA after "the usual suspects" (SAT or ACT, and high school GPA) were included.

But what happens when you include all the non-intellective factors in the model?

The order in which they are entered matters, of course, and the researchers offer a reasonable rationale for their choice; they start with the most global characteristic (conscientiousness) and work towards the more proximal contributors to grades (effort regulation, then test anxiety, then academic self-efficacy, then grade goal).

As they ran the model, SAT and high school GPA continued to be important predictors. So were effort regulation and grade goal.

You can usually quibble about the order in which variables were entered and the rationale for that ordering, and that's the case here.  As they put the data together, the most important predictors of college grade point average are: your grades in high school, your score on the SAT or ACT, the extent to which you plan for and target specific grades, and your ability to persist in challenging academic situations.

There is not much support here for the idea that demographic or psychosocial contextual variables matter much. Broad personality traits, most motivation factors, and learning strategies matter less than I would have guessed.

No single analysis of this sort will be definitive. But aside from that caveat, it's important to note that most admissions officers would not want to use this study as a one-to-one guide for admissions decisions. Colleges are motivated to admit students who can do the work, certainly. But beyond that they have goals for the student body on other dimensions: diversity of skill in non-academic pursuits, or creativity, for example.

When I was a graduate student at Harvard, an admissions officer mentioned in passing that, if Harvard wanted to, the college could fill the freshman class with students who had perfect scores on the SAT. Every single freshman-- 800, 800. But that, he said, was not the sort of freshman class Harvard wanted.

I nodded as though I knew exactly what he meant. I wish I had pressed him for more information.

References:
Richardson, M., Abraham, C., Bond, R. (2012). Psychological correlates of university students' academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138,  353-387.

What people learn (or don't) from games is such a vibrant research area we can expect fairly frequent literature reviews. It's been about a year since the last one, so I guess we're due.

The last time I blogged on this topic Cedar Riener  remarked that it's sort of silly to frame the question as "does gaming work?" It depends on the game.

The category is so broad it can include a huge variety of experiences for students. If there were NO games from which kids seemed to learn anything, you probably ought not to conclude "kids can't learn from games." To do so would be to conclude that distribution of learning for all possible games and all possible teaching would look something like this.

But this pattern of data seems highly unlikely. It seems much more probable that the distributions overlap more, and that whether kids learn more from gaming or traditional teaching is a function of the qualities of each.

So what's the point of a meta-analysis that poses the question "do kids learn more from gaming or traditional teaching?

I think of these reviews not as letting us know whether kids can learn from games, but as an overview of where we are--just how effective are the serious games offered to students?

The latest meta-analysis (Wouters et al, 2013) includes data from 56 studies and examined both learning outcomes (77 effect sizes), retention (17 effect sizes) and motivation (31 effect sizes).

The headline results featured in the abstract is "games work!" Games are reported to be superior to conventional instruction in terms of learning (d = 0.29) and retention (d = .36) but somewhat surprisingly, not motivation (d = .26).

The authors examined a large set of moderator variables and this is where things get interesting. Here are a few of these findings:

1. Students learn more when playing games in groups than playing alone.
2. Peer-reviewed studies showed larger effects than others. (This analysis is meant to address the bias not to publish null results. . . but the interpretation in this case was clouded by small N's.)
3. Age of student had no impact.

But two of the most interesting moderators significantly modify the big conclusions.

First, gaming showed no advantage over conventional instruction when the experiment used random assignment. When non-random assignment was used, gaming showed a robust advantage. So it's possible (or even likely) that games in these studies were more effective only when they interacted with some factor in the gamer that is self-selected (or selected by the experimenter or teacher). And we don't know yet what that factor is.

Second the researchers say that gaming showed and advantage over "conventional instruction" but followup analyses show that gaming showed no advantage over what they called "passive instruction"--that it, the teacher talk or reading a textbook. All of the advantage accrued when games were compared to "active instruction," described as "methods that explicitly prompt learners to learning activities (e.g., exercises, hypertext training.)" So gaming (in this data set) is not really better than conventional instruction; it's better than one type of instruction (which in the US is probably less often encountered.)

So yeah, I think the question in this review is ill-posed. What we really want to know is how do we structure better games? That requires much more fine-grained experiments on the gaming experience, not blunt variables. This will be painstaking work.

Still, you've got to start somewhere and this article offers a useful snapshot of where we are.

EDIT 5:00 a.m. EST 2/11/13. In the original post I failed to make explicit another important conclusion--there may be caveats on when and how the games examined are superior to conventional instruction, but they were almost never worse. This is not an unreasonable bar, and as a group the games tested pass it.

Wouters, P, van Nimwegen, C, van Oostendorp, H., & van der Spek, E. G. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology. Advance online publication. doi: 10.1037/a0031311

Michael Gove, Secretary of Education in Great Britain, certainly has a flair for oratory.

In his most recent speech, he accused his political opponents of favoring "Downton Abbey-style" education (meaning one that perpetuates class differences), he evoked a 13 year old servant girl reading Keats, and he cited as an inspiration the late British reality TV star Jade Goody (best known for being ignorant), and Marxist writer and political theorist Antonio Gramsci.

Predictably, press coverage in Britain has focused on these details. (So, of course, have the Tweets.) The Financial Times and the Telegraph pointed to Gove's political challenge to Labour. The Guardian led with the Goody & Gramsci angle.

But these points of color distract from the real aim. The fulcrum of the speech is the argument that a knowledge-based curriculum is essential to bring greater educational opportunity to disadvantaged children. (The BBC got half the story right.)

The logic is simple:

1) Knowledge is crucial to support cognitive processes. (e.g., Carnine & Carnine, 2004; Hasselbring, 1988; Willingham, 2006).

2) Children who grow up in disadvantaged circumstances have fewer opportunities to learn important background knowledge at home (Walker et al, 1994) and they come to school with less knowledge, which has an impact on their ability to learn new information at school (Grissmer et al 2010) and likely leads to a negative feedback cycle whereby they fall farther and farther behind (Stanovich, 1986).

Gove is right.  And he's right to argue for a knowledge-based curriculum. The curriculum is most likely to meliorate achievement gaps between advantaged and disadvantaged students because a good fraction of that difference is fueled by differences in cultural capital in the home--differences that schools must try to make up. (Indeed, a knowledge-based curriculum is a critical component of KIPP and other "no excuses" schools in the US.)

I'm not writing to defend all education policies undertaken by the current British government--I'm not knowledgeable enough about those policies to defend or attack them.

But I find the response from Stephen Twigg (Labour's shadow education secretary) disquieting, because he seems to have missed Gove's point.

"Instead of lecturing others, he should listen to business leaders, entrepreneurs, headteachers and parents who think his plans are backward looking and narrow. We need to get young people ready for a challenging and competitive world of work, not just dwell on the past." (As quoted in the Financial Times.)

It's easy to scoff at a knowledge-based curriculum as backward-looking. Memorization of math facts when we have calculators? Knowledge in the age of Google?

But if you mistake advocacy for a knowledge-based curriculum as wistful nostalgia for a better time, or as "old fashioned" you just don't get it.

Surprising though it may seem, you can't just Google everything. You actually need to have knowledge in your head to think well. So a knowledge-based curriculum is the best way to get young people "ready for the world of work."

Mr. Gove is rare, if not unique, among high-level education policy makers in understanding the scientific point he made in yesterday's speech. You may agree or disagree with the policies Mr. Gove sees as the logical consequence of that scientific point, but education policies that clearly contradict it are unlikely to help close the achievement gap between wealthy and poor.

References

Carnine, L., & Carnine, D. (2004). The interaction of reading skills and science content knowledge when teaching struggling secondary students. Reading & Writing Quarterly, 20(2), 203-218.

Grissmer, D., Grimm, K. J., Aiyer, S. M., Murrah, W. M., & Steele, J. S. (2010). Fine motor skills and early comprehension of the world: Two new school readiness indicators. Developmental psychology, 46(5), 1008.

Hasselbring, T. S. (1988). Developing Math Automaticity in Learning Handicapped Children: The Role of Computerized Drill and Practice. Focus on Exceptional Children, 20(6), 1-7.

Stanovich, K. E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading research quarterly, 360-407.

Walker, D., Greenwood, C., Hart, B., & Carta, J. (1994). Prediction of school outcomes based on early language production and socioeconomic factors. Child development, 65(2), 606-621.

Willingham, D. T. (2006). How knowledge helps. American Educator, 30(1), 30-37.

If the title of this blog struck you as brash, I came by it honestly: it's the title of a terrific new paper by three NYU researchers (Protzko, Aronson & Blair, 2013). The authors sought to review all interventions meant to boost intelligence, and they cast a wide net, seeking any intervention for typically-developing children from birth to kindergarten age that used a standard IQ test as the outcome measure, and that was evaluated in a random control trial (RCT) experiment.

A feature of the paper I especially like is that none of the authors publish in the exact areas they review. Blair mostly studies self-regulation, and Aronson, gaps due to race, ethnicity or gender. (Protzko is a graduate student studying with Aronson.) So the paper is written by people with a lot of expertise, but who don't begin their review with a position they are trying to defend. They don't much care which way the data come out.

So what did they find? The paper is well worth reading in its entirety--they review a lot in just 15 pages--but there are four marquee findings.

First, the authors conclude that infant formula supplemented with long chain polyunsaturated fatty acids boosts intelligence by about 3.5 points, compared to formula without. They conclude that the same boost is observed if pregnant mothers receive the supplement. There are not sufficient data to conclude that other supplements--riboflavin, thiamine, niacin, zinc, and B-complex vitamins--have much impact, although the authors suggest (with extreme caution) that B-complex vitamins may prove helpful.

Second, interactive reading with a child raises IQ by about 6 points. The interactive aspect is key; interventions that simply encouraged reading or provided books had little impact. Effective interventions provided information about how to read to children: asking open-ended questions, answering questions children posed, following children's interests, and so on.

Third, the authors report that sending a child to preschool raises his or her IQ by a little more than 4 points. Preschools that include a specific language development component raise IQ scores by more than 7 points. There were not enough studies to differentiate what made some preschools more effective than others.

Fourth, the authors report on interventions that they describe as "intensive," meaning they involved more than preschool alone. The researchers sought to significantly alter the child's environment to make it more educationally enriching. All of these studies involved low-SES children (following the well-established finding that low-SES kids have lower IQs than their better-off counterparts due to differences in opportunity. I review that literature here.)  Such interventions led to a 4 point IQ gain, and a 7 point gain if the intervention included a center-based component. The authors note the interventions have too many features to enable them to pinpoint the cause, but they suggest that the data are consistent with the hypothesis that the cognitive complexity of the environment may be critical. They were able to confidently conclude (to their and my surprise) that earlier interventions helped no more than those starting later.

Those are the four interventions with the best track record. (Some others fared less well. Training working memory in young children "has yielded disappointing results." )

The data are mostly unsurprising, but I still find the article a valuable contribution. A reliable, easy-to-undertand review on an important topic.

Even better, this looks like the beginning of what the authors hope will be a longer-term effort they are calling the Database on Raising Intelligence--a compendium of RCTs based on interventions meant to boost IQ. That may not be everything we need to know about how to raise kids, but it's a darn important piece, and such a Database will be a welcome tool.

How much help is provided to a teacher and student by the use of  manipulatives--that is, concrete objects meant to help illustrate a mathematical idea?

My sense is that most teachers and parents think that manipulatives help a lot. I could not locate any really representative data on this point, but the smaller scale studies I've seen support the impression that they are used frequently. In one study of two districts the average elementary school teacher reported using manipulatives nearly every day (Uribe-Florez & Wilkins, 2010).

Do manipulatives help kids learn? A recent meta-analysis (Carbonneua, Marley & Selif, 2012) offers a complicated picture. The short answer is "on average, manipulatives help. . . a little." But the more complete answer is that how much they help depends on (1) what outcome you measure and (2) how the manipulatives are used in instruction.

The authors analyzed the results of 55 studies that compared instruction with or without manipulatives. The overall effect size was d = .37--typically designated a "moderate" effect.

But there were big differences depending on content being taught: for example, the effect for fractions was considerable larger (d = .69) than the effect for arithmetic (d = .27) or algebra (d = .21).
More surprising to me, the effect was largest when the outcome of the experiment focused on retention (d = .59), and was relatively small for transfer (d = .13).

What are we to make of these results? I think we have to be terribly cautious about any firm take-aways. That's obvious from the complexity of the results (and I've only hinted at the number of interactions).

It seems self-evident that one source of variation is the quality of the manipulative. Some just may not do that great a job of representing what they are supposed to represent. Others may be so flashy and interesting that they draw attention to peripheral features at the expense of the features that are supposed to be salient.

It also seems obvious that manipulatives can be more or less useful depending on how effectively they are used. For example, some fine-grained experimental work indicates the effectiveness of using a pan-balance as an analogy for balancing equations depends on fairly subtle features of what to draw students’ attention to and when (Richland et al, 2007).

My hunch is that at least one important source of variability (and one that's seldom measured in these studies) is the quality and quantity of relevant knowledge students have when the manipulative is introduced. For example, we might expect that the student with a good grasp of the numerosity would be in a better position to appreciate a manipulative meant to illustrate place value than the student whose grasp is tenuous. Why?

David Uttal and his associates (Uttall, et al, 2009) emphasized this factor when they pointed out that the purpose of a manipulative is to help students understand an abstraction. But a manipulative itself is an abstraction—it’s not the thing-to-be-learned, it’s a representation of that thing—or rather, a feature of the manipulative is analogous to a feature of the thing-to-be-learned. So the student must simultaneously keep in mind the status of the manipulative as concrete object and as a representation of something more abstract. The challenge is that keeping this dual status in mind and coordinating them can be a significant load on working memory. This challenge is potentially easier to meet for those students who firmly understand concepts undergirding the new idea.

I’m generally a fan of meta-analyses. I think they offer a principled way to get a systematic big picture of a broad research literature. But the question “do manipulatives help?” may be too broad. It seems too difficult to develop an answer that won’t be mostly caveats.

So what’s the take-away message? (1) manipulatives typically help a little, but the range of effect (hurts a little to helps a lot) is huge; 2) researchers have some ideas as to why manipulatives work or don’t work. . .but not in a way that offers much help in classroom application.

This is an instance where a teacher’s experience is a better guide.

References

Carbonneau, K. J., Marley, S. C., & Selig, J. P. (in press). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology. Advance online publication.

Richland, R. E.  Zur, O. Holyoak, K. J. (2007). Cognitive Supports for Analogies in the Mathematics Classroom, Science, 316, 1128–1129.

Uribe‐Flórez, L. J., & Wilkins, J. L. (2010). Elementary school teachers' manipulative use. School Science and Mathematics, 110, 363-371.

Uttal, D. H., O’Doherty, K., Newland, R., Hand, L. L., & DeLoache, J.
(2009). Dual representation and the linking of concrete and symbolic
representations. Child Development Perspectives, 3, 156–159.

An experiment is a question which science poses to Nature, and a measurement is the recording of nature’s answer. --Max Planck

You can't do science without measurement. That blunt fact might give pause when people emphasize non-cognitive factors in student success and in efforts to boost student success.

"Non-cognitive factors" is a misleading but entrenched catch-all term for factors such as motivation, grit, self-regulation, social skills. . .  in short, mental constructs that we think contribute to student success, but that don't contribute directly to the sorts of academic outcomes we measure, in the way that, say, vocabulary or working memory do.

Non-cognitive factors have become hip. (Honestly, if I hear about the Marshmallow Study just one more time, I'm going to become seriously dysregulated) and there are plenty of data to show that researchers are on to something important. But are they on to anything that that educators are likely to be able to use in the next few years? Or are we going to be defeated by the measurement problem ?

There is a problem, there's little doubt. A term like "self-regulation" is used in different senses: the ability to maintain attention in the face of distraction, the inhibition of learned or automatic responses, or the quelching of emotional responses. The relation among them is not clear.
Further, these might be measured by self-ratings, teacher ratings, or various behavioral tasks.

But surprisingly enough, different measures do correlate, indicating that there is a shared core construct (Sitzman & Ely, 2011). And Angela Duckworth (Duckworth & Quinn, 2009) has made headway in developing a standard measure of grit (distinguished from self-control by its emphasis on the pursuit of a long-term goal).

So the measurement problem in non-cognitive factors shouldn't be overstated. We're not at ground-zero on the problem. At the same time, we're far from agreed-upon measures. Just how big a problem is that?

It depends on what you want to do.

If you want to do science, it's not a problem at all. It's the normal situation. That may seem odd: how can we study self-regulation if we don't have a clear idea of what it is? Crisp definitions of constructs and taxonomies of how they relate are not prerequisites for doing science. They are the outcome of doing science. We fumble along with provisional definitions and refine them as we go along.

The problem of measurement seems more troubling for education interventions.

Suppose I'm trying to improve student achievement by increasing students' resilience in the face of failure. My intervention is to have preschool teachers model a resilient attitude toward failure and to talk about failure as a learning experience. Don't I need to be able to measure student resilience in order to evaluate whether my intervention works?

Ideally, yes, but that lack may not be an experimental deal-breaker.

My real interest is student outcomes like grades, attendance, dropout, completion of assignments, class participation and so on. There is no reason not to measure these as my outcome variables. The disadvantage is that there are surely many factors that contribute to each outcome, not just resilience. So there will be more noise in my outcome measure and consequently I'll be more likely to conclude that my intervention does nothing when in fact it's helping.

The advantage is that I'm measuring the outcome I actually care about. Indeed, there would not be much point in crowing about my ability to improve my psychometrically sound measure of resilience if such improvement meant nothing to education.

There is a history of this approach in education. It was certainly possible to develop and test reading instruction programs before we understood and could measure important aspects of reading such as phonemic awareness.

In fact, our understanding of pre-literacy skills has been shaped not only by basic research, but by the success and failure of preschool interventions. The relationship between basic science and practical applications runs both ways.

So although the measurement problem is a troubling obstacle, it's neither atypical nor final.



References
Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the Short Grit Scale (GRIT–S). Journal of Personality Assessment, 91, 166-174.

Sitzmann, T, & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137,  421-442.