Students from disadvantaged backgrounds and students who are the first in their families to attend college are at heightened risk to leave school without a degree. It's not a matter of students coming to school inadequately prepared--something else is at work (e.g., Steele, 1997). It's not merely disheartening for these students, there can be potentially grave financial repercussions for dropping out. 

Most colleges and universities have programs meant to help students with the process of transitioning, and they often focus on practical skills like choosing classes and study strategies on the assumption that navigating academic life is a significant part of the problem.

In the last few years researchers have focused on a quite different approach: offering students a "lay theory" of the college experience. A lay theory is a set of beliefs that are used to interpret one's experiences. In the case of college, two beliefs have been flagged as especially important: that the transition to college inevitably involves setbacks, and that these setbacks are temporary.

College always includes disappointments, both academic and social. A student fails a test, or a callous professor tells them that their writing is beyond hope. (My freshman year of college an English professor wrote this as the entire comment on my exam essay: "No. D" Students get lonely, and have trouble making friends.  For students who grew up in families where it was always assumed that they would attend college, such disappointments are dispiriting, but not threatening. The student may even wonder if he or she belongs in college, but that doubt likely doesn't last. For a student who did not grow up in an environment where it was taken for granted that they would graduate college that doubt may persist. They may think that they are not smart enough to succeed, that they are "not college material," or that their cultural background is not compatible with college. 

Researchers have sought ways to instill a lay theory of college that would change that interpretation, focusing on two ideas: setbacks in college are common  (and therefore should not be taken as a sign that you don't belong) and setbacks are temporary (so things will get better). 

Researchers have had some success with this intervention in smaller experiments (Stephens et al, 2014; Walton & Cohen, 2011. Now a new study (Yeager et al, 2016) suggests that a simple, inexpensive intervention works at scale. 

Before they matriculated at college, students participated in an activity taking just 30 minutes, administered over the Internet. They were told that it was to help them think about the transition to college, and that they would have the chance to share their experiences, perhaps helping future students. 

There were three experimental conditions. The social belonging condition provided information showing that feeling out of place is common in the transition to college, but most students do make friends and succeed academically. The growth mindset condition provided information showing that intelligence is malleable, and that student can succeed with effort, coupled with effective strategies. The third condition combined both strategies. In each case, students were asked to write an essay about how the information they read might apply to them, as a way of cementing the information in memory, and to help them imagine making it applicable to their own experience.

One experiment targeted the entering class of a large public university. As shown in the table below, the intervention improved retention. All three of the intervention conditions were equally effective. 

Another experiment administered the intervention at a selective private university. The figure shows that disadvantaged students receiving the intervention earned higher GPAs in their first year of college 

Consistent with the theory, advantaged students don't benefit from the intervention; they already believe that they can succeed at college, and that they belong there, so their lay theory of college is likely already similar to the one described in the intervention.

Two things are noteworthy about this experiment. First, the reduction in the achievement gap is quite sizable, on the order of 30-40%. Second, this intervention was remarkably brief, and remarkably inexpensive. Obviously this work needs to be replicated and the interventions fine-tuned. (The growth mindset intervention didn't really work in Experiment 1.) But if this finding holds up, it must be counted as a huge success for social scientists, and for David Yeager and Greg Walton in particular. 

There's plenty of research on homework and the very brief version of the findings is probably well known to readers of this blog: homework has a modest effect on the academic achievement of older students, and no effect on younger students (Cooper et al, 2006).

In a way, this outcome seems odd. Practice is such an important part of certain types of skill, and much of homework is assigned for the purpose of practice. Why doesn't it help, or help more?

One explanation is that the homework assigned is not of very good quality, which could mean a lot of different things and absent more specificity sounds like a homework excuse. Another, better explanation is that practice doesn't do much unless there is rapid feedback, and that's usually absent at home. 

A third explanation is quite different, suggesting that the problem may lie in measurement. Most studies of homework efficacy have used student self-report of how much time they spend on homework. Maybe those reports are inaccurate. 

A new study indicates that this third explanation merits closer consideration. 

The researchers (Rawson, Stahovich & Mayer, in press) examined homework performance among three classes of undergraduate engineering students taking their first statics course. The homework assigned was typical for this sort of course; the atypical feature was that students were asked to complete their homework with Smartpens. These function like regular ink pens, but when coupled with special paper, they record time-stamped pen strokes.

The researchers were able to gather objective measures of time spent on homework, as well as other performance metrics.

A few of these measures proved interesting. For example, students who completed a lot of homework within 24 hours of the due date tended to earn lower course grades.

But the really interesting finding was a significant correlation of course grade and time spent on homework as measured by the Smartpen (r = .44) in the face of NO correlation between course grade and time spent on homework as reported by the students (r = -.16).

The relationship between homework and course grades is not the news. This is a college course and no matter what the format, it's only going to meet a few hours each week, and students will be expected to do a great deal of work on their own.

The news is that students were poor at reporting their time spent on homework; 88% reported more than the Smartpen showed they had actually spent. The correlation of actual time and reported time ranged from r = .16 to r = .35 for the three cohorts. 

In other words, with such a noisy measure of time spent on homework, there was little hope of observing a reliable relationship of homework with a course outcome. This finding ought to call into question much of the prior research on homework. 

Please don't take this blog posting as an enthusiastic endorsement of homework. For one thing, this literature seems pretty narrow in focusing solely on academic performance outcomes, given that many teachers and parents have other goals for homework such as increased self-directedness. For another thing, even if it were shown the certain types of homework led to certain types of improvement in academic outcomes, that doesn't mean every school and classroom ought to assign homework. That decision should be made in the context of broader goals.

But if teachers are going to assign homework, researchers should investigate its efficacy. This study should make us rethink how we interpret existing research in this area.

The idea that students would learn better in single-sex classrooms seems logical. The typical arguments include

• Boys find girls more distracting in class than they find other boys. Likewise, girls find boys more distracting.
• Sex differences in math and science achievement are a product of social influence. Those influences will be reduced or eliminated if girls are in classrooms only with girls.
• Boys dominate classroom discussion, and so girls are denied practice in articulating and defending their views.
• Boys and girls have different brains, and therefore learn differently. If they are taught separately, teachers can tune their instruction to the way each sex learns.

The last of these is frequently overwrought and over-interpreted, but generally, these reasons seem plausible. But that obviously doesn't prove that single-sex education confers any advantage to students.

A 2005 report written for the Department of Education (Mael et al, 2005) reported mixed effects, but generally a positive conclusion for single-sex classrooms in short-run academic outcomes. There was no indication of a boost to longer-term outcomes.

A new study (Pahlke, Hyde, & Allison, 2014) reports a meta-analysis of 184 studies representing 1.6 million students in K-12 across 21 nations. The authors place considerable emphasis on the problem of control in this research. They end up concluding that, with proper controls, analyses show that single-sex classrooms don't help students much.

The challenge in this sort of work is that comparisons of single-sex and coed classrooms often do not use random assignment. Students (or parents) choose a single-sex classroom. So for this review, the authors distinguished between controlled experiments (the original study either used random assignment or made some attempt to measure and statistically account for associated variables) and uncontrolled studies.

In controlled studies, there were statistically reliably, but numerically quite modest positive effects of single-sex classrooms for both boys and girls in mathematics achievement, science achievement, and verbal achievement (Hedges g in all cases less than 0.10). Girls showed an edge in single-sex classes for math attitude, science achievement, and overall academic achievement, but again, the gains were modest. If one restricts the analysis to U.S. students, virtually all of these small effects disappear.

There was no effect for attitudes towards school, gender stereotyping, educational aspirations, self-concept, interpersonal relationships, or body image.

There were not enough controlled studies to examine aggression, body image, interpersonal relations, interest in STEM careers, science attitudes, or victimization.

It's also notable that there was no dosage effect: the advantage was no larger when all classes within a school were single-sex classes, compared to when a single class was.

The authors were also interested in evaluating whether single sex classes were effective for boys of color. They reported that there were not enough controlled studies to answer this question, but even restricting the analysis to uncontrolled studies, the effects were minimal.

When you consider the factors that we know contribute substantially to academic attitudes and performance--the student's prior academic achievement, the curriculum, the home environment, the teacher's skill--it's easy to believe that the sex of the other students would have a modest effect, if any.

That said, it could be that a single sex school has a profound influence on a few students. A few years ago, friends of mine moved their 15 year old daughter to an all-girls school because she was "boy crazy." According to my friends, she didn't become any less interested in boys, but she did focus on work better during school hours. But then again it's possible my friends were kidding themselves.

References

Mael, F., Alonso, A., Gibson, D., Rogers, K., & Smith, M. (2005).
Single-sex versus coeducational schooling: A systematic review. Washington, DC: American Institutes for Research

Pahlke, E.,Hhyude, J. S, & Allison, C. M. (2014). The effects of single-sex cmopared with coeducational schooling on students' performance and attitudes: A meta-analysis. Psychological Bulletin, Advance online publication, http://dx.doi.org/10.1037/a0035740

The results of this experiment probably won't surprise you. What surprised me was the fact that we didn't already have data like this in hand.

The researchers (Sadler et al., 2013) tested 181 7th and 8th grade science teachers for their knowledge of physical science in fall, mid-year, and years end. They also tested their students (about 9,500) with the exact same instrument.

Each was a twenty-item multiple choice test. For 12 of the items, the wrong answers tapped a common misconception that previous research showed middle-schoolers often hold. For example, one common misconception is that burning produces no invisible gases. This question tapped that idea:

But the researchers didn't just ask the teachers to pick the right answer. They also asked teachers to pick the answer that they thought their students would pick.

What makes this study interesting is that it tests teacher subject-matter knowledge directly (instead of using a proxy like courses taken, or degrees) and that it directly measures one aspect of pedagogical content knowledge, namely, student misconceptions. The dependent measure of interest is student gain scores in content knowledge over the course of the year.

The results?

Teachers content knowledge was good, but not perfect. They got about 84% of the questions right.

Their knowledge of student misconceptions was not as good. Teachers correctly identified just 43% of those. (And their students had, as in previous studies, selected those incorrect items in high numbers.)

And what type of teacher knowledge matters to student learning? It turns out to interact with past student achievement, as measured by standard math and reading tests.

The graph shows gains in student knowledge, separated by items for which teachers have (or lack) various types of knowledge. Filled circles are for students who scored well on a math and reading test (high achievers), and open circles are students who scored poorly (low achievers)

Look first at learning for concepts without  a common misconception. If teachers have subject matter knowledge (SMK in the graph) students learn the concept better. In fact, low-achieving students learned nothing about a concept if teachers didn't know the concept themselves. High-achieving students did. The researchers speculate they may have learned the content from a textbook or other source.

For the strong misconception items, the low-achieving students learned very little, whatever the teacher knowledge. For high-achieving students, knowledge mattered, and they were most likely to learn when their teacher had both subject-matter knowledge and knew the misconceptions their students likely held (KoSM in the graph).

So the overall message is not that surprising. Students learn more when their teachers know the content, and when they can anticipate student misconceptions.

Somewhat more surprising (and saddening), low-achieving students are especially vulnerable when teachers lack knowledge. High-achieving students are more resilient.

There are limitations to this study, the most notable being that the sample is far from random (teachers were volunteers), and that the test was zero-stakes for all.

The strength was the direct measure of both types of knowledge, and that the researchers could examine the relationship of knowledge to performance at the level of individual items. One hopes we'll see more studies using this type of design.

Reference:
Sadler, P. M., Sonnert, G., Coyle, H.P., Cook-Smith, N., & Miller, J.L. (2013) Student learning in middle school science classrooms. American Educational Research Journal, 50, 1020-1049.

In Why Don't Students Like School? I pointed out that cognitive challenge is engaging if it's at the right level of difficulty, but boring if it's too easy or too hard. It sounds, then, like it would make sense to organize students into different classes based on their prior achievement.

It might make sense cognitively, but the literature shows that such a practice leads to bad outcomes for the kids in lower tracks. Those classes tend to have less demanding curricula and and lower expectations for achievement (e.g., Brunello & Checchi, 2007).

Further, assignment to tracks is often biased by race or social class (e.g., Maaz et al., 2007).

What tracking does to students self-perceptions has been less clear. A new international study (Chmielewski et al., 2013) examined data from the 2003 PISA data set to examine the association of different types of tracking and student self-perceptions of mathematics self-concept.

The authors compared systems with

• Between school streaming: in which students with different levels of achievement are sent to different schools.
• Within school streaming: in which students with different levels of achievement are put in different sequences of courses for all subjects.
• Course-by-course tracking: in which students are assigned to more or less advanced courses within a school, depending on their achievement within a particular subject.

Controlling for individual achievement and the average achievement of the track or stream, the researchers found that course tracking is associated with worse self-perceptions among low-achieving students, but streaming is associated with better self-perceptions.

This figure shows the difference between the self perceptions of higher and lower achieving students in individual countries, sorted by the type of tracking system.

The data suggest that when students are tracked for some but not all of their courses, they compare their achievement to other, more advanced students, perhaps because they see these students more often. Students who are streamed within or between schools, in contrast, compare their abilities to their fellow stream-mates.

But why is there self-concept higher than higher-achieving students? This effect may be comparable to a more general phenomenon that people are poorer judges of their competence for tasks that they perform poorly. If you're not very good, you're not good enough to realize what you lack.

The authors do not suggest that between school steaming is the way to go (since it's associated with higher confidence). They note that the association is just the reverse of that seen in achievement: kids who stream between schools seem to take the biggest hit to achievement.

References

Brunello, G., & Checchi, D. (2007). Does school tracking affect equality of opportunity? New international evidence. Economic Policy, 22, 781–861.

Chmielewski, A. K., Dumont, H. Trautwein, U. (2013). Tracking effects depend on tracking type: An international comparison of students' mathematics self-concept. American Educatioal Research Journal, 50,  925-957.

Maaz, K., Trautwein, U., Ludtke, O., & Baumert, J. (2008). Educational transitions and differential learning environments: How explicit between-school tracking contributes to social inequality in educational outcomes. Child Developmental Perspectives, 2, 99–106.

The cover story of latest New Republic wonders whether American educators have fallen in blind love with self-control. Author Elizabeth Weil thinks we have. Titled “American Schools Are Failing Nonconformist Kids: In Defense of the Wild Child” the article suggests that educators harping on self-regulation are really trying to turn kids into submissive little robots. And they do so because little robots are easier to control in the classroom.

But lazy teachers are not the only cause. Education policy makers are also to blame, according to Weil. She writes that “valorizing self-regulation shifts the focus away from an impersonal, overtaxed, and underfunded school system and places the burden for overcoming those shortcomings on its students.”

And the consequence of educators’ selfishness? Weil tells stories that amount to Self-Regulation Gone Wild. A boy has trouble sitting cross-legged in class—the teacher opines he should be tested because something must be wrong with him. During story time the author’s daughter doesn’t like to sit still and to raise her hand when she wants to speak. The teacher suggests occupational therapy.

I can see why Weil and her husband were angry when their daughter’s teacher suggested occupational therapy simply because the child’s behavior was an inconvenience to him. But I don’t take that to mean that there is necessarily a widespread problem in the psyche of American teachers. I take that to mean that their daughter’s teacher was acting like a selfish bastard.

The problem with stories, of course, is that there are stories to support nearly anything. For every story a parent could tell about a teacher diagnosing typical behavior as a problem, a teacher could tell a story about a child who really could do with some therapeutic help, and whose parents were oblivious to that fact.

What about evidence beyond stories?

Weil cites a study by Duncan et al (2007) that analyzed six large data sets and found social-emotional skills were poor predictors of later success.

She also points out that creativity among American school kids dropped between 1984 and 2008 (as measured by the Torrance Test of Creative Thinking) and she notes “Not coincidentally, that decrease happened as schools were becoming obsessed with self-regulation.”

There is a problem here. Weil uses different terms interchangeably: self-regulation, grit, social-emotional skills. They are not same thing. Self-regulation (most simply put) is the ability to hold back an impulse when you think that that the impulse will not serve other interests. (The marshmallow study would fit here.) Grit refers to dedication to a long-term goal, one that might take years to achieve, like winning a spelling bee or learning to play the piano proficiently. Hence, you can have lots of self-regulation but not be very gritty. Social emotional skills might have self-regulation as a component, but it refers to a broader complex of skills in interacting with others.

These are not niggling academic distinctions. Weil is right that some research indicates a link between socioemotional skills and desirable outcomes, some doesn’t. But there is quite a lot of research showing associations between self-control and positive outcomes for kids including academic outcomes, getting along with peers, parents, and teachers, and the avoidance of bad teen outcomes (early unwanted pregnancy, problems with drugs and alcohol, et al.). I reviewed those studies here. There is another literature showing associations of grit with positive outcomes (e.g., Duckworth et al, 2007).

Of course, those positive outcomes may carry a cost. We may be getting better test scores (and fewer drug and alcohol problems) but losing kids’ personalities. Weil calls on the reader’s schema of a “wild child,” that is, an irrepressible imp who may sometimes be exasperating, but whose very lack of self-regulation is the source of her creativity and personality.

But irrepressibility and exuberance is not perfectly inversely correlated with self-regulation. The purpose of self-regulation is not to lose your exuberance. It’s to recognize that sometimes it’s not in your own best interests to be exuberant. It’s adorable when your six year old is at a family picnic and impulsively practices her pas de chat because she cannot resist the Call of the Dance. It’s less adorable when it happens in class when everyone else is trying to listen to a story.

So there’s a case to be made that American society is going too far in emphasizing self-regulation. But the way to make it is not to suggest that the natural consequence of this emphasis is the crushing of children’s spirits because self-regulation is the same thing as no exuberance. The way to make the case is to show us that we’re overdoing self-regulation. Kids feel burdened, anxious, worried about their behavior.

Weil doesn’t have data that would bear on this point. I don’t either. But my perspective definitely differs from hers. When I visit classrooms or wander the aisles of Target, I do not feel that American kids are over-burdened by self-regulation.

As for the decline in creativity from 1984 and 2008 being linked to an increased focus on self-regulation…I have to disagree with Weil’s suggestion that it’s not a coincidence (setting aside the adequacy of the creativity measure). I think it might very well be a coincidence. Note that scores on the mathematics portion of the long-term NAEP increased during the same period. Why not suggest that kids improvement in a rigid, formulaic understanding of math inhibited their creativity?

Can we talk about important education issues without hyperbole? 

References

Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: perseverance and passion for long-term goals. Journal of personality and social psychology, 92(6), 1087.

Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., ... & Japel, C. (2007). School readiness and later achievement. Developmental psychology, 43(6), 1428.

Part of the fun and ongoing fascination of science of science is "the effect that ought not to work, yet does."

The impact of values of affirmation on academic performance is such an effect.

Values-affirmation "undoes" the effect of stereotype threat (also called identity threat). Stereotype threat occurs when a person is concerned about confirming a negative stereotype about his or her group. In other words a boy is so consumed with thinking "Everyone expects me to do poorly on this test because I'm African-American" that his performance actually is compromised (see Walton & Spencer, 2009 for a review).

One way to combat stereotype threat is to give the student better resources to deal with the threat--make the student feel more confident, more able to control the things that matter in his or her life.

That's where values affirmation comes in.

In this procedure, students are provided a list of values (e.g., relationships with family members, being good at art) and are asked to pick three that are most important to them and to write about why they are so important. In the control condition, students pick three values they imagine might be important to someone else.

Randomized control trials show that this brief intervention boosts school grades (e.g., Cohen et al, 2006).

Why?

One theory is that values affirmation gives students a greater sense of belonging, of being more connected to other people.

(The importance of social connection is an emerging theme in  other research areas. For example, you may have heard about the studies showing that people are less anxious when anticipating a painful electric shock if they are holding the hand of a friend or loved one.)

A new study (Shnabel et al, 2013) directly tested the idea that writing about social belonging might be a vital element in making values affirmation work.

In Experiment 1 they tested 169 Black and 186 White seventh graders in a correlational study. They did the values-affirmation writing exercise, as described above. The dependent measure was change in GPA (pre-intervention vs. post-intervention.) The experimetners found that writing about social belonging in the writing assignment was associated with a greater increase in GPA for Black students (but not for White students, indicating that the effect is due to reduction in stereotype threat.)

In Experiment 2, they used an experimental design, testing 62 male and 55 female college undergraduates on a standardized math test. Some were specifically told to write about social belonging and others were given standard affirmation writing instructions. Female students in the former group outscored those in the latter group. (And there was no effect for male students.)

The brevity of the intervention relative to the apparent duration of the effect still surprise me. But this new study gives some insight into why it works in the first place.

References:

Cohen, G. L., Garcia, J., Apfel, N., & Master, A. (2006). Reducing
the racial achievement gap: A social-psychological interven-tion. Science, 313, 1307-1310.

Shnabel, N., Purdie-Vaughns, V., Cook, J. E., Garcia, J., & Cohen, G. L. (2013). Demystifying values-affirmation interventions: Writing about social belonging is a key to buffering against identity threat. Personality and Social Psychology Bulletin,

Walton, G. M., & Spencer, S. J. (2009). Latent ability: Grades and test
scores systematically underestimate the intellectual ability of negatively stereotyped students. Psychological Science, 20, 1132-1139.

A great deal has been written about the impact of retrieval practice on memory. That's because the effect is sizable, it has been replicated many times (Agarwal, Bain & Chamberlain, 2012) and it seems to lead not just to better memory but deeper memory that supports transfer (e.g., McDaniel et al, 2013; Rohrer et al, 2010). 

("Retrieval practice" is less catchy than the initial name--testing effect. It was renamed both to emphasize that it doesn't matter whether you try to remember for the sake of a test or some other reason and because "testing effect" led some observers to throw up their hands and say "do we really need more tests?")

Now researchers (Szpunar, Khan, & Schacter, 2013) have reported testing as a potentially powerful ally in online learning. College students frequently report difficulty in maintaining attention during lectures, and that problem seems to be exacerbated when the lecture occurs on video.

In this experiment subjects were asked to learn from a 21 minute video lecture on statistics. They were also told that the lecture would be divided in 4 parts, separated by a break. During the break they would perform math problems for a minute, and then would either do more math problems for two more minutes ("untested group"), they would be quizzed for two minutes on the material they had just learned ("tested group"), or they would review by seeing questions with the answers provided ("restudy group.")

Subjects were told that whether or not they were quizzed would be randomly determined for each segment; in fact, the same thing happened for an individual subject after each segment except that each was tested after the fourth segment.

So note that all subjects had reason to think that they might be tested at any time.

There were a few interesting findings. First, tested students took more notes than other students, and reported that their minds wandered less during the lecture.

The reduction in mind-wandering and/or increase in note-taking paid off--the tested subjects outperformed the restudy and the untested subjects when they were quizzed on the fourth, final segment.

The researchers added another clever measure. There was a final test on all the material, and they asked subjects how anxious they felt about it. Perhaps the frequent testing made learning rather nerve wracking. In fact, the opposite result was observed: tested students were less anxious about the final test. (And in fact performed better: tested = 90%, restudy = 76%, nontested = 68%).

We shouldn't get out in front of this result. This was just a 21 minute lecture, and it's possible that the benefit to attention of testing will wash out under conditions that more closely resemble an on-line course (i.e., longer lectures delivered a few time each week.) Still, it's a promising start of an answer to a difficult problem.

References

Agarwal, P. K., Bain, P. M., & Chamberlain, R. W. (2012). The value of applied research: Retrieval practice improves classroom learning and recommendations from a teacher, a principal, and a scientist. Educational Psychology Review, 24,  437-448.

McDaniel, M. A., Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L. (2013). Quizzing in middle-school science: Successful transfer performance on classroom exams. Applied Cognitive Psychology. Published online Feb. 25

Rohrer, D., Taylor, K., & Sholar, B. (2010). Tests enhance the transfer of learning. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36, 233-239.

Szpunar, K. K., Khan, N. &, & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences, published online April 1, 2013 doi:10.1073/pnas.122176411

What people learn (or don't) from games is such a vibrant research area we can expect fairly frequent literature reviews. It's been about a year since the last one, so I guess we're due.

The last time I blogged on this topic Cedar Riener  remarked that it's sort of silly to frame the question as "does gaming work?" It depends on the game.

The category is so broad it can include a huge variety of experiences for students. If there were NO games from which kids seemed to learn anything, you probably ought not to conclude "kids can't learn from games." To do so would be to conclude that distribution of learning for all possible games and all possible teaching would look something like this.

But this pattern of data seems highly unlikely. It seems much more probable that the distributions overlap more, and that whether kids learn more from gaming or traditional teaching is a function of the qualities of each.

So what's the point of a meta-analysis that poses the question "do kids learn more from gaming or traditional teaching?

I think of these reviews not as letting us know whether kids can learn from games, but as an overview of where we are--just how effective are the serious games offered to students?

The latest meta-analysis (Wouters et al, 2013) includes data from 56 studies and examined both learning outcomes (77 effect sizes), retention (17 effect sizes) and motivation (31 effect sizes).

The headline results featured in the abstract is "games work!" Games are reported to be superior to conventional instruction in terms of learning (d = 0.29) and retention (d = .36) but somewhat surprisingly, not motivation (d = .26).

The authors examined a large set of moderator variables and this is where things get interesting. Here are a few of these findings:

1. Students learn more when playing games in groups than playing alone.
2. Peer-reviewed studies showed larger effects than others. (This analysis is meant to address the bias not to publish null results. . . but the interpretation in this case was clouded by small N's.)
3. Age of student had no impact.

But two of the most interesting moderators significantly modify the big conclusions.

First, gaming showed no advantage over conventional instruction when the experiment used random assignment. When non-random assignment was used, gaming showed a robust advantage. So it's possible (or even likely) that games in these studies were more effective only when they interacted with some factor in the gamer that is self-selected (or selected by the experimenter or teacher). And we don't know yet what that factor is.

Second the researchers say that gaming showed and advantage over "conventional instruction" but followup analyses show that gaming showed no advantage over what they called "passive instruction"--that it, the teacher talk or reading a textbook. All of the advantage accrued when games were compared to "active instruction," described as "methods that explicitly prompt learners to learning activities (e.g., exercises, hypertext training.)" So gaming (in this data set) is not really better than conventional instruction; it's better than one type of instruction (which in the US is probably less often encountered.)

So yeah, I think the question in this review is ill-posed. What we really want to know is how do we structure better games? That requires much more fine-grained experiments on the gaming experience, not blunt variables. This will be painstaking work.

Still, you've got to start somewhere and this article offers a useful snapshot of where we are.

EDIT 5:00 a.m. EST 2/11/13. In the original post I failed to make explicit another important conclusion--there may be caveats on when and how the games examined are superior to conventional instruction, but they were almost never worse. This is not an unreasonable bar, and as a group the games tested pass it.

Wouters, P, van Nimwegen, C, van Oostendorp, H., & van der Spek, E. G. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology. Advance online publication. doi: 10.1037/a0031311