Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

Meta-analysis: Learning from Gaming

2/10/2013

 
What people learn (or don't) from games is such a vibrant research area we can expect fairly frequent literature reviews. It's been about a year since the last one, so I guess we're due.

The last time I blogged on this topic Cedar Riener  remarked that it's sort of silly to frame the question as "does gaming work?" It depends on the game.

The category is so broad it can include a huge variety of experiences for students. If there were NO games from which kids seemed to learn anything, you probably ought not to conclude "kids can't learn from games." To do so would be to conclude that distribution of learning for all possible games and all possible teaching would look something like this.
Picture
But this pattern of data seems highly unlikely. It seems much more probable that the distributions overlap more, and that whether kids learn more from gaming or traditional teaching is a function of the qualities of each.

So what's the point of a meta-analysis that poses the question "do kids learn more from gaming or traditional teaching?

I think of these reviews not as letting us know whether kids can learn from games, but as an overview of where we are--just how effective are the serious games offered to students?
Picture
The latest meta-analysis (Wouters et al, 2013) includes data from 56 studies and examined both learning outcomes (77 effect sizes), retention (17 effect sizes) and motivation (31 effect sizes).

The headline results featured in the abstract is "games work!" Games are reported to be superior to conventional instruction in terms of learning (d = 0.29) and retention (d = .36) but somewhat surprisingly, not motivation (d = .26).

The authors examined a large set of moderator variables and this is where things get interesting. Here are a few of these findings:
  1. Students learn more when playing games in groups than playing alone.
  2. Peer-reviewed studies showed larger effects than others. (This analysis is meant to address the bias not to publish null results. . . but the interpretation in this case was clouded by small N's.)
  3. Age of student had no impact.

But two of the most interesting moderators significantly modify the big conclusions.

First, gaming showed no advantage over conventional instruction when the experiment used random assignment. When non-random assignment was used, gaming showed a robust advantage. So it's possible (or even likely) that games in these studies were more effective only when they interacted with some factor in the gamer that is self-selected (or selected by the experimenter or teacher). And we don't know yet what that factor is.

Second the researchers say that gaming showed and advantage over "conventional instruction" but followup analyses show that gaming showed no advantage over what they called "passive instruction"--that it, the teacher talk or reading a textbook. All of the advantage accrued when games were compared to "active instruction," described as "methods that explicitly prompt learners to learning activities (e.g., exercises, hypertext training.)" So gaming (in this data set) is not really better than conventional instruction; it's better than one type of instruction (which in the US is probably less often encountered.)

So yeah, I think the question in this review is ill-posed. What we really want to know is how do we structure better games? That requires much more fine-grained experiments on the gaming experience, not blunt variables. This will be painstaking work.

Still, you've got to start somewhere and this article offers a useful snapshot of where we are.

EDIT 5:00 a.m. EST 2/11/13. In the original post I failed to make explicit another important conclusion--there may be caveats on when and how the games examined are superior to conventional instruction, but they were almost never worse. This is not an unreasonable bar, and as a group the games tested pass it.

Wouters, P, van Nimwegen, C, van Oostendorp, H., & van der Spek, E. G. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology. Advance online publication. doi: 10.1037/a0031311

Darin Schmidt link
2/10/2013 09:34:05 pm

"...followup analyses show that gaming showed no advantage over what they called "passive instruction"--that it, the teacher talk or reading a textbook." This finding is persistent and ignored, and has been for a long time. TIMMS research in the 90's found a striking difference between the US and Japan in the amount of thinking students were doing. Recent comparisons with Finland reveal the same gap. Now we find the gap with games. Sit-n-git doesn't work, and it never has.

Dan Willingham
2/10/2013 10:27:13 pm

I think the effect is in the opposite direction than you thought it was. . passive = games and both are *better* than active.

Darin Schmidt link
2/10/2013 11:19:18 pm

I see what you're saying. I took the statement to mean that sit-n-git is no better with games than with lectures and reading. That wouldn't surprise me. I would be surprised to find that passive instruction is better than active learning. I'd have to see the methodology to be the least bit moved. This is a tricky knot to untangle, because poor instructors who fail to integrate content into active learning get mediocre results.

Annie Murphy Paul link
2/13/2013 04:02:30 pm

It sounds like you're looking for research that confirms your preexisting assumptions, Darin. The study's methodology was just fine when you thought it was agreeing with you.

Will Richardson link
2/10/2013 11:46:26 pm

Hey Dan,

One problem I have with these types of studies is we're not clear on what the "learning" part is. Are we talking about learning to multiply? Learning to think critically? Learning to learn from failure? My point is we're learning lots of different things when we play games, but some may be less quantifiable than others. So when we ask "do kids learn more from gaming or traditional teaching?" don't we also have to ask "What do we mean by learning?" (stolen from Seymour Sarason.)

Dan Willingham
2/11/2013 01:57:02 am

@Will I agree. Each researcher is going to come up with his or her own definition, measure. It almost feels like we have to take it game by game: "With *this* outcome measure, and *these* kids under *these* circumstances, we compared the game to *this* other method. . ."

Hal Pashler
2/11/2013 02:45:16 am

I am glad to learn of this study and find Dan's comments and the reader comments very interesting, but it all leaves me with the feeling "the categories and measures here are so heterogeneous that the topic is not really suitable for a meta-analysis." The medical equivalent would be "Are supplements or prescription medicines better for chronic disease?".

That said, it would be interesting to use this study to pull out, say, the top 5 positive learning effect sizes (Cohen's d) for games, and see what kind of learning was being measured, and whether the control group was a reasonable one. At least that would offer one useful perspective on game/learning.

Amanda Price
2/11/2013 02:30:34 am

If passive = gaming and both are better than active learning, then it sounds like conventional methods like lecture, which are becoming increasingly out of vogue, are actually preferable to the more challenging (for teacher and student) form of active learning. Or am I reading too much into this review?

Dan Willingham
2/11/2013 03:28:52 am

@Amanda I do think that's reading too much into it. First, if memory serves the direct comparison of passive/active wasn't made and we can't know for sure how it would have come out. This is a longstanding debate, as I'm sure you know. . . I've been looking at these data and have been surprised to see (1) there are fewer studies on this than I would have thought; (2) the data are more mixed than I would have thought.

Sylvie Hervieux link
2/11/2013 02:51:06 am

I agree that the definition of learning is crucial. In our field (learning how to write to young children), the effect of serious gaming seams to be very convincing. Our iOS app LetterSchool is currently being used by thousands of children around the world. We haven't conduct a scientific research yet, but all testimonies from teachers, parents and therapists are indicating a significant impact on the children’s handwriting.

We think the major influence factors for this are:

- By watching the animated letter formations, and by forming the letters and numbers themselves over and over again, children are able to fully internalize their shapes and functions.

- The kinesthetic practice, which occurs through fun and nonthreatening motor activity rather than passively listening to classroom instruction, reinforces the results.

- At home, parents don’t always know how they can best help their children learn to write.

- At school, overfull curricula leave little room for handwriting lessons.

- Handwriting lessons often make less than optimal use of the available time.

- Many schools consider handwriting skills a lesser subject

- Studies indicate that many teachers received poor handwriting instruction themselves.


Comments are closed.

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory