Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

Potentially big win in addressing the achievement gap

7/11/2016

 
Students from disadvantaged backgrounds and students who are the first in their families to attend college are at heightened risk to leave school without a degree. It's not a matter of students coming to school inadequately prepared--something else is at work (e.g., Steele, 1997). It's not merely disheartening for these students, there can be potentially grave financial repercussions for dropping out. 

Most colleges and universities have programs meant to help students with the process of transitioning, and they often focus on practical skills like choosing classes and study strategies on the assumption that navigating academic life is a significant part of the problem.

In the last few years researchers have focused on a quite different approach: offering students a "lay theory" of the college experience. A lay theory is a set of beliefs that are used to interpret one's experiences. In the case of college, two beliefs have been flagged as especially important: that the transition to college inevitably involves setbacks, and that these setbacks are temporary.

College always includes disappointments, both academic and social. A student fails a test, or a callous professor tells them that their writing is beyond hope. (My freshman year of college an English professor wrote this as the entire comment on my exam essay: "No. D" Students get lonely, and have trouble making friends.  For students who grew up in families where it was always assumed that they would attend college, such disappointments are dispiriting, but not threatening. The student may even wonder if he or she belongs in college, but that doubt likely doesn't last. For a student who did not grow up in an environment where it was taken for granted that they would graduate college that doubt may persist. They may think that they are not smart enough to succeed, that they are "not college material," or that their cultural background is not compatible with college. 

Researchers have sought ways to instill a lay theory of college that would change that interpretation, focusing on two ideas: setbacks in college are common  (and therefore should not be taken as a sign that you don't belong) and setbacks are temporary (so things will get better). 

Researchers have had some success with this intervention in smaller experiments (Stephens et al, 2014; Walton & Cohen, 2011. Now a new study (Yeager et al, 2016) suggests that a simple, inexpensive intervention works at scale. 

Before they matriculated at college, students participated in an activity taking just 30 minutes, administered over the Internet. They were told that it was to help them think about the transition to college, and that they would have the chance to share their experiences, perhaps helping future students. 

There were three experimental conditions. The social belonging condition provided information showing that feeling out of place is common in the transition to college, but most students do make friends and succeed academically. The growth mindset condition provided information showing that intelligence is malleable, and that student can succeed with effort, coupled with effective strategies. The third condition combined both strategies. In each case, students were asked to write an essay about how the information they read might apply to them, as a way of cementing the information in memory, and to help them imagine making it applicable to their own experience.

One experiment targeted the entering class of a large public university. As shown in the table below, the intervention improved retention. All three of the intervention conditions were equally effective. 
Picture
Another experiment administered the intervention at a selective private university. The figure shows that disadvantaged students receiving the intervention earned higher GPAs in their first year of college 
Picture
Consistent with the theory, advantaged students don't benefit from the intervention; they already believe that they can succeed at college, and that they belong there, so their lay theory of college is likely already similar to the one described in the intervention.

Two things are noteworthy about this experiment. First, the reduction in the achievement gap is quite sizable, on the order of 30-40%. Second, this intervention was remarkably brief, and remarkably inexpensive. Obviously this work needs to be replicated and the interventions fine-tuned. (The growth mindset intervention didn't really work in Experiment 1.) But if this finding holds up, it must be counted as a huge success for social scientists, and for David Yeager and Greg Walton in particular. 

Important new study of homework

7/5/2016

 
There's plenty of research on homework and the very brief version of the findings is probably well known to readers of this blog: homework has a modest effect on the academic achievement of older students, and no effect on younger students (Cooper et al, 2006).

In a way, this outcome seems odd. Practice is such an important part of certain types of skill, and much of homework is assigned for the purpose of practice. Why doesn't it help, or help more?

One explanation is that the homework assigned is not of very good quality, which could mean a lot of different things and absent more specificity sounds like a homework excuse. Another, better explanation is that practice doesn't do much unless there is rapid feedback, and that's usually absent at home. 

A third explanation is quite different, suggesting that the problem may lie in measurement. Most studies of homework efficacy have used student self-report of how much time they spend on homework. Maybe those reports are inaccurate. 

A new study indicates that this third explanation merits closer consideration. 

The researchers (Rawson, Stahovich & Mayer, in press) examined homework performance among three classes of undergraduate engineering students taking their first statics course. The homework assigned was typical for this sort of course; the atypical feature was that students were asked to complete their homework with Smartpens. These function like regular ink pens, but when coupled with special paper, they record time-stamped pen strokes.

The researchers were able to gather objective measures of time spent on homework, as well as other performance metrics.
Picture
A few of these measures proved interesting. For example, students who completed a lot of homework within 24 hours of the due date tended to earn lower course grades.

But the really interesting finding was a significant correlation of course grade and time spent on homework as measured by the Smartpen (r = .44) in the face of NO correlation between course grade and time spent on homework as reported by the students (r = -.16).

The relationship between homework and course grades is not the news. This is a college course and no matter what the format, it's only going to meet a few hours each week, and students will be expected to do a great deal of work on their own.

The news is that students were poor at reporting their time spent on homework; 88% reported more than the Smartpen showed they had actually spent. The correlation of actual time and reported time ranged from r = .16 to r = .35 for the three cohorts. 

In other words, with such a noisy measure of time spent on homework, there was little hope of observing a reliable relationship of homework with a course outcome. This finding ought to call into question much of the prior research on homework. 

Please don't take this blog posting as an enthusiastic endorsement of homework. For one thing, this literature seems pretty narrow in focusing solely on academic performance outcomes, given that many teachers and parents have other goals for homework such as increased self-directedness. For another thing, even if it were shown the certain types of homework led to certain types of improvement in academic outcomes, that doesn't mean every school and classroom ought to assign homework. That decision should be made in the context of broader goals.

But if teachers are going to assign homework, researchers should investigate its efficacy. This study should make us rethink how we interpret existing research in this area.

New report on self-regulation and social competence

12/8/2015

 

I (like everyone else) am always eager for documents that clearly summarize a large, complex literature. One such literature of urgent interest is the role of self-regulation in academic success. A new working paper from Transforming Education (full disclosure: I’m on their advisory board) does a great job of highlighting the important findings regarding non-cognitive skills, a not-very-precise term originating in economics that refers mostly to self-control and social competence.

The report is targeted at policymakers, but should be of interest to teachers and administrators as well.

The paper is organized around nine “headlines;” these are conclusions that the authors suggest are justified by the research literature. These headlines concern the relationship of non-cognitive skills to academics , careers, and general well-being.

1. Non-cognitive skills predict high school and college completion. 
2. Students with strong non-cognitive skills have greater academic achievement within K-12 schooling and college.
3. Fostering non-cognitive skills as early as preschool has both immediate and long-term impact.
4. Employers value non-cognitive skills and seek employees who have them.
5. Higher non-cognitive skills predict a greater likelihood of being employed.
6. Stronger non-cognitive skills in childhood predict higher adult earning and greater financial stability.
7 Adults with stronger non-cognitive skills are less likely to commit a crime and be incarcerated.
8. Strong non-cognitive skills decrease the likelihood of being a single or unplanned teenage parent.
9. The positive health effects associated with stronger non-cognitive skills include reduced mortality and lower rates of obesity, smoking, substance abuser, and mental health disorders.  

You not only get a brief, readable elaboration of each point, you also get the backing citations.
 
My only quibble is that, were I the author of this report, I would have been a bit more cautious in drawing a causal conclusion about the evidence of success in fostering non-cognitive skills in preschool (conclusion #3 above). It is of course possible that self-control is largely heritable and is changed little by the environment, so it’s important to know that the positive outcomes associated with non-cognitive skills can be promoted by practices in schools. The authors cite a 2014 report by Clancy Blair and Cybele Raver showing success, which is encouraging, but it is, according to Blair & Raver, the first experimental demonstration of.
 
That said, I encourage you download it, read it, and refer to it. It neatly sums up a complex and vital research literature. 

Single-sex schooling--tiny effects, if any

2/10/2014

 
The idea that students would learn better in single-sex classrooms seems logical. The typical arguments include
  • Boys find girls more distracting in class than they find other boys. Likewise, girls find boys more distracting.
  • Sex differences in math and science achievement are a product of social influence. Those influences will be reduced or eliminated if girls are in classrooms only with girls.
  • Boys dominate classroom discussion, and so girls are denied practice in articulating and defending their views.
  • Boys and girls have different brains, and therefore learn differently. If they are taught separately, teachers can tune their instruction to the way each sex learns.
The last of these is frequently overwrought and over-interpreted, but generally, these reasons seem plausible. But that obviously doesn't prove that single-sex education confers any advantage to students.

A 2005 report written for the Department of Education (Mael et al, 2005) reported mixed effects, but generally a positive conclusion for single-sex classrooms in short-run academic outcomes. There was no indication of a boost to longer-term outcomes.


A new study (Pahlke, Hyde, & Allison, 2014) reports a meta-analysis of 184 studies representing 1.6 million students in K-12 across 21 nations. The authors place considerable emphasis on the problem of control in this research. They end up concluding that, with proper controls, analyses show that single-sex classrooms don't help students much.
Picture
The challenge in this sort of work is that comparisons of single-sex and coed classrooms often do not use random assignment. Students (or parents) choose a single-sex classroom. So for this review, the authors distinguished between controlled experiments (the original study either used random assignment or made some attempt to measure and statistically account for associated variables) and uncontrolled studies.

In controlled studies, there were statistically reliably, but numerically quite modest positive effects of single-sex classrooms for both boys and girls in mathematics achievement, science achievement, and verbal achievement (Hedges g in all cases less than 0.10). Girls showed an edge in single-sex classes for math attitude, science achievement, and overall academic achievement, but again, the gains were modest. If one restricts the analysis to U.S. students, virtually all of these small effects disappear.

There was no effect for attitudes towards school, gender stereotyping, educational aspirations, self-concept, interpersonal relationships, or body image.

Picture
There were not enough controlled studies to examine aggression, body image, interpersonal relations, interest in STEM careers, science attitudes, or victimization.

It's also notable that there was no dosage effect: the advantage was no larger when all classes within a school were single-sex classes, compared to when a single class was.

The authors were also interested in evaluating whether single sex classes were effective for boys of color. They reported that there were not enough controlled studies to answer this question, but even restricting the analysis to uncontrolled studies, the effects were minimal.

When you consider the factors that we know contribute substantially to academic attitudes and performance--the student's prior academic achievement, the curriculum, the home environment, the teacher's skill--it's easy to believe that the sex of the other students would have a modest effect, if any.

That said, it
could be that a single sex school has a profound influence on a few students. A few years ago, friends of mine moved their 15 year old daughter to an all-girls school because she was "boy crazy." According to my friends, she didn't become any less interested in boys, but she did focus on work better during school hours. But then again it's possible my friends were kidding themselves.

References

Mael, F., Alonso, A., Gibson, D., Rogers, K., & Smith, M. (2005).
Single-sex versus coeducational schooling: A systematic review. Washington, DC: American Institutes for Research

Pahlke, E.,Hhyude, J. S, & Allison, C. M. (2014). The effects of single-sex cmopared with coeducational schooling on students' performance and attitudes: A meta-analysis. Psychological Bulletin, Advance online publication, http://dx.doi.org/10.1037/a0035740

What science teachers need to know

10/14/2013

 
The results of this experiment probably won't surprise you. What surprised me was the fact that we didn't already have data like this in hand.

The researchers (Sadler et al., 2013) tested 181 7th and 8th grade science teachers for their knowledge of physical science in fall, mid-year, and years end. They also tested their students (about 9,500) with the exact same instrument.

Each was a twenty-item multiple choice test. For 12 of the items, the wrong answers tapped a common misconception that previous research showed middle-schoolers often hold. For example, one common misconception is that burning produces no invisible gases. This question tapped that idea:
Picture
But the researchers didn't just ask the teachers to pick the right answer. They also asked teachers to pick the answer that they thought their students would pick.

What makes this study interesting is that it tests teacher subject-matter knowledge directly (instead of using a proxy like courses taken, or degrees) and that it directly measures one aspect of pedagogical content knowledge, namely, student misconceptions. The dependent measure of interest is student gain scores in content knowledge over the course of the year.

The results?

Teachers content knowledge was good, but not perfect. They got about 84% of the questions right.

Their knowledge of student misconceptions was not as good. Teachers correctly identified just 43% of those. (And their students had, as in previous studies, selected those incorrect items in high numbers.)

And what type of teacher knowledge matters to student learning? It turns out to interact with past student achievement, as measured by standard math and reading tests.
Picture
The graph shows gains in student knowledge, separated by items for which teachers have (or lack) various types of knowledge. Filled circles are for students who scored well on a math and reading test (high achievers), and open circles are students who scored poorly (low achievers)

Look first at learning for concepts without  a common misconception. If teachers have subject matter knowledge (SMK in the graph) students learn the concept better. In fact, low-achieving students learned nothing about a concept if teachers didn't know the concept themselves. High-achieving students did. The researchers speculate they may have learned the content from a textbook or other source.

For the strong misconception items, the low-achieving students learned very little, whatever the teacher knowledge. For high-achieving students, knowledge mattered, and they were most likely to learn when their teacher had both subject-matter knowledge and knew the misconceptions their students likely held (KoSM in the graph).

So the overall message is not that surprising. Students learn more when their teachers know the content, and when they can anticipate student misconceptions.

Somewhat more surprising (and saddening), low-achieving students are especially vulnerable when teachers lack knowledge. High-achieving students are more resilient.

There are limitations to this study, the most notable being that the sample is far from random (teachers were volunteers), and that the test was zero-stakes for all.

The strength was the direct measure of both types of knowledge, and that the researchers could examine the relationship of knowledge to performance at the level of individual items. One hopes we'll see more studies using this type of design.

Reference:
Sadler, P. M., Sonnert, G., Coyle, H.P., Cook-Smith, N., & Miller, J.L. (2013) Student learning in middle school science classrooms. American Educational Research Journal, 50, 1020-1049.

The association of tracking and math self-concept

9/30/2013

 
In Why Don't Students Like School? I pointed out that cognitive challenge is engaging if it's at the right level of difficulty, but boring if it's too easy or too hard. It sounds, then, like it would make sense to organize students into different classes based on their prior achievement.

It might make sense cognitively, but the literature shows that such a practice leads to bad outcomes for the kids in lower tracks. Those classes tend to have less demanding curricula and and lower expectations for achievement (e.g., Brunello & Checchi, 2007).

Further, assignment to tracks is often biased by race or social class (e.g., Maaz et al., 2007).

What tracking does to students self-perceptions has been less clear. A new international study (Chmielewski et al., 2013) examined data from the 2003 PISA data set to examine the association of different types of tracking and student self-perceptions of mathematics self-concept.

The authors compared systems with
  • Between school streaming: in which students with different levels of achievement are sent to different schools.
  • Within school streaming: in which students with different levels of achievement are put in different sequences of courses for all subjects.
  • Course-by-course tracking: in which students are assigned to more or less advanced courses within a school, depending on their achievement within a particular subject.

Controlling for individual achievement and the average achievement of the track or stream, the researchers found that course tracking is associated with worse self-perceptions among low-achieving students, but streaming is associated with better self-perceptions.

This figure shows the difference between the self perceptions of higher and lower achieving students in individual countries, sorted by the type of tracking system.
Picture
The data suggest that when students are tracked for some but not all of their courses, they compare their achievement to other, more advanced students, perhaps because they see these students more often. Students who are streamed within or between schools, in contrast, compare their abilities to their fellow stream-mates.

But why is there self-concept higher than higher-achieving students? This effect may be comparable to a more general phenomenon that people are poorer judges of their competence for tasks that they perform poorly. If you're not very good, you're not good enough to realize what you lack.

The authors do not suggest that between school steaming is the way to go (since it's associated with higher confidence). They note that the association is just the reverse of that seen in achievement: kids who stream between schools seem to take the biggest hit to achievement.

References

Brunello, G., & Checchi, D. (2007). Does school tracking affect equality of opportunity? New international evidence. Economic Policy, 22, 781–861.

Chmielewski, A. K., Dumont, H. Trautwein, U. (2013). Tracking effects depend on tracking type: An international comparison of students' mathematics self-concept. American Educatioal Research Journal, 50,  925-957.

Maaz, K., Trautwein, U., Ludtke, O., & Baumert, J. (2008). Educational transitions and differential learning environments: How explicit between-school tracking contributes to social inequality in educational outcomes. Child Developmental Perspectives, 2, 99–106.

Self-control Gone Wild?

9/9/2013

 
The cover story of latest New Republic wonders whether American educators have fallen in blind love with self-control. Author Elizabeth Weil thinks we have. Titled “American Schools Are Failing Nonconformist Kids: In Defense of the Wild Child” the article suggests that educators harping on self-regulation are really trying to turn kids into submissive little robots. And they do so because little robots are easier to control in the classroom.

But lazy teachers are not the only cause. Education policy makers are also to blame, according to Weil. She writes that “valorizing self-regulation shifts the focus away from an impersonal, overtaxed, and underfunded school system and places the burden for overcoming those shortcomings on its students.”
Picture
And the consequence of educators’ selfishness? Weil tells stories that amount to Self-Regulation Gone Wild. A boy has trouble sitting cross-legged in class—the teacher opines he should be tested because something must be wrong with him. During story time the author’s daughter doesn’t like to sit still and to raise her hand when she wants to speak. The teacher suggests occupational therapy.

I can see why Weil and her husband were angry when their daughter’s teacher suggested occupational therapy simply because the child’s behavior was an inconvenience to him. But I don’t take that to mean that there is necessarily a widespread problem in the psyche of American teachers. I take that to mean that their daughter’s teacher was acting like a selfish bastard.

The problem with stories, of course, is that there are stories to support nearly anything. For every story a parent could tell about a teacher diagnosing typical behavior as a problem, a teacher could tell a story about a child who really could do with some therapeutic help, and whose parents were oblivious to that fact.

What about evidence beyond stories?

Weil cites a study by Duncan et al (2007) that analyzed six large data sets and found social-emotional skills were poor predictors of later success.

She also points out that creativity among American school kids dropped between 1984 and 2008 (as measured by the Torrance Test of Creative Thinking) and she notes “Not coincidentally, that decrease happened as schools were becoming obsessed with self-regulation.”

There is a problem here. Weil uses different terms interchangeably: self-regulation, grit, social-emotional skills. They are not same thing. Self-regulation (most simply put) is the ability to hold back an impulse when you think that that the impulse will not serve other interests. (The marshmallow study would fit here.) Grit refers to dedication to a long-term goal, one that might take years to achieve, like winning a spelling bee or learning to play the piano proficiently. Hence, you can have lots of self-regulation but not be very gritty. Social emotional skills might have self-regulation as a component, but it refers to a broader complex of skills in interacting with others.

These are not niggling academic distinctions. Weil is right that some research indicates a link between socioemotional skills and desirable outcomes, some doesn’t. But there is quite a lot of research showing associations between self-control and positive outcomes for kids including academic outcomes, getting along with peers, parents, and teachers, and the avoidance of bad teen outcomes (early unwanted pregnancy, problems with drugs and alcohol, et al.). I reviewed those studies here. There is another literature showing associations of grit with positive outcomes (e.g., Duckworth et al, 2007).

Of course, those positive outcomes may carry a cost. We may be getting better test scores (and fewer drug and alcohol problems) but losing kids’ personalities. Weil calls on the reader’s schema of a “wild child,” that is, an irrepressible imp who may sometimes be exasperating, but whose very lack of self-regulation is the source of her creativity and personality.

Picture
But irrepressibility and exuberance is not perfectly inversely correlated with self-regulation. The purpose of self-regulation is not to lose your exuberance. It’s to recognize that sometimes it’s not in your own best interests to be exuberant. It’s adorable when your six year old is at a family picnic and impulsively practices her pas de chat because she cannot resist the Call of the Dance. It’s less adorable when it happens in class when everyone else is trying to listen to a story.

So there’s a case to be made that American society is going too far in emphasizing self-regulation. But the way to make it is not to suggest that the natural consequence of this emphasis is the crushing of children’s spirits because self-regulation is the same thing as no exuberance. The way to make the case is to show us that we’re overdoing self-regulation. Kids feel burdened, anxious, worried about their behavior.

Weil doesn’t have data that would bear on this point. I don’t either. But my perspective definitely differs from hers. When I visit classrooms or wander the aisles of Target, I do not feel that American kids are over-burdened by self-regulation.

As for the decline in creativity from 1984 and 2008 being linked to an increased focus on self-regulation…I have to disagree with Weil’s suggestion that it’s not a coincidence (setting aside the adequacy of the creativity measure). I think it might very well be a coincidence. Note that scores on the mathematics portion of the long-term NAEP increased during the same period. Why not suggest that kids improvement in a rigid, formulaic understanding of math inhibited their creativity?

Can we talk about important education issues without hyperbole? 

References

Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: perseverance and passion for long-term goals. Journal of personality and social psychology, 92(6), 1087.

Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., ... & Japel, C. (2007). School readiness and later achievement. Developmental psychology, 43(6), 1428.

Fighting stereotype threat in African American and in female students.

7/22/2013

 
Part of the fun and ongoing fascination of science of science is "the effect that ought not to work, yet does."

The impact of values of affirmation on academic performance is such an effect.

Values-affirmation "undoes" the effect of stereotype threat (also called identity threat). Stereotype threat occurs when a person is concerned about confirming a negative stereotype about his or her group. In other words a boy is so consumed with thinking "Everyone expects me to do poorly on this test because I'm African-American" that his performance actually is compromised (see Walton & Spencer, 2009 for a review).

One way to combat stereotype threat is to give the student better resources to deal with the threat--make the student feel more confident, more able to control the things that matter in his or her life.

That's where values affirmation comes in.

In this procedure, students are provided a list of values (e.g., relationships with family members, being good at art) and are asked to pick three that are most important to them and to write about why they are so important. In the control condition, students pick three values they imagine might be important to someone else.

Randomized control trials show that this brief intervention boosts school grades (e.g., Cohen et al, 2006).

Why?

One theory is that values affirmation gives students a greater sense of belonging, of being more connected to other people.

(The importance of social connection is an emerging theme in  other research areas. For example, you may have heard about the studies showing that people are less anxious when anticipating a painful electric shock if they are holding the hand of a friend or loved one.)

A new study (Shnabel et al, 2013) directly tested the idea that writing about social belonging might be a vital element in making values affirmation work.

In Experiment 1 they tested 169 Black and 186 White seventh graders in a correlational study. They did the values-affirmation writing exercise, as described above. The dependent measure was change in GPA (pre-intervention vs. post-intervention.) The experimetners found that writing about social belonging in the writing assignment was associated with a greater increase in GPA for Black students (but not for White students, indicating that the effect is due to reduction in stereotype threat.)

In Experiment 2, they used an experimental design, testing 62 male and 55 female college undergraduates on a standardized math test. Some were specifically told to write about social belonging and others were given standard affirmation writing instructions. Female students in the former group outscored those in the latter group. (And there was no effect for male students.)

The brevity of the intervention relative to the apparent duration of the effect still surprise me. But this new study gives some insight into why it works in the first place.

References:

Cohen, G. L., Garcia, J., Apfel, N., & Master, A. (2006). Reducing
the racial achievement gap: A social-psychological interven-tion. Science, 313, 1307-1310.

Shnabel, N., Purdie-Vaughns, V., Cook, J. E., Garcia, J., & Cohen, G. L. (2013). Demystifying values-affirmation interventions: Writing about social belonging is a key to buffering against identity threat. Personality and Social Psychology Bulletin,

Walton, G. M., & Spencer, S. J. (2009). Latent ability: Grades and test
scores systematically underestimate the intellectual ability of negatively stereotyped students. Psychological Science, 20, 1132-1139.

Testing helps maintain attention, reduce stress in online learning

4/8/2013

 
A great deal has been written about the impact of retrieval practice on memory. That's because the effect is sizable, it has been replicated many times (Agarwal, Bain & Chamberlain, 2012) and it seems to lead not just to better memory but deeper memory that supports transfer (e.g., McDaniel et al, 2013; Rohrer et al, 2010). 

("Retrieval practice" is less catchy than the initial name--testing effect. It was renamed both to emphasize that it doesn't matter whether you try to remember for the sake of a test or some other reason and because "testing effect" led some observers to throw up their hands and say "do we really need more tests?")

Now researchers (Szpunar, Khan, & Schacter, 2013) have reported testing as a potentially powerful ally in online learning. College students frequently report difficulty in maintaining attention during lectures, and that problem seems to be exacerbated when the lecture occurs on video.

In this experiment subjects were asked to learn from a 21 minute video lecture on statistics. They were also told that the lecture would be divided in 4 parts, separated by a break. During the break they would perform math problems for a minute, and then would either do more math problems for two more minutes ("untested group"), they would be quizzed for two minutes on the material they had just learned ("tested group"), or they would review by seeing questions with the answers provided ("restudy group.")

Subjects were told that whether or not they were quizzed would be randomly determined for each segment; in fact, the same thing happened for an individual subject after each segment except that each was tested after the fourth segment.

So note that all subjects had reason to think that they might be tested at any time.

There were a few interesting findings. First, tested students took more notes than other students, and reported that their minds wandered less during the lecture.
Picture
The reduction in mind-wandering and/or increase in note-taking paid off--the tested subjects outperformed the restudy and the untested subjects when they were quizzed on the fourth, final segment.
Picture
The researchers added another clever measure. There was a final test on all the material, and they asked subjects how anxious they felt about it. Perhaps the frequent testing made learning rather nerve wracking. In fact, the opposite result was observed: tested students were less anxious about the final test. (And in fact performed better: tested = 90%, restudy = 76%, nontested = 68%).

We shouldn't get out in front of this result. This was just a 21 minute lecture, and it's possible that the benefit to attention of testing will wash out under conditions that more closely resemble an on-line course (i.e., longer lectures delivered a few time each week.) Still, it's a promising start of an answer to a difficult problem.

References

Agarwal, P. K., Bain, P. M., & Chamberlain, R. W. (2012). The value of applied research: Retrieval practice improves classroom learning and recommendations from a teacher, a principal, and a scientist. Educational Psychology Review, 24,  437-448.

McDaniel, M. A., Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L. (2013). Quizzing in middle-school science: Successful transfer performance on classroom exams. Applied Cognitive Psychology. Published online Feb. 25

Rohrer, D., Taylor, K., & Sholar, B. (2010). Tests enhance the transfer of learning. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36, 233-239.

Szpunar, K. K., Khan, N. &, & Schacter, D. L. (2013). Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences, published online April 1, 2013 doi:10.1073/pnas.122176411

Meta-analysis: Learning from Gaming

2/10/2013

 
What people learn (or don't) from games is such a vibrant research area we can expect fairly frequent literature reviews. It's been about a year since the last one, so I guess we're due.

The last time I blogged on this topic Cedar Riener  remarked that it's sort of silly to frame the question as "does gaming work?" It depends on the game.

The category is so broad it can include a huge variety of experiences for students. If there were NO games from which kids seemed to learn anything, you probably ought not to conclude "kids can't learn from games." To do so would be to conclude that distribution of learning for all possible games and all possible teaching would look something like this.
Picture
But this pattern of data seems highly unlikely. It seems much more probable that the distributions overlap more, and that whether kids learn more from gaming or traditional teaching is a function of the qualities of each.

So what's the point of a meta-analysis that poses the question "do kids learn more from gaming or traditional teaching?

I think of these reviews not as letting us know whether kids can learn from games, but as an overview of where we are--just how effective are the serious games offered to students?
Picture
The latest meta-analysis (Wouters et al, 2013) includes data from 56 studies and examined both learning outcomes (77 effect sizes), retention (17 effect sizes) and motivation (31 effect sizes).

The headline results featured in the abstract is "games work!" Games are reported to be superior to conventional instruction in terms of learning (d = 0.29) and retention (d = .36) but somewhat surprisingly, not motivation (d = .26).

The authors examined a large set of moderator variables and this is where things get interesting. Here are a few of these findings:
  1. Students learn more when playing games in groups than playing alone.
  2. Peer-reviewed studies showed larger effects than others. (This analysis is meant to address the bias not to publish null results. . . but the interpretation in this case was clouded by small N's.)
  3. Age of student had no impact.

But two of the most interesting moderators significantly modify the big conclusions.

First, gaming showed no advantage over conventional instruction when the experiment used random assignment. When non-random assignment was used, gaming showed a robust advantage. So it's possible (or even likely) that games in these studies were more effective only when they interacted with some factor in the gamer that is self-selected (or selected by the experimenter or teacher). And we don't know yet what that factor is.

Second the researchers say that gaming showed and advantage over "conventional instruction" but followup analyses show that gaming showed no advantage over what they called "passive instruction"--that it, the teacher talk or reading a textbook. All of the advantage accrued when games were compared to "active instruction," described as "methods that explicitly prompt learners to learning activities (e.g., exercises, hypertext training.)" So gaming (in this data set) is not really better than conventional instruction; it's better than one type of instruction (which in the US is probably less often encountered.)

So yeah, I think the question in this review is ill-posed. What we really want to know is how do we structure better games? That requires much more fine-grained experiments on the gaming experience, not blunt variables. This will be painstaking work.

Still, you've got to start somewhere and this article offers a useful snapshot of where we are.

EDIT 5:00 a.m. EST 2/11/13. In the original post I failed to make explicit another important conclusion--there may be caveats on when and how the games examined are superior to conventional instruction, but they were almost never worse. This is not an unreasonable bar, and as a group the games tested pass it.

Wouters, P, van Nimwegen, C, van Oostendorp, H., & van der Spek, E. G. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology. Advance online publication. doi: 10.1037/a0031311

<<Previous

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    April 2022
    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory