Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

Working memory training: Are the studies accurate?

10/15/2012

 
Last June I posted a blog entry about training working memory, focusing on a study by Tom Redick and his colleagues, which concluded that training working memory might boost performance on whatever task was practiced, but it would not improve fluid intelligence.

(Measures of fluid intelligence are highly correlated with measures of working memory, and improving intelligence would be most people's purpose in undergoing working memory training.)

I recently received an email from Martin Walker, of MindSparke.com, which offers brain training. Walker sent me a polite email arguing that the study is not ecologically valid: that is, the conclusions may be accurate for the conditions used in the study, but the conditions used in the study do not match those typically encountered outside the laboratory. Here's the critical text of his email, reprinted with his permission:

"There is a significant problem with the design of the study that invalidates all of the hard work of the researchers--training frequency.  The paper states that the average participant completed his or her training in 46 days.  This is an average frequency of about 3 sessions per week.  In our experience this frequency is insufficient.  The original Jaeggi study enforced a training frequency of 5 days per week.  We recommend at least 4 or 5 days per week.

With the participants taking an average of 46 days to complete the training, the majority of the participants did not train with sufficient frequency to achieve transfer.  The standard deviation was 13.7 days which indicates that about 80% of the trainees trained less frequently than necessary.  What’s more, the training load was further diluted by forcing each session to start at n=1 (for the first four sessions) or n=2, rather than starting where the trainee last left off.
"

I forwarded the email to Tom Redick, who replied:

"Your comment about the frequency of training was something that, if not in the final version of the manuscript, was questioned during the review process. Perhaps it would’ve been better to have all subjects complete all 20 training sessions (plus the mid-test transfer session) within a shorter prescribed amount of time, which would have led to the frequency of training sessions being increased per week. Logistically, having subjects from off-campus come participate complicated matters, but we did that in an effort to ensure that our sample of young adults was broader in cognitive ability than other cognitive training studies that I’ve seen. This was particularly important given that our funding came from the Office of Naval Research – having all high-ability 18-22 year old Georgia Tech students would not be particularly informative for the application of dual n-back training to enlisted recruits in the Army and Marines.

However, I don’t really know of literature that indicates the frequency of training sessions is a moderating factor of the efficacy of cognitive training, especially in regard to dual n-back training. If you know of studies that indicate 4-5 days per week is more effective than 2-3 days week, I’d be interested in looking at it.

As mentioned in our article, the Anguera et al. (2012) article that did not include the matrix reasoning data reported in the technical report by Seidler et al. (2010) did not find transfer from dual n-back training to either BOMAT or RAPM [Bochumer Matrices Test and Raven's Advanced Progressive Matrices, both measures of fluid intelligence], despite the fact that “Participants came into the lab 4–5 days per week (average = 4.5 days) for approximately 25 min of training per session” (Anguera et al., 2012), for a minimum of 22 training sessions. In addition, Chooi and Thompson (2012) administered dual n-back to participants for either 8 or 20 days, and “Participants trained once a day (for about 30 min), four days a week”. They found no transfer to a battery of gF and gC tests, including RAPM.

In our data, I correlated the amount of dual n-back practice gain (using the same method as Jaeggi et al) during training and the number of days it took to finish all 20 practice sessions (and 1 mid-test session). I would never really trust a correlation of N = 24 subjects, but the correlation was r = -.05.'.


I re-analyzed our data, looking only at those dual n-back and visual search training subjects that completed the 20 training and 1 mid-test session within 23-43 days, meaning they did an average of at least 3 sessions of training per week. For the 8 gF tasks (the only ones I analyzed), there was no hint of an interaction or pattern suggesting transfer from dual n-back.

So to boil Redick's response down to a sentence, he's  pointing out that other studies have observed no impact on intelligence when using a training regimen closer to that advocated by Walker, and Redick finds no such effect in a follow-up analysis of his own data (although I'm betting he would acknowledge that the experiment was not designed to address this question, and so does not offer the most powerful means of addressing it.)

So it does not seem that training frequency is crucial. 

A final note: Walker commented in another email that customers of MindSparke consistently feel that the training helps, and Redick remarked that subjects in his experiments have the same impression. It just doesn't bear out in performance.




Chris
10/15/2012 04:34:37 am

I'd be interested to hear what your thoughts and reactions are to the Lumosity brain training program and its supporting research. It claims 25 million users and is based on research gathered or conducted by something called the Human Cognition Project. Thanks for all of your work to help the broader public better understand cognition.

Dan Willingham
10/15/2012 05:36:35 am

@chris A quick database search turned up no studies. But maybe another reader knows of something that's in press or in the works elsewhere.

Martin Walker link
10/15/2012 06:37:53 am

Thanks, Daniel.

I should also have mentioned that the original researchers -- Jaeggi, Buschkuehl, Jonides -- and others have completed several follow-up studies replicating the gains in fluid intelligence and narrowing down the factors that influence these gains.

Earlier this year for instance Jaeggi and Buschkuehl found greater benefits for some personality types when training with single n-back as opposed to dual n-back.

It seems critically important -- scientifically and methodologically -- to determine why some studies find transfer to fluid intelligence and others don't.

Best wishes,
Martin Walker
www.mindsparke.com

gwern link
10/23/2012 01:39:11 pm

> Earlier this year for instance Jaeggi and Buschkuehl found greater benefits for some personality types when training with single n-back as opposed to dual n-back.

They did? Wasn't that Studer-Luethi 2012, which was just a retrospective re-analysis of Jaeggi et al 2010?

Martin Walker link
10/24/2012 06:06:40 am

Jaeggi and Buschkuehl published a new study: http://www.sciencedirect.com/science/article/pii/S0191886912000815

Martin Walker link
10/24/2012 06:10:27 am

Sorry, Gwern. Now I see what you mean. It's not clear from the abstract that this was a retrospective analysis. My point still stands that subsequent studies have been mixed with some finding transfer, others not.

Dan Willingham
10/25/2012 08:18:22 am

And here's a new failure to replicate
http://www.sciencedirect.com/science/article/pii/S0160289612000839

gwern link
10/25/2012 08:29:45 am

That's not new at all, unless you missed Chooi's 2011 thesis.

KH
10/26/2012 08:09:02 pm

Here's successful replication
Jaušovec, N. and K. Jaušovec (2012). "Working memory training: Improving intelligence – Changing brain activity." Brain and Cognition 79(2): 96-106.

Actually I've done the experiment myself and it came out the result depends on neuroticism of the subject as mentioned by Studer-Luethi 2012, but I have to look into it furthur.

Another thing I want to point out is that the reliability of IQ test. Redick's APM is composed of just 12 problems and I doubt it has high reliability. I know he mentioned short form of APM in his paper, but I also want to point that the short form was composed of problems that showed high correlation with total score. So it must be much less that that particular short form.

Like I mentioned the test was 12 mins with 12 problems. But the Jaeggi's test was 18 problems with 12 mins and I see Jaušovec's test was even shorter. So I think that might be the source of the discrepancy.

Dan Willingham
10/26/2012 10:08:26 pm

@KH good points, thanks for raising them.


Comments are closed.

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    April 2022
    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory