Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

How can you know if it's *really* "research-based?"

7/18/2012

 
Picture
My new book, When Can You Trust the Experts: How to Tell Good Science from Bad in Education is now available. (There’s a link for a free download of Chapter 1 on this page.) EDIT (7:17 pm 7/18) I just saw that it's listed as "Recommended," with a micro-review on the Scientific American website.

I wrote the book out of frustration with a particular problem: the word “research” has become meaningless in education. Every product is claimed to be research-based. But we all know that can’t be the case. How are teachers and administrators supposed to know which claims are valid?

It’s notable that this problem exists in many other fields. However good your training, research doesn’t stand still. So how does a pediatrician who has been in practice 10 years know that what she learned in medical school as the optimal treatment for, say, croup, is still the best treatment?

The answer is NOT that the pediatrician is expected to comb through medical journals each night to keep up to date. Who has the time? Instead, reliable summaries are published (usually annually) that bring doctors up to date.

But there is no institution in education that publishes such summaries, and indeed, there’s not an institution that would have the credibility to do so. The Institute of Educational Sciences tried with the What Works Clearinghouse, but it has not been a success, mostly because the criteria it used to evaluate products was so stringent, and also because (at least initially) it was dogged by suspicions that its conclusions would not be politically neutral.

Teachers and administrators are on their own.

But doing a good job of evaluating research carries two requirements: you need a certain level of expertise in evaluating research (which requires knowing the existing research literature) AND you need a good deal of time.

The purpose of When Can You Trust the Experts is to offer a short cut, a work-around, so that you can size up the likelihood that a claim like “all the research supports it” is actually true. 

The first half of the book focuses on what cues we use to tell us “this is probably true,” and how they can be misleading. You need to understand why we believe what we do in order to make it less likely that you’ll believe something that is false,

I also describe at some length how when can know when science might help with a particular problem and when it can’t. I think science is terrific for answering certain types of questions, but there are many questions it can’t answer.

The second half of the book describes the short cut, which consists of four steps:

  1. Strip it. Clear away the verbiage and look at the actual claim. What exactly is the claim suggesting a teacher or parent should do, and what outcome is promised?
  2. Trace it. Who created this idea, and what have others said about it? It's common to believe something because an authority confirms it, and this is often a reasonable thing to do. I think people rely heavily on credentials when evaluating education research, but I argue that it’s a weak indicator of truth.
  3. Analyze it. Why are you being asked to believe the claim is true? What evidence is offered, and how does the claim square with your own experience?
  4. Should I do it? You're not going to adopt every educational program that is scientifically backed, and it may make sense to adopt one that has not been scientifically evaluated.
 My purpose is not to turn you into a researcher by reading one book. Anyone who tells you that’s possible is trying to pick your pocket. What I can promise is this: whatever you current level of research sophistication, I can help you ask better questions about an educational product, and think through the logic of adopting it for your home, classroom, or school district.

I should also mention that When Can You Trust the Experts makes a splendid gift for weddings, anniversaries, birthdays, christenings, etc.

Free download of Chapter 1 on this page.


Obi-Wandreas
7/18/2012 02:56:22 pm

Thank you very much for this. Most people I have worked with repeat "research-based" the way a zombie says "brains!" They have no concept that not all research is of equal quality. It will be invaluable to have some help in making an actual rational judgement.

Holly Shapiro link
7/19/2012 02:31:19 am

Obe-Wandreas, that was funny. In a sad way. I also thank you, Dan Willingham I will be purchasing several copies and handing them out to parents. This is much needed. Brains! Sorry, I'm still laughing.

Bill Brown link
7/19/2012 01:00:16 pm

I just finished the book a few days ago and I really liked it. Its application to claims outside of education was a little murky, less explicit than I was hoping. I loved your first book even though I am not a teacher because I could immediately transfer your ideas about learning to learning in general.

Did you write this specifically with a target of educational research or did you intend this to apply more generally?

Dan Willingham
7/19/2012 01:41:31 pm

@Bill I thought long and hard about trying to make it more general and in the end decided it was too hard (for me at least). Methods and standards in different scientific disciplines vary too much. Also, the need seems less acute. On a big question--is there a vaccine/autism link, for example--finding the opinion of the majority of scientists is easy, because the relevant scientific bodies publish statements on such questions. So if you want to know "what does the research say?" you've got a fairly reliable guide. If that were available in education I likely would not have written the book.

Bill Brown link
7/20/2012 06:35:43 am

I completely understand. Plus, applying it more generally would be a task for philosophers specializing in epistemology. It was a worthwhile book and I enjoyed it. Thanks for clarifying!

Pete yeomans link
7/20/2012 11:18:55 am

There is a way past this lack of research informed summary. We are a new community seeking to encourage teachers to contribute their small part of reading to the knowledge base. Have a look please

James Koontz link
7/25/2012 03:12:14 pm

Pete, glad to find your site, I'm just beginning a like-minded blog of my own. Hopefully, books like Daniel Willingham's will really help build a critical mass of educators that understand how to conduct and evaluate research.

Jimmy link
7/26/2012 08:23:51 pm

I read your post and i understand the topic you discussed above. As I always interested to know this. anyway, thanks for bringing this.

JMcK
8/7/2012 07:50:45 am

After reading I think we need a new word for research based. Since we can't actually recreate the research, and many Changes are not researched at all, perhaps we should use more cautionary language. We might better refer to our practices as "research related" or "research referenced" rather than research based.

Matthew link
8/26/2012 03:31:48 pm

I read Daniel Kahneman's "Thinking Fast and Slow" earlier this summer so your points on peripheral clues, and confirmation bias in the sample chapter landed on well-prepared ground.

Looking forward to reading the full book shortly.

I can only hope more parents and members of the tax-paying public pick it up. The alternate seems a world where we just prove ever more frequently the truth of the phrase "there's a sucker born every minute."

homemade baby gifts link
8/28/2012 06:27:43 am

Thank you so much for posting this! I really appreciate it. :)


Comments are closed.

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    April 2022
    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory