Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

How qualitative research contributes

6/13/2014

 
This column was originally published on RealClearEducation.com on May 13, 2014.

Last week I discussed the case Carl Wieman made that education research and physics have more in common than people commonly appreciate. I mentioned in passing Wieman’s suggestion that random control trials—often referred to as the “gold standard” of evidence—are not the be-all and end-all of research, and that qualitative research contributes too. But I didn’t say how it contributes, and neither did Wieman (at least in the paper I discussed). Here, I offer a suggestion.

Qualitative research is usually contrasted with quantitative research. In quantitative research the dependent variable—that is, the outcome—is measured as a quantity, usually using a measurement that is purportedly objective. In qualitative research, the outcome is not measured as a quantity, but as a quality.

For example, suppose I’m curious to know what students think about their school’s use of technology. In a quantitative study, I might develop a questionnaire that solicits student’s opinions about, say, the math and reading software they’ve been using, and also asks students to compare them to the paper versions they used last year. I’d probably try to get most or all of the students in the school to respond. I would end up with ratings which I can treat as quantitative data.  

Or I could do a qualitative study, in which I use focus groups to solicit their opinions. The result of this research is not numeric ratings, but transcripts of what people said. I’d try to find common themes in what was said. Because focus groups are time-consuming, I’d probably talk to just a handful of students.

People who criticize qualitative research treat it as a weak version of quantitative methods. For example, a critic would point out that you can’t trust the results of the focus groups, because I might have, by chance, interviewed students with idiosyncratic views. Another problem is that focus groups are not very objective; responses are a product of the dynamic between the interviewer and the subjects, and among the subjects themselves.

These complaints are valid, or would be if we hoped to treat the focus group results the way we treat the outcomes of other experiments. I suggest we should not.

Here’s a simple graphic I used in a book to describe how science works. We begin with observations about the world. These can come from our casual, everyday observations or from previous experiments. We then try to synthesize those observations into a theory; we try to find simplifying rules that describe the many observations we’ve made. This theory can be used to generate a prediction, something we haven’t yet observed, but ought to. Then we test the prediction, and that test leads to a new observation, a new fact about the world. And we continue around the circle, always seeking to refine the theory.
Picture
The criticism of qualitative research is that it does not provide a very reliable test of a theory. But I think it’s better viewed as providing a new observation of the world.

An advantage of qualitative over quantitative data is the flexibility in how its collected. If I want to know what students think of the new technology instruction and I create a quantitative scale to measure it, I’m really guessing that I know the important dimensions of student opinion. I may try to capture their views on effectiveness and ease-of-use, for example, but what students really care about is the fact that so many websites are blocked.

Qualitative studies allow the subject to tell you what he thinks is important. For example in this qualitative study, students suggested that the schools policy on blocking websites affected their relationships with teachers—they felt untrusted. Maybe that’s the kind of thing a researcher would have been looking for, but I doubt it.

In addition, qualitative data tend to be richer. I’m letting the subject describe in his or her own words what she thinks, rather than asking her to select from reactions that I’ve framed.

Naturally, either type of research—qualitative or quantitative--can be poorly conducted. Each has its own rules of the road. It’s important to judge the quality of research by its own set of best-practice rules, and to judge it by how well it fulfils the purpose to which it is suited.

That’s why I believe qualitative research has an undeserved bad reputation. It is judged by standards of quantitative research, and deemed unable to serve purposes that ought to be served by quantitative research. But I agree with Wieman that qualitative research, well-conducted, makes a valuable contribution, one to which quantitative research is ill-suited.


Tom Berend
6/14/2014 12:34:25 am

There is excitement in education about 'Mixed Methods' research, which offers ways to triangulate between QUAL and QUAN at each step of the research process. That's how product developers work, iterating designs on both esthetic and performance feedback. But it's hard to find successful examples in academic research.

Juanes
6/19/2014 05:51:27 am

All this is based on foundations for a better understanding.
http://careers-schools.com

Margaret R. Roller link
6/23/2014 02:20:00 am

Thank you for this post. It is heartening to see an ever-increasing acknowledgement as well as thoughtful discussions of qualitative research, esp., within disciplines that have hung on to a traditionally pure quantitative model. The emergence of journals such as Qualitative Psychology and various academic papers bodes well for the future of qualitative in psychology. A book to be released in Dec-Jan discusses the application of quality criteria to qualitative research -- criteria that supports credible, analyzable, transparent, and useful qualitative outcomes. I also discuss these issues on my design blog at www.researchdesignreview.com. Thanks again.


Comments are closed.

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    April 2022
    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory