Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

Measurement of non-cognitive factors

1/21/2013

 
An experiment is a question which science poses to Nature, and a measurement is the recording of nature’s answer. --Max Planck

You can't do science without measurement. That blunt fact might give pause when people emphasize non-cognitive factors in student success and in efforts to boost student success.

"Non-cognitive factors" is a misleading but entrenched catch-all term for factors such as motivation, grit, self-regulation, social skills. . .  in short, mental constructs that we think contribute to student success, but that don't contribute directly to the sorts of academic outcomes we measure, in the way that, say, vocabulary or working memory do.
Picture
Non-cognitive factors have become hip. (Honestly, if I hear about the Marshmallow Study just one more time, I'm going to become seriously dysregulated) and there are plenty of data to show that researchers are on to something important. But are they on to anything that that educators are likely to be able to use in the next few years? Or are we going to be defeated by the measurement problem ?

There is a problem, there's little doubt. A term like "self-regulation" is used in different senses: the ability to maintain attention in the face of distraction, the inhibition of learned or automatic responses, or the quelching of emotional responses. The relation among them is not clear.
Further, these might be measured by self-ratings, teacher ratings, or various behavioral tasks.

Picture
But surprisingly enough, different measures do correlate, indicating that there is a shared core construct (Sitzman & Ely, 2011). And Angela Duckworth (Duckworth & Quinn, 2009) has made headway in developing a standard measure of grit (distinguished from self-control by its emphasis on the pursuit of a long-term goal).

So the measurement problem in non-cognitive factors shouldn't be overstated. We're not at ground-zero on the problem. At the same time, we're far from agreed-upon measures. Just how big a problem is that?

It depends on what you want to do.

If you want to do science, it's not a problem at all. It's the normal situation. That may seem odd: how can we study self-regulation if we don't have a clear idea of what it is? Crisp definitions of constructs and taxonomies of how they relate are not prerequisites for doing science. They are the outcome of doing science. We fumble along with provisional definitions and refine them as we go along.

The problem of measurement seems more troubling for education interventions.

Suppose I'm trying to improve student achievement by increasing students' resilience in the face of failure. My intervention is to have preschool teachers model a resilient attitude toward failure and to talk about failure as a learning experience. Don't I need to be able to measure student resilience in order to evaluate whether my intervention works?

Ideally, yes, but that lack may not be an experimental deal-breaker.

My real interest is student outcomes like grades, attendance, dropout, completion of assignments, class participation and so on. There is no reason not to measure these as my outcome variables. The disadvantage is that there are surely many factors that contribute to each outcome, not just resilience. So there will be more noise in my outcome measure and consequently I'll be more likely to conclude that my intervention does nothing when in fact it's helping.

The advantage is that I'm measuring the outcome I actually care about. Indeed, there would not be much point in crowing about my ability to improve my psychometrically sound measure of resilience if such improvement meant nothing to education.

There is a history of this approach in education. It was certainly possible to develop and test reading instruction programs before we understood and could measure important aspects of reading such as phonemic awareness.

In fact, our understanding of pre-literacy skills has been shaped not only by basic research, but by the success and failure of preschool interventions. The relationship between basic science and practical applications runs both ways.

So although the measurement problem is a troubling obstacle, it's neither atypical nor final.



References
Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the Short Grit Scale (GRIT–S). Journal of Personality Assessment, 91, 166-174.

Sitzmann, T, & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137,  421-442.


Steve Straight
1/21/2013 12:24:08 am

This is a great little post, Dan. For our community colleges in Connecticut, the legislature has passed a bill trying to require us to get rid of most developmental writing classes (our school now has three seemingly necessary levels). They proposed that all students have the right to take college-level classes (as opposed to, say, instituting regents exams needed to graduate from high school ready for college). And, they gave us one semester to come up with pilots, test them, and go with what worked best. Yowza. So this post helps my thinking about this huge, quick change very much. It's OK to fumble along with provisional solutions and refine them as we go along.

Michael Langdon link
1/21/2013 06:02:13 am

I have come to the conclusion that in order for our education system to truly improve, we need to enact the same consumer protection laws that exist for medicine and other professions. If anyone else agrees with this goal please let me know here.

crazedmummy
1/21/2013 07:50:05 am

But, if we don't understand what it is, then how do we improve it? My concern is not that we can't fumble along and make changes, but we don''t really know what we're changing. In science sometimes this gives us a lucky opportunity to find something new. Or else we mess up really badly. Want to control schizophrenic behavior? Jab this spike up the patient's nose and wave it about a bit. Calms them down wonderfully.
In education, I fear we are at that same level of lobotomy. We want outcomes like grades, attendance, dropout, completion of assignments, class participation, all of which indicate compliance rather than persistence or education. I consider that students who are highly compliant, strive for "all As," and spend their time pleasing the teacher, have given up the struggle or autonomy, and exhibit a lack of tenacity.

Michael Langdon link
1/21/2013 08:48:30 am

When it comes to assessments there are two measures: reliability and validity. So, what measures are we using and are they valid indicators of what? Now we are on the validity part and what we are discovering is that when it comes to education we don't really know what is a valid indicator of what? We don't know if high graduation rates are a result of persistence or hand holding. The problem is that students know that there is pressure to get graduation rates up and keep them up. Educational statistics are like Google search results: easily manipulated by people with enough money.

Darin Schmidt
1/21/2013 10:59:24 am

To address Dr. Willingham's concern about educational interventions - I think a good step would be to acknowledge that GPA (which is made up of grades that probably include attendance, homework, collaborative work, etc.) is a better predictor of future performance than standardized tests. A lot of education reforms focus on test prep, testing out, and using virtual classes for credit recovery. Anybody who buys into the work of Tough or the "Crossing the Finish Line" authors would hesitate to go down the road of testing up and out.

Michael Langdon link
1/22/2013 08:09:46 am

I disagree. Testing can be effective if done properly; however, most testing is done on the cheap. Board certification exams are an example of an effective method of testing. The problem is that any solution is going to be extremely limited in cost. Taxpayers want cheap not correct.

Tom Sundstrom link
1/21/2013 12:17:02 pm

If improving student achievement in cognitive areas is the goal, i.e. higher test scores, and non-cognitive skills are important for that achievement, then non-cognitive skills should be part of the normal curriculum. While they aren't really part of the knowledge and skill content, they are a big factor in the “how” part of learning achievement.

It doesn't seem as important to measure these skills directly as it is to teach them and reap the indirect benefit of higher cognitive achievement. Do we know how to effectively teach non-cognitive skills? How much of this teaching is happening in today’s classrooms?

Mike G
1/22/2013 02:00:16 am

Hi Tom,

Scott Seider describes a few programs in his new book. I review it, along with Paul Tough's, here.

http://www.commonwealthmagazine.org/Departments/Book-Reviews/2013/Winter/Content-of-their-character.aspx

Tom Sundstrom link
1/22/2013 06:53:18 am

Mike,

Thanks for the reply and the link. In reading your book review, I was struck by the known significance of these non-cognitive factors contrasted with the minimal scientific understandings of their mechanisms, interactions and methods of teaching. Intuitively, we know they are a critical ingredient for high learning achievement and life success. As an engineer, I’m troubled by the mushiness of something so clearly important.

Your book review comment regarding teachers having their plates full is true, so we need to be careful about additions. But, there must but at least one good recipe for a balanced education approach with focus on both cognitive and non-cognitive skills that just fills the teacher’s plate. Teachers might not be asked to teach non-cognitive skills explicitly as a unit of instruction, but model it as their own consistent behavior and deeds.

Tom Sundstrom link
1/22/2013 06:54:20 am

Mike,

Thanks for the reply and the link. In reading your book review, I was struck by the known significance of these non-cognitive factors contrasted with the minimal scientific understandings of their mechanisms, interactions and methods of teaching. Intuitively, we know they are a critical ingredient for high learning achievement and life success. As an engineer, I’m troubled by the mushiness of something so clearly important.

Your book review comment regarding teachers having their plates full is true, so we need to be careful about additions. But, there must but at least one good recipe for a balanced education approach with focus on both cognitive and non-cognitive skills that just fills the teacher’s plate. Teachers might not be asked to teach non-cognitive skills explicitly as a unit of instruction, but model it as their own consistent behavior and deeds.

Tom Sundstrom link
1/22/2013 06:54:45 am

Mike,

Thanks for the reply and the link. In reading your book review, I was struck by the known significance of these non-cognitive factors contrasted with the minimal scientific understandings of their mechanisms, interactions and methods of teaching. Intuitively, we know they are a critical ingredient for high learning achievement and life success. As an engineer, I’m troubled by the mushiness of something so clearly important.

Your book review comment regarding teachers having their plates full is true, so we need to be careful about additions. But, there must but at least one good recipe for a balanced education approach with focus on both cognitive and non-cognitive skills that just fills the teacher’s plate. Teachers might not be asked to teach non-cognitive skills explicitly as a unit of instruction, but model it as their own consistent behavior and deeds.

Illyrian
1/21/2013 02:35:29 pm

The problem with non-cognitive factors is not that we still don't yet agree upon the exact constructs of these factors or their measures. We know that motivation, self-control, delayed gratification, or having long-term goals (grit) play a significant role on the educational outcomes, grades, retention, and graduation rates. To me, they are somehow related to personality traits and habits of mind. However, it is not what they are or how we measure them that is going to help students improve their performance. Rather is: how can we direct students toward them and make them have these good habits of mind.

Michael Langdon link
1/22/2013 07:34:41 am

Actually, Martin Seligmann's work on learned helplessness provides a great many ways to measure resiliency in a cognitive manner. His work was instrumental in improving the productivity of MetLife's sales force. They call it, CAVEing, which stands for Content Analysis of Verbatim Explanations I believe. I you Google it you will see.

Seligmann's book, Learned Optimism is a great place to start.

Matthew
1/23/2013 12:49:57 am

Tough contrasts the response to GRIT at Riverdale - the high-performing private school in his book with that of the KIPP schools: They decide not to measure it. Why?

Because for high SES families who look at education as a way to ensure their kids get (or stay) ahead, GRIT would just become another test to ace. The school head feared a rash of $300 an hour "GRIT" tutors.

So the subjects' response to being measured is, just as with traditional metrics like state tests, really important to consider.

And FWIW I think Seligman's work with high schools in Swarthmore PA did not lead to the results they expected. Kids gained vocabulary around character strengths, but fewer of the strengths themselves, at least in the way his team measured it.


Comments are closed.

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory