Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

The instruction manual for value-added data will be ignored

2/29/2012

 
Am I stupid if I can't turn on my stove? The picture below (or one very similar) appears in most textbooks on human factors psychology.
Picture
The arrangement of controls is spatially incompatible with the arrangement of stove elements, so if I want to turn on the back left element, I may very well turn on the front left one.

What's notable is that this stove likely came with an instruction book, describing which knob goes with which burner.

But something about that feels wrong. It feels like the designer of the stove should have known how my mind works, and taken that into account, rather than shrugging and saying "well, it's in the manual. It's not my fault if you don't read the manual."

The stove reminds me of value-added measures of teacher effectiveness.

Even the staunchest boosters of value-added measures agree that they should not be the whole story, that there should be multiple measures of teacher effectiveness. But I'm afraid that asking people to remember that fact is a little like asking people to remember which knob goes with which burner on their stove. It's not that people can't do it, but you are swimming upstream of the mind's biases.

To be clear, I don't think that there are data to prove this contention, but let me describe why I'm guessing it's true.

We're talking about a case of missing information: you tell people: "Teacher Smith's value-added score is X. By the way, value-added scores are incomplete as a measure of teacher effectiveness"

How do people interpret information that they know to be incomplete? It varies with the situation. Sometimes they assume the missing information is positive. ("I haven't heard that the roads are closed, so I guess all's well.") Sometimes they assume missing information is negative ("He left 'prior experience' blank, so I guess he doesn't have any.")

And sometimes missing information is forgotten or discounted. My guess--and I emphasize that it's a guess--is that will be the case here. I make this guess in part by analogy to the evaluation of college applicants.

A student's high school record has lots of "soft" components, the values of which are tricky to evaluate: participation in sports and clubs, leadership positions, recommendations from teachers. . .. even a student's grade point average must be evaluated in light of the difficulty of the courses taken and the competitiveness of the high school.

But then there's the SAT. It has the gloss of being numeric, and it is easy to make comparisons across students. Make no mistake, I believe that the SAT does what it's supposed to do--predict success in the freshman year of college. But it's often interpreted to be much more meaningful than that. That's the problem.

I'm afraid that value-added measures will have the same problem. They are produced via a fancy formula, they make it simple to make comparisons, and they are numeric, which can lead one to conclude that they are more precise than they really are.  And at this point, we don't even have any of the other "soft" measures to round out the picture of teacher effectiveness.

I don't think value-added measures are meaningless. But handing people value-added measures with the bland warning "these are incomplete" is like giving me a stove with a bad mapping plus an instruction booklet.

The solution to the stove problem is straightforward


Picture
The solution to teacher evaluation is not straightforward, and I won't attempt to resolve it in a blog posting.

My purpose here is simply to highlight the problem in publishing value-added data for individual teachers, with the caveat "these measures are incomplete." I predict that caveat will go unnoticed or be forgotten.

New York Times with part of the story on income and education

2/10/2012

 
An article in yesterday’s New York Times covered some recent research on the increasing education achievement gap between rich and poor. It’s worth a read, but it misses a couple of important points.      

     Regarding reasons for the gap, the article dwells on one hypothesis, commonly called the investment theory: richer families have more money to invest in their kids. (The article might have mentioned that richer families not only have more financial capital, but more human capital and social capital.) The article does not mention at all another major theory of the economics of educational achievement; stress theory. Kids (and parents) who live in poverty live under systemic stress. A great deal of research in the last ten years has shown that this stress has direct cognitive consequences for kids, and also affects how parents treat their kids. (Any parent knows that you’re not at your best when you’re stressed.) An open-access review article on this research can be found here.

      Another important point the article misses concerns what might be done. It ends with a gloomy quote from an expert: “No one has the slightest idea what will work. The cupboard is bare.”

     I think there is more reason for optimism, because other countries are doing a better job with this problem than we are. The OECD analyzes the PISA results by reported family SES. In virtually every country, high SES kids outperform low SES kids. But in some countries, the gap is smaller, and that’s it’s not just countries that have smaller income gaps.

      Economic inequality within a country is often measured with a statistic called the Gini coefficient which varies from 0 (everyone has the same net worth) to 1 (one person has all the money, and the other has nothing). Rich children score better than poor children in countries with large Gini coefficients (like the US) and the rich outscore the poor in countries with lower Gini coefficients (like Norway). Being poor predicts lower scores everywhere, but the disparity of wealth means more in the US than it does in other countries. What’s significant is that the relationship between income and test performance is stronger in the US than it is in most countries. (The US has the 3rd strongest relationship between income and student performance in Science and 10th highest for math, in the 2006 PISA results).

         Some countries, (e.g., Hong Kong), despite an enormous disparity between rich and poor, manage to even the playing field when the kids are at school. The US does a particularly poor job at this task; wealthy kids enjoy a huge advantage over poor kids. People generally argue that the US is different than Hong Kong, we’re a large, heteroogenous country, and so forth. All true, but the defeatist attitude won’t get us anywhere. We need more systematic study of how those countries solve the problem.

Forward>>

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory