Freakonomics and the application of science to education.

8/6/2012

Steven Levitt, of Freakonomics fame, has unwittingly provided an example of how science applied to education can go wrong.

On his blog, Levitt cites a study he and three colleagues published (as an NBER working paper). The researchers rewarded kids for trying hard on an exam. As Levitt notes, the goal of previous research has been to get kids to learn more. That wasn't the goal here. It was simply to get kids to try harder on the exam itself, to really show everything that they knew.

Among the findings: (1) it worked. Offering kids a payoff for good performance prompted better test scores; (2) it was more effective if, instead of offering a payoff for good performance, researchers gave them the payoff straight away and threatened to take it away if the student didn't get a good score (an instance of a well-known and robust effect called loss aversion); (3) children prefer different rewards at different ages. As Levitt puts it "With young kids, it is a lot cheaper to bribe them with trinkets like trophies and whoopee cushions, but cash is the only thing that works for the older students."

There are a lot of issues one could take up here, but I want to focus on Levitt's surprise that people don't like this plan. He writes "It is remarkable how offended people get when you pay students for doing well – so many negative emails and comments." Levitt's surprise gets at a central issue in the application of science to education.

Scientists are in the business of describing (and thereby enabling predictions of) the Natural world. One such set of phenomenona concerns when students put forth effort and when they don't.

Education is a not a scientific enterprise. The purpose is not to describe the world, but to change it, to make it more similar to some ideal that we envision. (I wrote about this distinction at some length in my new book. I also discussed on this brief video.)

Thus science is ideally value-neutral. Yes, scientists seldom live up to that ideal; they have a point of view that shapes how they interpret data, generate theories, etc., but neutrality is an agreed-upon goal, and lack of neutrality is a valid criticism of how someone does science.

Education, in contrast, must entail values, because it entails selecting goals. We want to change the world--we want kids to learn things--facts, skills, values. Well, which ones? There's no better or worse answer to this question from a scientific point of view.

A scientist may know something useful to educators and policymakers, once the educational goal is defined; i.e., the scientist offers information about the Natural world that can make it easier to move towards the stated goal. (For example, if the goal is that kids be able to count to 100 and to understand numbers by the end of preschool, the scientist may offer insights into how children come to understand cardinality.)

What scientists cannot do is use science to evaluate the wisdom of stated goals.

And now we come to people's hostility to Levitt's idea of rewards for academic work.

I'm guessing most people don't like the idea of rewards for the same reason I don't. I want my kids to see learning as a process that brings its own reward. I want my kids to see effort as a reflection of their character, to believe that they should give their all to any task that is their responsibility, even if the task doesn't interest them.

There is, of course, a large, well-known research literature on the effect of extrinsic rewards on motivation. Readers of this blog are probably already familiar with it--if so, skip the next paragraph.

The problem is one of attribution. When we observe other people act, we speculate on their motives. If I see two people gardening--one paid and the other unpaid--I'm likely to assume that one gardens because he's paid and the other because he enjoys gardening. It turns out that we make these attributions about our own behavior as well. If my child tries her hardest on a test she's likely to think "I'm the kind of kid who always does her best, even on tasks she don't care for." If you pay her for her performance she'll think "I'm the kind of kid who tries hard when she's paid." This research began in the 1970's and has held up very well. Kids work harder for rewards. . . until the rewards stop. Then they engage in the task even less than they did before the rewards started. I summarized some of this work here.

In the technical paper, Levitt cites some of the reviews of this research but downplays the threat, pointing out that when motivation is low to start with, there's not much danger of rewards lowering it further. That's true, and I've made a close argument: cash rewards might be used as a last-ditch effort for a child who has largely given up on school. But that would dictate using rewards only with kids who were not motivated to start, not in a blanket fashion as was done in Levitt's study. And I can't see concluding that elementary school kids were so unmotivated that they were otherwise impossible to reach.

In addressing the threat to student motivation with research, Levitt is approaching the issue in the right way (even if I think he's incorrect in how he does so.)

But on the blog (in contrast to the technical paper), Levitt addresses the threat in the wrong way. He skips the scientific argument and simply belittles the idea that parents might object to someone paying their child for academic work. He writes:

Perhaps the critics are right and the reason I’m so messed up is that my parents paid me $25 for every A that I got in junior high and high school. One thing is certain: since my only sources of income were those grade-related bribes and the money I could win off my friends playing poker, I tried a lot harder in high school than I would have without the cash incentives. Many middle-class families pay kids for grades, so why is it so controversial for other people to pay them?

I think Levitt is getting "so many negative emails and comments" because he's got scientific data to serve one type of goal (get kids to try hard on exams) the application of which conflicts with another goal (encourage kids to see academic work as its own reward). So he scoffs at the latter.

I see this blog entry as an object lesson for scientists. We offer something valuable--information about the Natural world--but we hold no status in deciding what to do with that information (i.e., setting goals).

In my opinion Levitt's blog entry shows he has a tin ear for the possibility that others do not share his goals for education. If scientitists are oblivious to or dismissive of those goals, they can expect not just angry emails, they can expect to be ignored.

Stuart Buck

8/6/2012 07:33:55 am

What do you make of all the back and forth between Deci and colleagues vs. Pierce/Cameron? E.g., http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2731358/ or http://rer.sagepub.com/content/66/1/39.abstract

More recently, see Fryer's student incentive where he expressly found that there was no effect on student motivation: http://www.edlabs.harvard.edu/pdf/studentincentives.pdf

Dan Willingham

8/6/2012 10:02:44 am

@Stuart; I think the weight of evidence is with Deci, as is the extent to which his interpretation squares with other social psych theory. That's not to say *all* the data show the predicted effect--there are def. exceptions. You could easily imagine this situation: a kid doesn't read much, does so solely for the $$, and then discovers "hey! this is really fun!" (The Fryer paper is not the best example, IMO, though he claimed it was, because the main data they had supporting the contention was student self-ratings of interest. More decisive are behaviors--the prediction would be that once you stop paying them to read books, they will read fewer.) In the summary of this lit. I wrote, I likened rewards to taking out a loan. There's very likely to be a cost. . .but taking out a loan can make good financial sense in the right circumstances, and so can extrinsic rewards.

Darin Schmidt

8/6/2012 10:23:22 am

In Don't Shoot the Dog, Pryor emphasizes the importance of fading rewards and putting them on a variable schedule in order to maintain behavior. I wonder if the research addresses this concept adequately?

David B

8/6/2012 02:39:18 pm

Lots of scientific studies have been done on rewards and their effect on motivation during the past 40 years. Overwhelmingly, these studies demonstrate that intrinsic motivation is undermined by rewards, i.e., after people have been rewarded, and the reward is withdrawn, they are less likely to do what they were rewarded to before the reward was given.

Alfie Kohn presents a conglomeration of such research in his book, Punished by Rewards. If Levitt is unaware of this research, he's being dishonest, lazy, or ignorant...take your pick.

David B

8/6/2012 02:40:58 pm

Sorry about the typo. I meant to say "...less likely to do what they were rewarded for..."

Stuart Buck

8/7/2012 03:08:36 am

David -- as Dan Willingham has demonstrated in the past, Kohn cannot be relied upon for fair descriptions of other scholars' original research. See http://www.britannica.com/blogs/2009/02/alfie-kohn-is-bad-for-you-and-dangerous-for-your-children/

Dan -- You mention conflicting goals, one of which is "encourage kids to see academic work as its own reward." What's the evidence that more than a tiny handful of, say, middle schoolers to high schoolers ever think this way?

After all, consider the following: Overt anti-intellectualism in our schools has been lamented from James Coleman's "The Adolescent Society" book in 1961 to my own "Acting White" (Yale, 2010), and schoolkids commonly ridicule high achievers as "acting white" or "geeks" or "nerds." There are depressingly few American adults who ever read non-fiction books that aren't diet books or celebrity biographies. Americans are more familiar with Snow White's seven dwarfs than with the Supreme Court (http://www.businesswire.com/news/home/20060814005496/en/National-Poll-Finds-Americans-Snow-Whites-Dwarfs), and in 2007, fully 31% of American adults could not even identify Dick Cheney as the Vice-President (http://www.people-press.org/2007/04/15/public-knowledge-of-current-affairs-little-changed-by-news-and-information-revolutions/). Even among college students, the vast majority do not really value knowledge for its own sake, which is why they are generally happy (rather than outraged) if class is canceled or if the professor reduces a 20-page paper to a 10-page paper.

To state it as provocatively as possible: worrying about undermining students' internal motivation to learn geometry or world history is like worrying about drowning when you're in the middle of the Sahara.

I wonder if this worry arises because a few of us really are internally motivated to love knowledge for its own sake (and do things like read statistics books for fun), and we tend to forget that most people do not feel the same way at all.

Dan Willingham link

8/7/2012 03:58:20 am

Stuart--I don't know the data on how many kids see academic work as it's own reward--obv. would depend on how you measure it. It's true that most of pop culture is not oriented in this direction at all, and there's lots of info that seems central that most people don't know. On the other hand, people choose to work puzzles -- crossword and sudoku -- in their spare time, history and science programs on t.v. have small (but nontrivial) audiences.

I'd like for some K-12 teachers to weigh in with their experiences--teachers, what do you think of the state of intellectual curiosity in kids today?

Myself, if I became convinced that intellectual motivation was irretrievably low, I'd rather work to figure out how to boost that figure, rather than to accept it and offer rewards as an alternative. At least not until we'd made an all-out effort in that direction!

Angie link

8/8/2012 04:57:14 am

For the past three years, I have been a regular substitute teacher working in grades K-5 for a school in the middle of our town, where our students come from families who struggle with instability and lower incomes. Young students often come from chaotic home lives and spend a great deal of time playing video games when they are not at school. They do not come pre-disposed to learn in a formal setting.

I have noticed that in the classrooms with teachers who are consistent managers and strong planners, students are excited about learning new content--for its own sake. I have seen this particularly in a Kindergarten class and in a fifth grade class.

The level of interest in the fifth grade with the same strong teacher varied from one year to the next. The fifth grade class I guest taught in the year before this was extremely excited about science, social studies, etc. as exhibited by percentage of hands raised and insightfulness of comments. The spark was there in this year's crop of fifth graders, but they just weren't at the level of the previous year's kids. However, they were productive and attentive. The teacher's conscientiousness in management gave them little choice.

The Kindergarteners this year with the high-quality instruction liked nothing better than listening to and discussing non-fiction books. They were very attentive, they remembered what we read, and they connected the information in succeeding lessons.

When discipline and organization are strong, and there is less chance for repeated disruption, students have little choice but to listen, and when they do pay attention, they find that they are hooked. These types of classrooms seem to appeal to children at all levels of ability.

On the other hand, when disruptions are constant, as in most classrooms, students tend to be frustrated and less inclined to engage. Strong planning without the consistent discipline piece boosts interest (I know a great planner, weak on discipline), but everyone including the teacher ends the day exhausted and experiences are less positive. I hope teachers everywhere will be reading Teach Like a Champion this summer.

K Patel link

8/9/2012 07:42:03 am

I'd like for some K-12 teachers to weigh in with their experiences--teachers, what do you think of the state of intellectual curiosity in kids today?

Years of teaching and recent experiences in "under performing" school classrooms convince me that most children are curious, excited about making discoveries and above all want to relate with adults and their peers. What matters is the material and the context by which I mean the manner in which material is presented to children. Some questions I ask: are children familiar with the material or is it something totally out of their range? Is it pitched at their developmental level? Is the situation conducive to inquiry - hard to engage children in a chaotic classroom. Monetary rewards seems a short term solution. There are alternatives for sure.

Roger Sweeny

8/7/2012 10:48:12 am

I'm science teacher in a relatively average high school in Massachusetts, the state that always comes out first or second in tests of academic achievement. My experience is that most students are not intrinsically motivated to learn what is being taught in my school.

Perhaps this is partly because they have deliberately been extrinsically motivated since they were young. They have all been told over and over, "To get a decent job, you must have a high school diploma. To get a good job, you need to go to college. To get a very good job, you need to get into and graduate from a selective college." How can students do that? They get decent, good, or very good grades.

It is extraordinarily difficult to assess how much a student has actually learned in high school. What we do instead is work on a "unit" for a few weeks. At the end of a unit, there is a "summative assessment." The material in that unit is then pretty much dropped unit the mid-term or final, at which point it is, um "reviewed." Teachers who do not do a fair amount of review find that lots of students--even the ones who did well on the summative assessments--have forgotten most of what was in the term's units.

Many students have developed the attitude, "Tell me what you want me to tell you. I will memorize it and tell it back to you on a test or paper or project. Then, I will do the same thing for the next unit. Do not, however, ask me to understand it or to retain it past the unit assessment."

This has been a successful strategy for them in the past and they will be very displeased if the teacher changes things so it no longer works. They will be frustrated and mad at the teacher. They will also get bad grades. No one wants bad grades, students, teachers, or administrators. So there is constant pressure to keep the "you tell me, I tell you, we move on" wheel turning.

I think kids are curious about lots of things: their future, their peers, how to be popular with others as friends or lovers, music, tv, movies, sports, and so much else. However, most post-pubescent students don't have a lot of intrinsic interest in the things that they are supposed to learn in high school. This is partly because it has been a job for so long and they are tired of the job. It is also because most of their high school courses are simplified college courses, and, to be somewhat unfair, most college courses are "academic"; they are the first step to becoming a professor in those disciplines. Most high school students are not looking to become professors.

Dan Willingham

8/8/2012 02:21:55 am

@Roger . . .thanks very much for this comment. I actually see the same attitude in many of the students I teach; "tell me what to do to get an A, and I'll do it." I also see it in many districts I visit, esp. wealthy ones. There, most teachers tell me that they strongly believe kids picked up this attitude at home.
BUT, like kids elsewhere, they are intrinsically interested in things, as you note: music, social status, and even what we would deem intellectual matters (when it's embedded in other things). How to transfer this to school work? I'm just not ready to believe that the only way out is to pay them.

Roger Sweeny

8/8/2012 07:15:48 am

We teachers would love to believe that "kids picked up this attitude at home." But I think that, mostly, we unintentionally taught them. To a large extent, that attitude WORKS. It gets them what they want, which is good grades, and eventually admission into a selective college.

We would love to intrinsically motivate them, but we learn from experience that we have limited ability in that regard. So we are willing the fall back on the external. It's always there. We rarely have to say it. "If you don't tell me what I want to hear, you will fail and spend the rest of your life saying, 'Do you want fries with that?' If you are really good at telling me what I want to hear, you will eventually have an interesting, good-paying job. Don't blow it."

As an alternative, we could try to teach kids what they are already intrinsically interested in learning. But as far as most of us in the business are concerned, what is worth learning is what college professors do. So that's not going to happen.

Roger Sweeny

8/7/2012 11:01:35 am

Perhaps one reason that so many people were disturbed by Levitt's post is that we do extrinsically motivate but we don't want to. We don't do it directly. We do it at one remove ("To get a good job, get a good education.") and we pretend that we aren't doing it at all. Levitt is pushing into an area that many of us have a buried guilt about.

Dan Willingham

8/8/2012 05:04:32 am

@Angie thanks for sharing your experiences. It's really cheering (and instructive) to hear what strong teachers can achieve. I would bet that these teachers not only had strong classroom management but also warm relationships with their students. . .true?

Angie link

8/8/2012 05:29:08 am

Daniel Willingham: I would bet that these teachers not only had strong classroom management but also warm relationships with their students. . .true?

Yes, for sure. Students bond with the teachers at the school, with few exceptions, but they really love these two teachers and their classrooms. I notice that they love to share their passions and hobbies in the fifth grade class; I think the teacher really takes a genuine interest. And they fill notebooks with writing because when they have downtime or a fun day, it's usually to "read or creative write."

8/15/2012 04:26:09 am

One part of the argument on paying students for grades that often gets overlooked is that most students receive a free education. If we want to replicate the natural world wherein people are paid for jobs well done, the we would need to require students to pay back some of their earnings to the people and institutions who made their work possible: teachers, tutors, schools, etc. It does not make much sense to give something away for free, then pay for an outcome of the free service.

Sherwood Botsford link

8/16/2012 05:15:59 am

I live in a community where the average level of completed education is grade 8. Some of this average is due to a large proportion of older folks. It's rural town. Lots of farmers. Lots of truckers. Lots of oil workers.

A person with a grade 10 education can get a job refurbishing drill bits for $18 an hour. $2 shift differential for night shift, lots of overtime, and a 30% production bonus paid quarterly. This works out to 60,000 per year. On the flip side, a person with 25 years of computer experience can get only $13 on a help desk.

A millwright can make $50/hour. That's a 4 year apprenticeship with a total of 32 weeks of class, the rest being work experience. The WE part of it typically is at $18, $21, $24, and $27 per hour plus overtime for anything over 40 hours a week, and double time for anything over 60 hours a week.

We have lots of cases in this province where someone graduates from university with a degree other than hard science or engineering, can't find a job, and goes to a technical institute to pick up a trade. NAIT even has an ad campaign based on this.

As to motivation:

In general I found in my career teaching that enthusiasm was infectious. If I acted excited about a subject the kids picked this up. If I was running through the motions, so did the kids.

Ideally people get wrapped up in something if it is right at the edge of their ability. Too easy and it is boring. To hard and it is frustrating.

Dan Willingham

8/16/2012 09:54:36 pm

@Sherwood: I really agree re: the infectious nature of enthusiasm--I've found that same thing in my teaching. At first I found it a little funny that students so often commented on my enthusiasm, and then later realized "If I seem bored, why in the world would *they* be interested?" I don't know why I didn't get it right away.

Mike G

8/16/2012 05:34:01 am

Great post and comments thread.

I wanted to tackle Stuart's question about the state of intellectual curiosity in kids.

A) Stuart, it feels similar to intellectual curiosity of kids for past 20 years. Ie, I'm not noticing any obvious uptick or downtick.

B) An analogy.

Let's say I wanted Stuart and Dan to learn the following:

*Organic chemistry
*History of Argentina
*Fractals
*The Complete Works of Edward Albee

Now the 2 of you are both intellectuals. Maybe even one of these is of legit interest to you. Presumably you could, if necessary, find SOME way to try to rationalize learning the other 3 or 4 topics.

But that would still be "forcing it." Mostly you'd think "I'm really not that interested in these topics."

Similarly, I think hoping for "intrinsic interest" in math, English, science, and history as a PRIMARY motivator in middle and high school for a typical kid -- just not plausible.

Instead, I think Dan has it right above:

*"Strong classroom management and warm relationships with their students" gets you from home plate to second base.

*Other aspects of "good teaching" get you to third base....curriculum, pedagogy, urgency, etc. And it also "naturally" grows the number of children who DEVELOP intrinsic interest in any given subject.

*It's still not easy to get to home plate with many reluctant students.

If incentives can play a role here, I'd love to know about how and under what conditions. Then let's apply the values of whether it's worth it.

Dan Willingham

8/16/2012 09:57:50 pm

@Mike G: my personal experience has been that virtually any topic can be interesting or boring to me, depending on how it's handled. I'm fond of pointing out that I love cognitive psychology--but I'm often bored at cognitive psychology conference talks. In contrast, I've been fascinated by topics during TED talks that, on the face of it, sound really dull.

hesi test link

8/16/2012 07:10:43 pm

Thanks for sharing this great article! That is very interesting I love reading and I am always searching for informative information like this..

Comments are closed.

Freakonomics and the application of science to education.

Purpose

Archives

Categories