The results of this experiment probably won't surprise you. What surprised me was the fact that we didn't already have data like this in hand.

The researchers (Sadler et al., 2013) tested 181 7th and 8th grade science teachers for their knowledge of physical science in fall, mid-year, and years end. They also tested their students (about 9,500) with the exact same instrument.

Each was a twenty-item multiple choice test. For 12 of the items, the wrong answers tapped a common misconception that previous research showed middle-schoolers often hold. For example, one common misconception is that burning produces no invisible gases. This question tapped that idea:

But the researchers didn't just ask the teachers to pick the right answer. They also asked teachers to pick the answer that they thought their students would pick.

What makes this study interesting is that it tests teacher subject-matter knowledge directly (instead of using a proxy like courses taken, or degrees) and that it directly measures one aspect of pedagogical content knowledge, namely, student misconceptions. The dependent measure of interest is student gain scores in content knowledge over the course of the year.

The results?

Teachers content knowledge was good, but not perfect. They got about 84% of the questions right.

Their knowledge of student misconceptions was not as good. Teachers correctly identified just 43% of those. (And their students had, as in previous studies, selected those incorrect items in high numbers.)

And what type of teacher knowledge matters to student learning? It turns out to interact with past student achievement, as measured by standard math and reading tests.

The graph shows gains in student knowledge, separated by items for which teachers have (or lack) various types of knowledge. Filled circles are for students who scored well on a math and reading test (high achievers), and open circles are students who scored poorly (low achievers)

Look first at learning for concepts without  a common misconception. If teachers have subject matter knowledge (SMK in the graph) students learn the concept better. In fact, low-achieving students learned nothing about a concept if teachers didn't know the concept themselves. High-achieving students did. The researchers speculate they may have learned the content from a textbook or other source.

For the strong misconception items, the low-achieving students learned very little, whatever the teacher knowledge. For high-achieving students, knowledge mattered, and they were most likely to learn when their teacher had both subject-matter knowledge and knew the misconceptions their students likely held (KoSM in the graph).

So the overall message is not that surprising. Students learn more when their teachers know the content, and when they can anticipate student misconceptions.

Somewhat more surprising (and saddening), low-achieving students are especially vulnerable when teachers lack knowledge. High-achieving students are more resilient.

There are limitations to this study, the most notable being that the sample is far from random (teachers were volunteers), and that the test was zero-stakes for all.

The strength was the direct measure of both types of knowledge, and that the researchers could examine the relationship of knowledge to performance at the level of individual items. One hopes we'll see more studies using this type of design.

Reference:
Sadler, P. M., Sonnert, G., Coyle, H.P., Cook-Smith, N., & Miller, J.L. (2013) Student learning in middle school science classrooms. American Educational Research Journal, 50, 1020-1049.

Elsewhere I have written about the potential power of narrative to help students understand and remember complex subject matter (Willingham, 2004; 2009). Now a new study (Arya & Maul, 2012) provides fresh evidence that putting to-be-learned material in a story format improves learning outcomes.

The experiment tested 209 7th and 8th grade students in the U.S. on texts about the discoveries of Galileo OR the discoveries of Marie Curie. The texts were developed to be as similar as possible in terms of syntactic complexity, vocabulary, accuracy, and other measures, and vary only in whether the information was presented in a typical expository fashion or in terms of a personal story of the scientist.

For example, one section of the expository text included this passage

The corresponding passage in the narrative version read this way:

Students comprehension and memory for the information in the text was measured immediately after reading it, and again one week later. The difference in recall between the narrative and non-narrative versions are shown as difference scores below.

Science lends itself naturally to narrative structure--authors can tell the stories of individual scientists, their struggles, their discoveries, and so on.

There's a case to be made that it also lends itself to a triumphalist view of science that is not accurate; scientists as heroes in an ever-progressing march towards Truth. Since Kuhn, that more or less Popperian view of science has been viewed as at least too simple, and more likely inaccurate.

But if it helps middle schoolers understand science, I'm inclined not worry too much about that point.

Instead, I'd like to broaden the view of "narrative." (I made this point in Why Don't Students Like School.) You don't have to think of narrative just as the story of an individual or group of people; you can think more abstractly conflict, complications, and the eventual resolution of conflict as the core of narrative structure.

I prefer to think of narrative in this broader sense because it is more flexible, and gives teachers more options, and also better captures the aspects of narrative structure that I suspect are behind the advantage conferred.

Reference

  Arya, D. J. & Maul, A. (2012). The role of the scientific discovery narrative in middle school science education: An experimental study. Journal of Educational Psychology, 104, 1022-1032.

My Facebook feed today has lots of links to this article. The upshot: a new Pew study showing that Americans think that US 15 year olds rank "near the bottom" on international science tests, whereas the truth is that they "rank in the middle among developed countries."

I guess "the middle" covers a lot of terrain, but the way I look at the data, this assertion doesn't hold.

The international comparison in question is the 2009 PISA. Here are the rankings. (Click for larger image)

Most everyone would agree that it's not appropriate to compare scores of US kids to those of poorer countries with little infrastructure and funding to support education.

That's why the article specifies the ranking of the US among "developed countries," and by the author's reckoning, kids from 12 developed countries scored better, and kids from 9 developed countries scored worse. That would put US kids at the 41st percentile.

The US is ranked 30th on the list. Just eyeballing it, it's hard to see how 17 of the countries scoring better could be considered "not developed."

On measures of "developed" status would be the International Monetary Fund's definition of "advanced economies" which includes: Australia, Austria, Belgium, Canada, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hong Kong, Iceland, Ireland, Israel, Italy, Japan, Luxembourg, Malta, Netherlands, New Zealand, Norway, Portugal, San Marino, Singapore, Slovakia, Slovenia, South Korea, Spain, Sweden, Switzerland, Taiwan, United Kingdom, United States (Click image for larger image

By this definition of "advanced" US kids are 23rd out of 32 countries, or the 28th percentile.

It's true that "near the bottom" is too grim an assessment. But I can't see a way to put the 2009 PISA data together such that American kids are scoring about average.

A new bill just passed the Education committee in the Oklahoma House of representatives, as reported in the Oklahoman. Titled "The Scientific Education and Academic Freedom Act," the bill purports to protect the rights of students, teachers and administrators to explore fully scientific controversies.

The bill supposes that some people currently feel inhibited in their pursuit of truth regarding "biological evolution, the chemical origins of life, global warming, and human cloning" and so the bill forbids school administrators and boards of education from disallowing such "exploration."

According to opinion pieces in the Daily Beast, The Week, and Mother Jones, the bill is a fairly transparent attempt to allow intelligent design into science classrooms, one that is being pursued in other states as well.

Yeah, that's what it sounds like to me too.

But even if we take the purported motive of the bill at face value, it's still a terrible idea.

Why shouldn't science teachers "teach the controversy?" Isn't it the job of teachers to sharpen students critical thinking skills? Isn't it part of the scientific method to evaluate evidence? If evolution proponents are so sure their theory is right, why are they afraid of students scrutinizing the ideas?

Imagine this logic applied in other subjects. Why shouldn't students study and evaluate the version of US history offered by white supremacists? Rather than just reading Shakespeare and assuming he's a great playwright, why not ask students to read Shakespeare and the screenplay to Battlefield Earth, and let students decide? And hey, why is such deference offered to Euclid? My uncle Leon has an alternative version of plane geometry and it shows Euclid was all wrong. I think that theory deserves a hearing.

You get the point. Not every theory merits the limited time students have in school. There is a minimum bar of quality that has to be met in order to compete. I'm not allowed to show up at the Olympics, hoping to jump in the pool and swim the 100 m butterfly against Michael Phelps.

Indeed, the very inclusion of a theory in a school discussion signals to students that it must have some validity--why else would the teacher discuss it?

The obvious retort from supporters of the bill is that intelligent design is actually a good theory, much better than the comparisons I've drawn.

That belief may be sincere, but it's due, I think, to a lack of understanding of scientific theory.  So here are a few of the important features of how scientists think about theories, and how they bear on this debates.

1) It's not telling that legitimate scientists point out unanswered questions, problems, or lacunae in the theory of evolution. Every theory, even the best theories, have problems. People who make this point may be thinking about the status of scientific laws as scientists did until the early part of the 20th century--as immutable laws. Scientists today think of all theories as provisional, and open to emendation and improvement.

2) A vital aspect of a good scientific theory is that it be open to falsification. It's not obvious what sort of data would falsify intelligent design theories, especially young-earth theories, which make predictions that are already disconfirmed by geology, astrophysics, etc., and yet are maintained by their adherents. Evolution, in contrast, has survived tests and challenges for 100 years--indeed, the theory has changed and improved in response to those challenges.

3) In the case of old-earth intelligent design theories, the focus is much more on the putative beginnings of the universe of or life on Earth, and these don't have the feel of a scientific theory at all. They seem much more like philosophical queries because they focus on large-scale questions and how these questions ought to formulated--they never get to detailed questions that might be answerable by experiment, the meat-and-potatoes of science.

4) Good scientific theories are not static. They not only change in the face of new evidence, they continue to spawn new and interesting hypotheses. Evolution has been remarkably successful on this score for over 100 years. Intelligent design has been static and unfruitful.

These are some of the reasons that scientists think that intelligent design does not qualify as a good scientific theory, and therefore does not merit close attention in K-12 science classes, and more than my uncle's theory of geometry does.

If you're going to write bills about what happens in science class, it's useful to know a little science.

EDIT: 2/22/13 1:20 p.m. EST: typos

There is a lot of talk these days about STEM--science, technology, engineering, and math--and the teachers of STEM subjects. It would seem self-evident that these teachers, given their skill set, would be in demand in business and industry, and thus would be harder to keep in the classroom.

A new study (Ingersoll & May, 2012) offers some surprising data on this issue.

Using the national Schools and Staffing Survey and the Teacher Follow-Up Survey, they found that science and math teachers have NOT left the field at rates higher than that of other teachers. In this data set (1988-2005)  math teachers and science teachers left teaching at about the same rate as teachers in other subjects: about 6% each year.

Furthermore, when these teachers do leave a school, they are no more likely to take a non-education job than other teachers: about 8% of "leavers" took another job outside of education. Much more common reasons to leave the classroom were retirement (about 15%) or an education job other than teaching (about 17%).

The authors argue that teacher turnover, not teachers leaving the field, is the engine behind staffing problems for math and science classes.

So what prompts teacher turnover?

The authors argue that on this dimension math and science teachers differ. Both groups are, unsurprisingly, motivated by better working conditions and higher salaries, but the former matter more to math teachers, and science teachers care more about the latter.

But in both cases, the result is that math and science teachers tend to leave schools with large percentages of low-incomes kids in order to move to schools with wealthier kids.



Ingersoll, R. M., & May, H. (2012). The magnitude, destinations, and determinants of mathematics and science teacher turnover. Educational Evaluation and Policy Analysis, 34,  435-464.

In an op-ed piece in August 19th's New York Times, Bronwen Hruska tells of her experiences with her son, Will, between the 3rd and 5th grade. Will was misdiagnosed with ADHD.

Hruska and her husband were initially approached by Will's teacher, who thought his behavior indicated ADHD. Though they were doubtful, they took him to a psychiatrist who said that Will did indeed have ADHD and prescribed stimulant medication. Will took the medication for two years but stopped when he concluded that Aderall is dangerous. Now a happy high school sophomore, there is not much reason to think that the medication was ever necessary.

How did this happen?

The title of the piece--"Raising the Ritalin Generation"--provides a clue to the author's conclusion. Hruska suggests that our society is sick. Teachers are too quick to suggest medication for kids. Schools "want no part" of average kids; they expect kids to be exceptional, extraordinary. And we, as a society, are teaching kids that average is not good enough, and that if you're only average you should take a pill.

But there's an important piece missing from this picture--parents.

From what's written, it sure does sound like Will was misdiagnosed. But I can't help but wonder why his parents didn't know it at the time.

ADHD diagnosis requires that symptoms be present in at least two settings. So it's not enough that Will shows troubling symptoms in school: he would also need to show them at home, in social settings, or in some other context for him to be diagnosed. There's no indication of a problem outside of school.

It's also notable that the mere presence of symptoms is not enough: the symptoms must be clinically significant; in other words, they obstruct the child's ability to function well in that setting and Hruska maintains that Will seems like a typical kid to her.

This is where Hruska loses me. Why would she accept the diagnosis if symptoms were observed in just one context, and if she believed there was limited evidence that the symptoms were clinically significant in that context? Why wouldn't she challenge the physician who diagnosed him?

I'm led to wonder if she knew the diagnostic criteria. They aren't hard to find. Google "adhd diagnosis." The first link is the CDC site that offers a reader-friendly version of the DSM IV criteria.

Are our kids pill-happy? Are we raising a Ritalin generation? If so, the solution is not to lay all of the blame on schools and society or even on physicians who make mistakes, and to portray parents as powerless victims. The solution is for parents to make better use of the wealth of scientific information available to us, and to ask questions when a doctor or other authority makes claims that fly in the face of our experience.

Making a change to education that seems like a clear improvement is never easy. Or almost never.

Judith Harackiewicz and her colleagues have recently reported an intervention that is inexpensive, simple, and leads high school students to take more STEM courses.

The intervention had three parts, administered over 15-months when students were in the 10th and 11th grades. In October of 10th grade researchers mailed a brochure to each household titled Making Connections: Helping Your Teen Find Value in School. It described the connections between math, science, and daily life, and included ideas about how to discuss this topic with students.

In January of 11th grade a second brochure was sent. It covered similar ideas, but with different examples. Parents also received a letter that included the address of a password-protected website devised by researchers, which provided more information about STEM and daily life, as well as STEM careers.

In Spring of 11th grade, parents were asked to complete an online questionnaire about the website.

There were a total of 188 students in the study: half received this intervention, and the control group did not.

Students in the intervention group took more STEM courses during their last two years of high school (8.31 semesters) than control students (7.50) semesters.

This difference turned out to be entirely due to differences in elective, advanced courses, as shown in the figure below.

An important caveat about this study: all of the subjects are participating in the Wisconsin Study of Families and Work. This study began in 1990. when women were in their fifth month of pregnancy.

The first brochure that researchers sent to subjects included a letter thanking them for their ongoing participation in the longer study. Hence, subjects could reasonably conclude that the present study was part of the longer study.

That's worth bearing in mind because ordinary parents might not be so ready to read brochures mailed to them by strangers, nor to visit suggested websites.

But that's not a fatal flaw of the research. It just means that we can't necessarily count on random parents reading the materials with the same care.

To  me, the effect is still remarkable. To put it in perspective, researchers also measured the effect of parental education on taking STEM courses. As many other researchers have found, the kids of better-educated parents took more STEM courses. But the effect of the intervention was nearly as large as the effect of parental education!

Clearly, further work is necessary but this is an awfully promising start.

Harackiewicz, J. M, Rozek, C. S., Hulleman, C. S & Hyde, J. S. (in press). Helping parents to motivate adolescents in mathematics and science: An experimental test of a utility-value intervention. Psychological Science.

Steven Levitt, of Freakonomics fame, has unwittingly provided an example of how science applied to education can go wrong.

On his blog, Levitt cites a study he and three colleagues published (as an NBER working paper). The researchers rewarded kids for trying hard on an exam. As Levitt notes, the goal of previous research has been to get kids to learn more. That wasn't the goal here. It was simply to get kids to try harder on the exam itself, to really show everything that they knew.

Among the findings: (1) it worked. Offering kids a payoff for good performance prompted better test scores; (2) it was more effective if, instead of offering a payoff for good performance, researchers gave them the payoff straight away and threatened to take it away if the student didn't get a good score (an instance of a well-known and robust effect called loss aversion); (3) children prefer different rewards at different ages. As Levitt puts it "With young kids, it is a lot cheaper to bribe them with trinkets like trophies and whoopee cushions, but cash is the only thing that works for the older students."

There are a lot of issues one could take up here, but I want to focus on Levitt's surprise that people don't like this plan. He writes "It is remarkable how offended people get when you pay students for doing well – so many negative emails and comments." Levitt's surprise gets at a central issue in the application of science to education.

Scientists are in the business of describing (and thereby enabling predictions of) the Natural world. One such set of phenomenona concerns when students put forth effort and when they don't.

Education is a not a scientific enterprise. The purpose is not to describe the world, but to change it, to make it more similar to some ideal that we envision. (I wrote about this distinction at some length in my new book. I also discussed on this brief video.)

Thus science is ideally value-neutral. Yes, scientists seldom live up to that ideal; they have a point of view that shapes how they interpret data, generate theories, etc., but neutrality is an agreed-upon goal, and lack of neutrality is a valid criticism of how someone does science.

Education, in contrast, must entail values, because it entails selecting goals. We want to change the world--we want kids to learn things--facts, skills, values. Well, which ones? There's no better or worse answer to this question from a scientific point of view.

A scientist may know something useful to educators and policymakers, once the educational goal is defined; i.e., the scientist offers information about the Natural world that can make it easier to move towards the stated goal. (For example, if the goal is that kids be able to count to 100 and to understand numbers by the end of preschool, the scientist may offer insights into how children come to understand cardinality.)

What scientists cannot do is use science to evaluate the wisdom of stated goals.

And now we come to people's hostility to Levitt's idea of rewards for academic work.

I'm guessing most people don't like the idea of rewards for the same reason I don't. I want my kids to see learning as a process that brings its own reward. I want my kids to see effort as a reflection of their character, to believe that they should give their all to any task that is their responsibility, even if the task doesn't interest them.

There is, of course, a large, well-known research literature on the effect of extrinsic rewards on motivation. Readers of this blog are probably already familiar with it--if so, skip the next paragraph.

The problem is one of attribution. When we observe other people act, we speculate on their motives. If I see two people gardening--one paid and the other unpaid--I'm likely to assume that one gardens because he's paid and the other because he enjoys gardening.  It turns out that we make these attributions about our own behavior as well. If my child tries her hardest on a test she's likely to think "I'm the kind of kid who always does her best, even on tasks she don't care for." If you pay her for her performance she'll think "I'm the kind of kid who tries hard when she's paid."  This research began in the 1970's and has held up very well. Kids work harder for rewards. . . until the rewards stop. Then they engage in the task even less than they did before the rewards started. I summarized some of this work here.

In the technical paper, Levitt cites some of the reviews of this research but downplays the threat, pointing out that when motivation is low to start with, there's not much danger of rewards lowering it further. That's true, and I've made a close argument: cash rewards might be used as a last-ditch effort for a child who has largely given up on school.  But that would dictate using rewards only with kids who were not motivated to start, not in a blanket fashion as was done in Levitt's study. And I can't see concluding that elementary school kids were so unmotivated that they were otherwise impossible to reach.

In addressing the threat to student motivation with research, Levitt is approaching the issue in the right way (even if I think he's incorrect in how he does so.)

But on the blog (in contrast to the technical paper), Levitt addresses the threat in the wrong way. He skips the scientific argument and simply belittles the idea that parents might object to someone paying their child for academic work. He writes:

Perhaps the critics are right and the reason I’m so messed up is that my parents paid me $25 for every A that I got in junior high and high school.  One thing is certain: since my only sources of income were those grade-related bribes and the money I could win off my friends playing poker, I tried a lot harder in high school than I would have without the cash incentives.  Many middle-class families pay kids for grades, so why is it so controversial for other people to pay them?

I think Levitt is getting "so many negative emails and comments" because he's got scientific data to serve one type of goal (get kids to try hard on exams) the application of which conflicts with another goal (encourage kids to see academic work as its own reward). So he scoffs at the latter.

I see this blog entry as an object lesson for scientists. We offer something valuable--information about the Natural world--but we hold no status in deciding what to do with that information (i.e., setting goals).

In my opinion Levitt's blog entry shows he has a tin ear for the possibility that others do not share his goals for education. If scientitists are oblivious to or dismissive of those goals, they can expect not just angry emails, they can expect to be ignored.

There is a great deal of attention paid to and controversy about, the promise of training working memory to improve academic skills, a topic I wrote about here.

But working memory is not the only cognitive process that might be a candidate for training. Spatial skills are a good predictor of success in science, mathematics, and engineering.

Now on the basis of a new meta-analysis (Uttal, Meadow, Tipton, Hand, Alden, Warren & Newcombe, in press) researchers claim that spatial skills are eminently trainable. In fact they claim a quite respectable average effect size of 0.47 (Hedge's g) after training (that's across 217 studies).

Training tasks across these many studies included things like visualizing 2D and 3D objects in a CAD program, acrobatic sports training, and learning to use a laparascope (an angled device used by surgeons). Outcome measures were equally varied, and included standard psychometric measures (like a paper-folding test), tests that demanded imagining oneself in a landscape, and tests that required mentally rotating objects.

Even more impressive:

1) researchers found robust transfer to new tasks
2) researchers found little, if any effect of delay between training and test--the skills don't seem to fade with time, at least for several weeks. (Only four studies included delays of greater than one month.)

This is a long, complex analysis and I won't try to do it justice in a brief blog post. But the marquee finding is big news. What we'd love to see is an intervention that is relatively brief, not terribly difficult to implement, reliably leads to improvement, and transfers to new academic tasks.

That's a tall order, but spatial skills may fill all the requirements.

The figure below (from the paper) is a conjecture--if spatial training were widely implemented, and once scaled up we got the average improvement we see in these studies,  how many more people could be trained as engineers?

The paper is not publicly available, but there is a nice summary here from the collaborative laboratory responsible for the work. I also recommend this excellent article from American Educator on the relationship of spatial thinking to math and science, with suggestions for parents and teachers.

Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N.S. (2012, June 4). The Malleability of Spatial Skills: A Meta-Analysis of Training Studies. Psychological Bulletin. Advance online publication. doi: 10.1037/a0028446

Newcombe, N. S. (2010) Picture this: Increasing math and science learning by improving spatial thinking. American Educator, Summer, 29-35, 43.

I made another of my garage-band quality videos, this one on the relationship of science and education, titled "Is Education an Art or a Science?"