I’m leery of value-added measures as a metric of individual teacher quality. Aside from the straight psychometric challenges, I’ve always worried that there’s too much that teachers give students that a VAM would miss. For example, I think of a LAUSD high school teacher who told me with considerable excitement that he had managed to get one of his kids who was near to dropping out to come to his class for five days in a row. The student wasn’t really participating in any way yet, but the teacher had a plan in mind for how to try to coax this student back to trying to engage with math just one more time, after many years of difficulty and frustration.

Who would tell this teacher “that child is a bad investment of your time?” Yet where is the VAM that will measure the value the teacher is adding to this student?

A recent paper (Master, Loeb & Wyckoff, 2017; preprint here) tried to shed light on this problem. The authors note that the short-term effects of ELA teachers are usually smaller than those of math teachers (as measured by VAM). This outcome is easy to understand from a cognitive perspective—students learn more outside of school that can be applicable to ELA tests (compared to math), and thus the contribution of a teacher and school will be relatively smaller. But the contribution of ELA and math teachers to long-term outcomes (e.g., graduation) is equivalent. Why?

One possibility is that students learn different kinds of things from each. They may learn more subject-specific content from math teachers that persist to math performance next year. But a good ELA teacher may be more likely to impart different, more persistent skills to students—they may improve their self-image as students, for example.

To examine this possibility the authors compared VAMs across years in ELA and math both within subjects and between them. In other words, if an ELA teacher was really effective, we know that effect will be observable in English class the next year. Will it be observable in math class as well?

The authors had two enormous data sets with which to investigate this question: standardized test scores from 3rd through 8th students in New York City and Miami-Dade County from 2003/04 to 2011/12.

This study, like previous studies, found that about 25 or 30% of VAM persists into the next year. In this study, those quantities were similar in math and in ELA. The startling result came when investigators used VAM in one subject to predict VAM in the other subject in subsequent years. (More precisely, they examined student achievement in year 2 in subject A, accounting for achievement in year 1 in subjects A and B, and the VAM estimates of the teachers in subjects A and B in year 1.)

As noted, about ¼ to 1/3 of the VAM in ELA from one year carried over to ELA achievement the following year. And about 46% of that effect also carried over to math achievement in Miami-Dade. In New York City, it was 70%.

But having a really effective math teacher had very little impact on ELA achievement the following year. In both districts, this carryover was around 5%.

There was good evidence that these effects persist into a third year in New York City, but in Miami-Dade the results were inconclusive, according to the researchers because measurements there were less precise.

The results point to three conclusions, given the caveats typical to this work (and often forgotten): subjects other than ELA and math were not measured, and students were not randomly assigned to teachers (and in fact, there is likely systematic bias in assignment)

First, these results help us to understand why ELA short-term VAMs are smaller than math short-term VAMs, yet the predictive value for long-run outcomes (like graduation) is the same. The ELA short-term VAMs may typically be smaller, but they contribute across subjects (which the math do not). And these cross-subject effects may last years.

Second, the authors don’t speculate much on the mechanism of transfer, but at least two routes seem plausible. First, ELA teachers may, on average, provide a bigger boost to what are usually called non-cognitive skills: self-regulation, persistence, seeing oneself as belonging in school, and so on. Second, better ELA skills—especially better reading skills like decoding, fluency, deployment of comprehension strategies, and self-monitoring of comprehension—seem likely to pay dividends in many subjects.

​Third, when it comes to policy, I’ll leave the conclusion to the authors: “educators and policymakers may miss valuable information if they rely only on short-term within-subject student learning to evaluate teachers’ “value added” to student achievement.” Student achievement gains prompted by teacher X could easily be misattributed to another teacher in another subject, years later.  

Over the weekend I posted a link to a new study (Singer & Alexander, 2017) comparing reading comprehension when reading from a screen and reading from paper. Ninety undergraduates read four texts each: two book excerpts and two newspaper articles, all on various topics concerning childhood ailments. Two were read digitally, and two on paper. The results showed that subjects reading from a screen or from paper were equally proficient in identifying the main idea, but subjects reading paper did a better job when asked to list key points of the text, and to describe how it related to the main idea. Despite this pattern, 69% of subjects thought they performed better on the screen-based texts, and just 18% thought they had done better with paper.

I didn’t think all that much about this study because there have been lots of comparable studies. But my tweet garnered more comments and retweets than mine typically do, so I figured this finding must be news to some of my Twitter followers.

That’s when I decided to write a blog post flagging some of the relevant studies.

The following studies compared reading comprehension from a screen and paper, and concluded paper is better:

• Chen et al, 2014;
• Jeong, 2012;
• Lauterman & Ackerman (2014);
• Kim & Kim, 2013;
• Mangen et al, 2013; 
• Rasmusson et al , 2015;

The following studies reported no difference in comprehension, but a difference in reading time. The common-sense interpretation is that if comprehension is more difficult on screen, the reader has the option to trade time for accuracy, and that’s’ what these readers; 

• Ackerman & Lauterman, 2012;
• Connell et al, 2012;
• Daniel & Woody, 2013;

That makes it more difficult to interpret the following three studies. The experimenters report no difference in reading comprehension, but they did not measure reading time:

• Margolin et al 2013;
• Rockinson-Szapkiw, 2013; 
• Subrahmanyam et al, 2013;  

One exception is Porion et al 2016 who report no difference in comprehension, but did restrict reading time.

So ten studies report that paper is superior and four call it a draw, but three of these did not measure reading time. Other researchers reviewing the literature draw same conclusion that I do: comprehension is better when reading from paper: Tees, 2010; Sidi 2016; Zucker et al, 2009;  Walsh, 2016 concludes that reading from paper is better only for complex documents.

I doubt I’ve captured all of the extant literature. These studies tend to be published in far-flung places, and quite honestly, not always in top-drawer journals. I think that’s because it seems like the same question is being posed over and over…but actually there are a lot of potentially important modifying variables: four obvious ones would be subject matter knowledge, reader age, the purpose of reading (textbook vs leisure, for example) and experience with e-readers. So when you read any one of these articles, you can’t help but think “hmm. How generalizable is this?”

The other caveat to this conclusion is that it’s almost certainly a moving target. The fifth important variable is the interface used by the e-book. It’s my hunch that that’s the vital factor in this (small) screen decrement. Software engineers are working on it, as hardware engineers have made great improvements in the brightness and contrast of screens. I think it’s just a matter of time before ereaders are as easy to read as paper.

​​That college grades have risen over the last fifty years is not in dispute. The average course grade in college was 2.5 in 1960 (on a 4.0 grading scale) and had risen to 3.1 in 2006.

I often hear two reasons offered for the increase. During the 1960s, Professors ​were eager to help students stay in the school, so as to avoid the draft and the Vietnam war. (The lax standards were extended to female students for verisimilitude.)

In more recent years, increases in grades are attributed to the consumer era of higher education. Students see college courses as a commodity they purchase, and professors, hoping for high enrollements and good course evaluations to boost their tenure prospects, hand out the A's.

This student-as-consumer account seems to be bolstered by the fact that grade inflation has been higher at private colleges and universities than at public ones. 

Both accounts propose that grade inflation is solely due to changes in professors' grading policies, and that students have not changed. A recent paper challenges that assumption, and offers some persuasive data. 

​Rey Hernández-Julián and Adam Looney brought economic models of inflation to bear on the question. As they point out, higher prices don't always signal inflation. For example, price may increase as quality increases. 

They examined 20 years of transcript data from 90,000 students at Clemson University over the years 1982-2001. During that time grades increased from 2.67 to 2.99, comparable to the increase observed elsewhere. They also had data on the characteristics of these students. 

The researchers observed shifts in student characteristics and course-taking behavior that accounted for a little more than half of the increase in grades.

First, the students coming to Clemson were better prepared, as measured by the SAT. Math scores increased by 29 points over the period, and Verbal scores by 15 points.

Second, student enrollment drifted towards departments that tend to assign higher grades. More students took courses in Speech and Communication, Spanish, Graphic Communications, and yes, Education; all these department assign average grades above the means of other departments. Tough-grading departments--Math, Accounting, Physics, Mechanical Engineering--saw drops in enrollment.

Third, the percentage of women at Clemson increased, and women earn higher grades than men do. This increase was modest (4% points) and accounts for little of the grade increase, but the authors point out that the increase in women enrollment is high nationwide (from 41% to 56% between 1970 and 2000) and so may account for more of the grade increase at other institutions. 

The upshot is that a little less than half of the increase in grades is attributable to grade inflation. The authors identify possible contributors to grade increase, and whatever they cannot account for is labeled "inflation." They point out that the actual value of grade inflation could be lower.

I would add that the actual value could, in theory, also be higher. Unobserved variables like course difficulty and student effort could be decreasing. Obviously measuring such variables at broad scale is difficult if not impossible. 

A prominent theory of the difference in cognition between those with Autism Spectrum Disorder and typical adults has it that the former have difficulty understanding and predicting what others think. This ability is often called Theory of Mind. A recent experiment indicated that, at least in one task, this process is not difficult for adults on the Autism spectrum (Pantelis & Kennedy, 2017).

The authors used a task called the Beauty Contest game. It’s deceptively simple. You’re told that a large group of people will each pick one number from 1 to 100. Whoever picks the number closest to 2/3 the value of the mean guess is the winner.

So consider why this is a theory of mind task.

• One possibility is that a player will say to himself “this is complicated, I have no idea how to be strategic, so I’ll pick a random number between 1 and 100. Call this a “0th order” player.
• A 1st order player assumes that others will be 0th order players. They will choose randomly, so the mean of their choices will be 50, so the 1st order player chooses a value of 2/3*50 or 33.
• A 2nd order player assumes that some of the other players will be 0th order, but some will figure out how to be 1st order players, and will choose numbers accordingly. The 2nd order players chooses her number according to how many 1st order and how many 0th order players she expects to be among her opponents.
• A 3rd order player assumes some other players will be 2nd order, as well as 1st and 0th order…and in principle it can continue from there.

You can see how this theory of mind task is recursive. You might start by asking “what will my opponent think?” They you can ask “will my opponent guess what I’m thinking he’ll think?” and so on. You can also see that thinking through what others might do leads you to pick a lower number.

(It’s called the Beauty Contest because the idea was first forwarded by John Maynard Keynes in the form of a newspaper beauty contest, where entrants were to select the six most attractive faces from 100 pictures, with the best pickers getting a prize. Keynes pointed out that entrants might simply pick the ones they thought most attractive, but their choices might also be informed by what they thought others would do.)

Pantelis & Kennedy first gave this task to 250 undergraduates; each was also asked to complete the Autism-Spectrum Quotient questionnaire (AQ), which is meant to measure sub-clinical personality traits consistent with autism. There was no correlation between guesses and scores on the AQ.

There is also a method of estimating the “depth” of recursive thinking in a large group of subjects. The experimenters separated the 250 subjects into those scoring relatively high on the AQ and those scoring relatively low. They observed no difference in the depth of recursive thinking in the two groups.

More telling, experiment 2 tested 30 adults with autism or Asperger’s, all of whom were described as high-functioning and of typical to high intelligence (as measured by a standard IQ test). The researchers observed no difference between this group and typical controls in the mean number guessed, nor in the model’s estimate of the depth of theory-of-mind thinking.

The researchers are appropriately cautious in drawing conclusions from this research. As they note, theory of mind is complex and it’s easy to believe that it has multiple components. The beauty contest likely taps just one. Then too, autism spectrum disorder is likely complex, with a constellation of abilities and challenges which may vary considerably from person to person. And that is perhaps the main point of this experiment: the simple characterization: “autism is mainly a theory of mind problem” won’t do. 

Students come to school to learn, but that doesn’t mean they begin school devoid of knowledge. As much as policymakers have focused in recent years on the potential richness of the home environment (and the edge that may give children) it’s equally true that children come to school with erroneous beliefs—beliefs that teachers have long recognized can be an obstacle to learning.

That’s especially true in some areas of science. Humans must know how to interact with their world, and that necessity appears to have led to a set of intrinsic, intuitive beliefs about the nature of objects in the world which have some utility for individuals, but also conflict with important principles of biology. Researchers have, in the last ten years or so, documented some of these. A new report (Coley et al, 2017) is perhaps the most thorough in doing so.

Researchers administered a series of measures to three groups of subjects (total N = 211) : 8th grade students, college biology majors, and college non-biology majors. The measures focused three intuitive beliefs:

​Anthropocentric thinking: Using humans as the basis for reasoning about other biological species (i.e., thinking of humans as the norm to which others are compared) and seeing humans as biologically unique and discontinuous compared to other species. For example, adults are slower to confirm that plants are living things, consistent with the idea that organisms that seem more distant from humans must not share other qualities with humans.

Teleological thinking: believing that the apparent goal or function of an object or event is the cause of the event. Hence, a young child might reason that lions exist so that they can be in zoos for us to see. An older child might reason that it rains in order that plants grow.

Essentialist thinking: the belief that objects have a fundamental essence that specifies their category membership, and that all members of this category share this essence. For example, an animal is either a bird or not a bird—there’s no degree of “birdiness” in the final analysis, and all birds share the bird essence. Young children believe that parents and their offspring tend to share not only physical characteristics (e.g., eye color) but also preferences (e.g., liking sweets).  
​
The researchers wanted to examine (1) whether these intuitive beliefs change in late adolescence and (2) whether training in biology affected the prevalence or depth of these beliefs. There was an effect for both factors, but in both cases, smaller than the researchers expected. The graphs below show results for one question type from each category—the final analysis used a composite of questions from each category, but these are representative. 

At left, responses to a anthropocentric question. Students were given a number of organisms and asked whether each shared a common ancestor and any point in evolution with humans. Higher bars reflect better performance.
At center, responses to a teleological question. Students read an "explanation" of a biological phenomenon that appealed to causality (e.g., "worms exist to aerate the soil, which benefit plants"). Shorter bars reflect better performance.
At right, responses to an essentialist question. Students were asked questions about absolute category membership (e.g., something is either a mammal or it isn't, and there's never an in-between). Shorter bars reflect better performance. 

All in all there are effects of age (and associated effects like further general education) as well as effects of training in biology. But all three groups show the same trend for anthropocentric and teleological thinking. Curiously, essentialist thinking grows stronger with development. The authors suggest a few possible interpretations of this finding the most plausible of which is that it is broadly consistent with how biology is frequently taught.

The core finding---that these intuitive ways of thinking are quite resistant to education—matches the findings regarding naïve  physics (e.g., Potvin et al., 2015) and highlights the challenges science teachers face. 

The BBC has a website which (based on the title) I gather is to help students prepare for the GCSE. The "science" area has a section on "Brain and Memory" which, unfortunately, has several inaccuracies. 

This figure appears on page 1. Some of the functions attributed to some brain areas are roughly correct--motor cortex is shown as more toward the front of the brain than sensory cortex, which is right, and vision is where it ought to be. But it's hard to tell what's what because the brain is simply drawn wrong (see below) so localization is hard to pinpoint (e.g., there's no central sulcus, which would separate motor and somatosensory areas).

Some functions--reading--are not localized in one spot. "Speech" is misleading--do they mean producing speech? Or language overall? Whatever, it's wrong.

Things don't improve on the next two pages, which cover some cognitive aspects of memory. I've annotated them.

This is not advanced material. This is stuff I teach in my Introduction to Cognition course. 
Come on, BBC. The American election is hard enough on us. Don't add to our misery.

Printing Nicholson Baker’s article in yesterday’s Magazine was a terrible, terrible decision.

The decision deserves two “terribles” because it was a double mistake.

First, you published an article on a topic that entails conflicting priorities in setting goals for public good, policy constraints in achieving these goals, the science of learning, distribution of wealth, and doubtless other complexities that I’m too exhausted to identify and enumerate. The author of the article has no expertise on any of these matters. That he appears to believe his 28 days as a substitute teacher gives him much insight into schooling only makes him less credible. The most fundamental limitations of his experience—for example, that teachers might choose a lesson for the substitute because it is easy to teach, even if it’s less interesting for the students—seem to have escaped him.

The second “terrible” is, unsurprisingly, the content. The author commits the common education newcomer blunder: “The school that would have been perfect for me, would be perfect for everyone.” He cannot understand why high school must be so stifling and soulless. Part of the blame goes to curriculum, where otherwise interesting topics are made dull, but there’s no mistaking that the teachers who inflict this boring stuff on students deserve blame as well. Baker reminisces fondly about his own experience at an alternative high school, where students studied what they wished. 

To be more specific, NYTimes editors, here’s a probably incomplete list of problems in Baker’s argument:
                1. There is actually evidence regarding classroom instructional quality in this country (e.g., here). He might have made use of it. (It shows, by the way, that the emotional tone is, on average, much more positive than he lets on. Instructional quality, however, is not much better.)
                2. Baker is not the first to suppose that much greater freedom for students would lead to greater motivation and better outcomes. The lesson over the last hundred years seems to be that such schools are wonderful when they work, but reproducing the successes has proven more difficult than most observers would guess.
                3. Some parents prefer a lot of structure. The private schools in  my town do not all follow the lots-of-choice model, a al Waldorf, Montessori, or Reggio Emilia. More parents pay to send their children to highly structured, traditional schools.
                4. There are good arguments in favor of a common curriculum.

While I have your attention, please don’t publish similarly one-note, blinkered pieces centering on the ideas like these:
1) Technology is poised to revolutionize learning and schools.
2) Competition would solve all problems in American education.
3) American education is the best in the world and all challenges in educational outcomes are due to poverty.
4) Teachers are fools, and the teacher’s unions are organized crime syndicates dedicated to protecting them.
5) All of America’s problems in education can be traced to standardized tests and if teachers were simply allowed to teach as they wished, all would be well.

Thanks,
Dan

A piece appeared in the New York Post on August 27 with the headline "It's digital heroin: How screens turn kids into psychotic junkies." 

Even allowing for the fact that authors don't write headlines, this article is hyperbolic and poorly argued. I said as much on Twitter and my Facebook page, and several people asked me to elaborate. So....

First, to say "Recent brain imaging research is showing that they [games] affect the brain’s frontal cortex — which controls executive functioning, including impulse control — in exactly the same way that cocaine does," is transparently false. Engaging in a behavior cannot affect your brain in exactly the same way a  psychoactive drug does. Saying it does is a scare tactic. 

Lots of activities give people pleasure and it's sensible that these activities show some physiological similarities, given the similarity in experience. But if you want to suggest that the analogy (games and cocaine both change dopamine levels, therefore they are similar in other ways) extends to other characteristics, you need direct evidence for those other characteristics. Absent that, it's as though you buy a pet bunny and I say "My God, bunnies have four legs. Don't you realize TIGERS have four legs? You've brought a tiny tiger into your home!"

On the addiction question: The American Psychiatric Association has considered including Internet Addiction in the DSM V and has elected not too, though it's still being considered. Research is ongoing, and technology changes quickly, so it's wouldn't make sense to close the book on the issue. 

To qualify as an addiction, you need more than the fact that the person likes it a lot and does it a lot. Addictions are usually characterized by: 

•Tolerance--increased amount of the behavior becomes necessary
•Withdrawal symptoms if behavior is stopped
•Person wants to quit but can’t
•Lots of time spent on the behavior
•Engage in the behavior even though it’s counterproductive

The last two of these seem a good fit for kids who spend a huge amount of time on gaming or other digital technologies. The others, the less obviously so. 

Let me be plain: I have plenty of concerns regarding the content and volume of children's time spent with digital technologies. I've written about this issue elsewhere, and my own children who are still at home (age 13, 11, and 9) face stringent restrictions on both.

But thinking through a complex issue like the social, emotional, cognitive, and motivational consequences of ominipresent screens in daily life requires clear-headed thinking, not melodramatic claims based on thin analogies. 

Nothing is more familiar to those who follow the literature on early educational interventions. A program meant to boost children’s reading or math or school readiness works wonderfully, helping children who started pre-k at a disadvantage achieve at levels comparable to other kids…but follow-up studies in later years show that the boost was not long-lasting.  The results faded.

The most common explanation (and the one that I had always assumed was right) centers on the content these children were taught after the intervention ended. Instruction must continue to challenge these children, to extend their accomplishments. If teachers emphasize more basic material, naturally we’ll observe fadeout.

I’ve sometimes used this metaphor: early intervention is not like setting the trajectory of a rocket, a one-time event that, if you get it right, you  needn’t think about again. It’s more like extra fuel in the booster rocket; it gets kids to the right altitude early on, but you’ve got to ensure that they have the same fuel in their rocket that other kids do after the intervention.

A recent article by Drew Baily and colleagues (2016) casts doubt on this explanation. They call it the Constraining Content hypothesis, and set forth a competing explanation they call the Pre-Existing Differences hypothesis.

​You identified a bunch of students who were either behind or at risk for being behind. You intervened. At the end of the year, they are no longer behind. Fine, but you didn’t select students for the intervention randomly. You picked them because they were behind, and at least some of the reasons they were behind will still be present at the end of the intervention.

Maybe their home environment does not support mathematics achievement, for any of a large number of reasons. Maybe these children’s beliefs about mathematics and expectations of themselves differ. Maybe their working memory capacity and/or general intelligence differ. Whatever the reasons children start preK behind, is there any good reason to suppose those factors have magically disappeared by the end of the intervention? Or that they won’t affect math achievement any more?

Here’s how Baily and colleagues compared the constraining content and the pre-existing differences hypotheses. They used a preK math intervention that is known to work (Building Blocks). They measured math ability at the start of preK, at the end of preK (after the intervention) and at the end of Kindergarten. You’d expect to see better scores for the kids getting the intervention (compared to controls) at the end of preK, but then a diminution of that advantage at the end of Kindergarten—classic fadeout—and that’s what you see. Here are the results of the overall treatment and control groups.

Here’s the interesting part of the experiment. Researchers compared scores of control students (those who had been randomly selected not to receive the intervention) who nevertheless scored well at the end of preK; even though they had not received the intervention during the prior year, their scores were comparable to kids who did receive the intervention.

​All children were to receive the same instruction in the kindergarten. So if the Constraining Content hypothesis is right, the two groups should show comparable learning. But the Pre-Existing Differences hypothesis makes a different prediction. The control kids who nevertheless scored as well as the intervention kids had something going for them during the preK year—lots of support at home, lots of math smarts, whatever. Those factors will still contribute in kindergarten, so these control kids will score better than the intervention kids at the end of kindergarten. 

It makes sense that kids who manage to score well after the intervention without actually experiencing the intervention were better at the pre-test.

And crucially, those out-of-school factors are still present at followup. Even though they experienced the same instruction during kindergarten and began the year with comparable math knowledge, by the end of the year they are doing better.

The researchers had another way to compare the Constraining Content and the Pre-Existing Differences hypotheses. Students are paired by score—one control and one intervention kid who scored comparably on the post-test. They sorted these pairs so there is a higher- and a lower-achieving group. Then they looked at the followup scores of each group. The Constraining Content hypothesis predicts that fadeout will be worse for higher scoring kids…they are the ones who are most affected by the not-very-challenging content….lower scoring kids should be catching up to the higher scoring kids because for them, the instruction is challenging. BUT the data showed exactly equivalent gains in the high- and low-scoring pairs.

​We need a new metaphor. Intervention for at-risk students is not resetting the trajectory of the rocket, but it’s not just extra fuel in the booster rocket to get them to altitude, after which one must ensure they still have fuel in the rocket. If they are to keep pace with their peers, they continue to need extra fuel in the rocket after the intervention.