Once children are fluent decoders, the most frequent problem in reading is poor comprehension due to a failure to make inferences. Even seemingly straightforward anaphoric inferences can elude these students: they might read “Bob gave Tamisa some of his snack because she was hungry” and still be unsure of the referent for “she.” Other inferences require bridging information from long term memory, and these are still more challenging. For example, “Kevin said he was cold. Zeke gave him his coat.” Even if these two sentences are understood, each on its own, deeper comprehension entails making the (probably accurate) inference that Zeke gave Kevin the coat because Kevin said he was cold, which requires knowing that putting on a coat is something one does when cold.

Educators have sought to improve inferencing. In some cases they can teach students reading comprehension strategies: create a summary, for example, or create a graphic organizer. The task provides some structure that will prompt the student to make the necessary inferences. Alternatively, students might be taught more directly to make inferences, usually by instruction to elaborate on what they read in the text,  to use cues in the text that provide clues to inferencing (e.g., words like “because,” or “so,”), and to monitor their comprehension.

​A new meta-analysis of inference instruction (Elleman, 2017) shows that it’s quite effective, but carries some important caveats…ones that I’ve mentioned before.

The overall mean effect size on comprehension of inference instruction was a healthy g = .58. Elleman also evaluated separately comprehension due to inferences, and literal comprehension, and observed a difference moderated by ability. 

As the figure shows, both skilled and less skilled readers improve in inference-making ability, but inference instruction provides a boost to understanding of things stated explicitly in the text only for less skilled readers.
​
Still more interesting to me was the report that the amount of instructional time had no impact on the effectiveness of the intervention, which I’ve shown in the graph below, compiled from a data table in the paper.

Each dot represents a study condition. Two things are notable: first, most studies entail rather little practice, but nevertheless show a large effect. Second, more practice does not lead to a larger effect.

​This finding is important because it provides an important clue to the mechanism by which this instruction helps. Practice usually helps, especially early in training. The curve looks like this

Performance improves with practice, and the curve is negatively accelerating. You get the most bang for the buck from practice early in training. The data from this meta-analysis don’t show either effect. It’s true that the range is relatively small…most studies use very little practice, so it’s harder to observe any effect of practice. That still doesn’t explain why you get such a big effect with very little practice.

​This failure to observe a practice effect is more understandable if the relationship of practice and performance for inference generation look more like this: 

The plot would look like this if what kids learn during inference instruction is easy to learn, and easy to implement, but carries a one-time benefit. For example, this instruction might prompt children to better understand the importance of making the effort to coordinate meaning across sentences. It might teach them to look for cues to inference possibilities like the word “because,” or time cues like “later.”

As Elleman notes, theories of reading have centered on the knowledge from long-term memory needed to make inferences (Kintsch, 1998) and/or the working memory capacity needed to hold information in mind simultaneously so meanings can be compared (Daneman & Carpenter, 1980). Neither of these is going to change over the course of 10 hours of instruction.

And that’s the practical implication of this meta-analysis. There’s a big benefit to inference instruction, but all of the benefit accrues rapidly. There seems to be no point in spending extended classroom time on the practice.

Similar results have been observed in meta-analyses of studies examining the impact of reading comprehension strategy instruction. Gail Lovette and I (2014) reported on nine meta-analyses of typically developing readers and of readers either identified with a reading disability, or at risk. In each case, the pattern was the same: a large effect with few hours of instruction, and no benefit to more instruction. (This piece was published as a commentary in Teachers College Record and seems to have disappeared from the website. Email me if you’d like a copy.)

This interpretation is also consistent with theories of reading comprehension. Inferences are situation specific: you can’t really teach how to make inferences because the inference to be made depends on the content of the text. Rather, you can teach them to seek and use some cues (probably a causal connection here) and you can teach them that it’s important to make inferences. But making them requires broad background knowledge in long term memory.

That is an essential goal to improve reading comprehension, but it is a goal requiring years of planning, not hours. 

Tomorrow, March 18, is Trisomy-18 awareness day. It’s important to me because one of my daughters, Esprit, has Trisomy-18. In the spirit of the day I’m going to offer just a little background for those who are unfamiliar with it, but focus mostly on one interaction small children typically have with Esprit—staring at her.

By way of background, Trisomy-18 is a chromosome disorder. Each cell in the human body has 23 pairs of chromosomes, strands of DNA. Trisomy means that there are three copies, not two, of one of the pairs. Three copies of the 21st chromosome is Trisomy-21, also called Downs Syndrome. Three copies of the 18th chromosome give you Trisomy-18, also called Edwards Syndrome. (So now you know why they picked the eighteenth of March—3/18—as Trisomy-18 awareness day. )

There’s no particular cause—it’s a fluke. There’s also no cure, and over 90% of the children born with Trisomy-18 die within the first year.

Esprit is unusual for still being here at age 13, but her profile is typical of Trisomy-18 in other ways. She cannot walk or speak, and she learned to sit upright about a year ago. Cognitively, she’s like a one-year old on many dimensions. 

My goal here is not to raise awareness about the medical side of Trisomy-18—if you’ve read to this point, you’ve come close to the end of my knowledge—but rather to consider the manner in which you are most likely to encounter a child with Trisomy-18; out and about with your own child. Older kids (and their parents) will sneak a surreptitious glance at Esprit. Many adults will smile and some will approach her, always with warmth.

But kids aged two to six are generally flummoxed and show it. Parents are usually not prepared to respond to their child’s curiosity and bafflement.

​Your five-year-old will notice that Esprit doesn’t look like other children. She has facial features typical of Trisomy-18 kids. Her head is small, her ears low-set, her chin recedes, and her eyelids droop, so she usually looks sleepy. 

But your child wouldn’t be paying attention to these facial features, because others are much more noticeable. Esprit is tiny for her age, weighing just fifty pounds, and she’s usually in a large wheelchair that provides support for her back. She also wears a TLSO around her midsection, and orthotics on her feet.

To a five-year old, this is a sight.

So usually he stares. (I'll call the child "he" to keep pronouns unambiguous, cf Esprit.) That makes the parent uncomfortable, and they try to call the child away, distract him, so he’ll unlock his gaze. The parent doesn’t say “don’t stare;” doing so would acknowledge that he’s staring, and that there’s something to stare at. Eventually, the parent might drag or chivy the child away, with the child looking back the whole time, staring.

I appreciate that you don’t want your child to stare, but ignoring his interest or trying to get the hell away doesn’t work well and sends your child the message that there’s something wrong here. And admonishing the child “it’s not nice to stare” once out of earshot will not make him accept a disabled kid as part of the shopping mall crowd next time he sees one. Kids this age are too young for that. (If there's any effect, it will be to make him a sneakier starer.)

Here’s an alternative. Encourage your child to add a social element to the staring. It’s natural and unobjectionable that he’s curious about a child who looks different. Staring feels wrong because interaction with another person demands some outward acknowledgement that there is a fellow human in front of you. You can gape at a skyscraper or a sunset, but no matter how interesting another person is to behold and for whatever reason, you must give social signals that you recognize that they are not an object.

Once you add social signals, staring doesn’t feel like staring. Staring while smiling, for example, seems perfectly appropriate. Sure, a five-year old’s voluntary smile is comically phony, but who cares? It’s the thought that counts. Or encourage your child to say “hi,” or to wave. Any of these changes the dynamic from “observation of a spectacle” to a bid for social interaction. (It will also thrill Esprit.)

More outgoing kids will ask questions, usually of my wife or I rather than Esprit. Please don’t shush them, and please don’t worry about what they will ask. Parents (and strangers) don’t expect social graces at this age, so we know we’ll hear “what’s wrong with her?” or “what’s that? (pointing to her orthotic). We’re very used to talking with children about Esprit’s disability. Questions are a way of initiating social interaction. They’re great.

​Every family’s experience is different, of course, and I’d never claim to speak for all parents of disabled children. But the next time your little one stares at someone different, give the social signal strategy a try. Let me know how it goes. 

​Everyone knows that it’s dangerous to use a cell phone while driving. The danger lies in distraction; even with a hands-free phone, the conversation saps attentional resources that should be devoted to the road.

But what if you’re not driving? Suppose you’re a student working a multi-step math problem and an announcement comes in over the school PA system. 

Lab studies provide some disquieting answers. Coming back to your main task after an interruption carries a cost both in time lag and in error rates…in one field study errors were observed where the main task was the administration of medication.

In a new study Erik Altmann and his colleagues investigated the effect of the duration of the interruption. They were especially interested in duration because delay is known to have a robust effect on memory, but is likely to have an effect on attention. (The importance of delay is explained below.)

​In this experiment, the multi-step main task involved a display of a digit and a letter, about which subjects had to make seven judgments, in a particular sequence. The graphic below explains the task. 

​Interruptions occurred randomly, on average every 6 trials (but with a lag of at least three trials). Subjects saw a letter string that they were to type on the computer keyboard. Duration of the interruption was controlled by the number of strings and the number of characters in each string. 

Let’s return to the importance of the interruption delay. Suppose you’re executing a multi-step procedure and you’re interrupted. When you return to your main task any problems you have might be due to memory or attention. If the problem is one of memory (but your attention is fully back on task) we’d expect that you might forget where you are in the sequence, e.g., you were supposed to the “R” task in the UNRAVEL sequence, but you forget and repeat the “N” task. But we wouldn’t expect you’d necessarily make a mistake when answering the “N” task, because attention is fully back on task.

But if the problem is that your distracted, we’d think that both types of problems would be more likely—an interruption would make you lose your place in the sequence (sequence errors) and make you more prone to mistakes in when answering (answer errors).

​The results were quite clear, as shown in the figure

The interruption made subjects forget where they were in the sequence, but it did not seem to affect attention much once they returned to the task, because the interruption didn’t make them more error prone.

​As in other experiments, this one showed that an interruption incurred a cost to response time upon return to the main task. In this experiment, this cost increased with delay, and in light of the error data, we’d interpret this effect as being due to subjects struggling to remember where they were in the sequence. 

This memory-based account of the effect of interruptions on sequential tasks is consistent with decades of experimental work showing that the contents of short term memory is compromised by delay.  These results don’t mean attention is not a relevant factor in interruptions, but they do speak to the relative roles of memory and attention in sequenced tasks.

So administrators...if you haven't set a policy that the PA system is silent during class, think about doing so!

I’m leery of value-added measures as a metric of individual teacher quality. Aside from the straight psychometric challenges, I’ve always worried that there’s too much that teachers give students that a VAM would miss. For example, I think of a LAUSD high school teacher who told me with considerable excitement that he had managed to get one of his kids who was near to dropping out to come to his class for five days in a row. The student wasn’t really participating in any way yet, but the teacher had a plan in mind for how to try to coax this student back to trying to engage with math just one more time, after many years of difficulty and frustration.

Who would tell this teacher “that child is a bad investment of your time?” Yet where is the VAM that will measure the value the teacher is adding to this student?

A recent paper (Master, Loeb & Wyckoff, 2017; preprint here) tried to shed light on this problem. The authors note that the short-term effects of ELA teachers are usually smaller than those of math teachers (as measured by VAM). This outcome is easy to understand from a cognitive perspective—students learn more outside of school that can be applicable to ELA tests (compared to math), and thus the contribution of a teacher and school will be relatively smaller. But the contribution of ELA and math teachers to long-term outcomes (e.g., graduation) is equivalent. Why?

One possibility is that students learn different kinds of things from each. They may learn more subject-specific content from math teachers that persist to math performance next year. But a good ELA teacher may be more likely to impart different, more persistent skills to students—they may improve their self-image as students, for example.

To examine this possibility the authors compared VAMs across years in ELA and math both within subjects and between them. In other words, if an ELA teacher was really effective, we know that effect will be observable in English class the next year. Will it be observable in math class as well?

The authors had two enormous data sets with which to investigate this question: standardized test scores from 3rd through 8th students in New York City and Miami-Dade County from 2003/04 to 2011/12.

This study, like previous studies, found that about 25 or 30% of VAM persists into the next year. In this study, those quantities were similar in math and in ELA. The startling result came when investigators used VAM in one subject to predict VAM in the other subject in subsequent years. (More precisely, they examined student achievement in year 2 in subject A, accounting for achievement in year 1 in subjects A and B, and the VAM estimates of the teachers in subjects A and B in year 1.)

As noted, about ¼ to 1/3 of the VAM in ELA from one year carried over to ELA achievement the following year. And about 46% of that effect also carried over to math achievement in Miami-Dade. In New York City, it was 70%.

But having a really effective math teacher had very little impact on ELA achievement the following year. In both districts, this carryover was around 5%.

There was good evidence that these effects persist into a third year in New York City, but in Miami-Dade the results were inconclusive, according to the researchers because measurements there were less precise.

The results point to three conclusions, given the caveats typical to this work (and often forgotten): subjects other than ELA and math were not measured, and students were not randomly assigned to teachers (and in fact, there is likely systematic bias in assignment)

First, these results help us to understand why ELA short-term VAMs are smaller than math short-term VAMs, yet the predictive value for long-run outcomes (like graduation) is the same. The ELA short-term VAMs may typically be smaller, but they contribute across subjects (which the math do not). And these cross-subject effects may last years.

Second, the authors don’t speculate much on the mechanism of transfer, but at least two routes seem plausible. First, ELA teachers may, on average, provide a bigger boost to what are usually called non-cognitive skills: self-regulation, persistence, seeing oneself as belonging in school, and so on. Second, better ELA skills—especially better reading skills like decoding, fluency, deployment of comprehension strategies, and self-monitoring of comprehension—seem likely to pay dividends in many subjects.

​Third, when it comes to policy, I’ll leave the conclusion to the authors: “educators and policymakers may miss valuable information if they rely only on short-term within-subject student learning to evaluate teachers’ “value added” to student achievement.” Student achievement gains prompted by teacher X could easily be misattributed to another teacher in another subject, years later.