There was a brief, lively thread on Twitter over the weekend concerning the definition of learning. To tip my hand here at the outset, I think this debate—on Twitter and elsewhere--is a good example of the injunction that scientists ought not to worry overmuch about definitions. That might seem backwards—how can you study learning if you aren’t even clear about what learning is? But from another perspective don’t we expect that a good definition of learning might be the result of research, rather than a prerequisite? 

The Twitter thread began when Old Andrew asked whether John Hattie’s definition (shown below) was not “really terrible."

I'll first consider this definition (and one or two others) as our instincts would dictate they be considered. Then I'll suggest that's a bad way to think about definitions, and offer an alternative. 

Hattie's definition has two undesirable features. First, it entails a goal (transfer) and therefore implies that anything that doesn’t entail the goal is not learning. This would be….weird. As Dylan Wiliam pointed out, it seems to imply that memorizing one’s social security number is not an example of learning. 

The second concern with Hattie’s definition is that it entails a particular theoretical viewpoint; learning is first shallow, and then later deep. It seems odd to include a theoretical perspective in a definition. Learning is the thing to be accounted for, and ought to be independent of any particular theory. If I’m trying to account for frog physiology, I’m trying to account for the frog and it's properties, which have a reality independent of my theory. 

The same issue applies to Kirschner, Sweller and Clark's definition, "Learning is a change in long-term memory." The definition is fine in the context of particular theories that specify what long term memory is, and how it changes. Absent that, it invites those questions: “what is long term memory? What prompts it to change?” My definition of learning seems to have no reality independent of the theory, and my description of the thing to be explained changes with the theory.

It’s also worth noting that Kirscher et al’s definition does not specify that the change in long term memory must be long-lasting…so does that mean that a change lasting a few hours (as observed in repetition or semantic priming) qualifies? Nor does their definition specify that the change must lead to positive consequences…does a change in long term memory that results from Alzheimer’s disease qualify as learning? How about a temporary change that’s a consequence of transcranial magnetic stimulation? 

I think interest in defining learning has always been low, and always for the same reason: it’s a circular game. You offer a definition of learning, then I come up with a counter-example that fits your definition, but doesn’t sit well with most people’s intuitions about what “learning” means, you revise your definition, then I pick on it again and so on. That's what I've done in the last few paragraphs, and It’s not obvious what’s gained. 

The fading of positivism in the 1950s reduced the perceived urgency (and for most, the perceived possibility) of precise definitions. The last well-regarded definition of learning was probably Greg Kimble's in his revision of Hilgard & Marquis’s Conditioning and Learning, written in 1961: “Learning is a relatively permanent change in a behavioral potentiality that occurs as a result of reinforced practice,” a formulation with its own problems.

Any residual interest in defining learning really faded in the 1980s when the scope of learning phenomena in humans was understood to be larger than anticipated, and even the project of delineating categories of learning turned out to be much more complicated than researchers had hoped. (My own take (with Kelly Goedert) on that categorization problem is here, published in 2001, about five years after people lost interest in the issue.)

I think the current status of “learning” is that it’s defined (usually narrowly) in the context of specific theories or in the context of specific goals or projects. I think the Kirschner et al were offering a definition in the context of their theory. I think Hattie was offering a definition of learning for his vision of the purpose of schooling. I can't speak for these authors, but I suspect neither hoped to devise a definition that would serve a broader purpose, i.e., a definition that claims reality independent of any particular theory, set of goals, or assumptions. (Edit, 6-28-17. I heard from John Hattie and he affirmed that he was not proposing a definition of learning for all theories/all contexts, but rather was talking about a useful way to think about learning in a school context.)

​This is as it should be, and neither definition should be confused with an attempt at a all-purpose, all-perspectives, this-is-the-frog definition.

Educators have been concerned about students' ability to vet information they find on the internet. Better put, the problem lies not just in students’ ability to vet information, but the extent to which they see the need to do so. This issue took on new urgency during the 2016 Presidential campaign, amidst charges that individuals and groups were willfully spreading false stories about candidates they opposed—stories that were accepted by readers and spread on social media.

​A new study indicates that people are somewhat more prone to be credulous when they think they are part of a group of readers.

​Subjects read purported headlines from a news organization. The headlines covered a variety of topics, and they were evenly split for veracity. Subjects were asked to mark them as True or False, or to “flag” the fact, which indicated they would find out the truth or falsehood of that statement at the end of the series. Subjects got a small monetary reward when they were right and a same-size penalty when they were wrong. Raising the flag paid nothing in one experiment and a small reward in another.

​The critical variable of interest is whether subjects thought they performed this task alone or with others, in which case they saw 100 or so other names of people they were told were doing the task at the same time. They were told their performance was not influenced by these other people; they just happened to be doing the task at the same time.

​Across 8 experiments, there was a small but consistent bias to check facts less often if subjects thought other people performed the task too. Of the 8, the most interesting experiment was one where researchers changed the presentation format to make the headlines look like part of a Facebook feed. In that case, it didn’t matter whether they told subjects they were alone; the format was enough to make it feel social, as shown on the graph below. 

The authors considered (and tested) several hypotheses about what’s going on.

1) Flagging something doesn’t signal “I want to know the answer.” It signals “I’m not sure.” So being in a group, you’re less sure of your answers because you unconscious think “in this big group, someone else really knows the answer, so I won’t pretend to know.” But the researchers collected confidence ratings when people said “true” or “false” and those didn’t differ between the answering alone and answerd with others conditions.

2)  Social loafing. When you’re in a crowd, you figure others are doing at least some of the work (in this case, checking facts) so you don’t feel the need. To check on this, the researchers tried to make subjedts feel like they stood out in the group by making the subjects name on the screen red. This had no impact on the effect.

3) The presence of others makes it a social context, and taking people at their word is a social norm. But you would predict this would also make subjects say that more statements are true and they didn’t do this.

The researchers of a safety-in-numbers interpretation. When people are in a crowd, they feels safe; “there are lots of people here, so there’s no need to be vigilant, to gather more information. If there were a problem, someone would notice it.” The researchers offer some evidence for this: they found that people performed worse on a proof-reading task when they believed they were in a group.

This theory is certainly plausible. It calls to mind Jerry Clore's theory about the impact of emotion on cognition. Clore suggests that negative emotions (anger, sadness) put you in a mode to collect more information—things are not going well (hence the negative emotion) so you need more information, and to focus on details, to figure out what’s going wrong. When you’re happy, in contrast, you’re content to do the sort of thing you usually do in the current circumstance, i.e., to rely on memory and don’t worry so much about collecting new information. Everything is fine, so you keep doing what you’re doing.

That’s all very logical, but it certainly seems to create a difficult situation. If true, the we are all of us slightly more ready to believe anything we read on social media. No, logging into Instagram does not turn you into a credulous zombie—spend five minutes on Twitter, for example, and you’ll see as much disbelief (and anger) as you care for in a day. But the social nature of social media may be one of many contributing factors.

​It highlights the fact that we are still working out the kinks in new media, trying to take advantage of the enormous increase in the our ability to create and to consume content while identifying and thwarting the drawbacks attendant to those increases. 

We cannot remind ourselves often enough that the predictions for education drawn from the learning sciences that obviously, 100%, HAVE to work…often don’t.

The latest example comes from a recent paper reporting a randomized control trial of adaptive vs. static practice in Dutch schools.

It seems self-evident that adaptive practice will be superior to static. In static practice, each student receives the same set of practice materials of graded difficulty: easy, medium, hard, with difficulty defined by the performance of a large cohort of students. In the adaptive algorithm, the proportion of questions correctly answered is factored into the probability of seeing a particular type of question: if a student is getting all of the easy questions right, what’s the point of posing more? Why not move on to more challenging content?

Adaptive practice is one of the reasons offered that personalized learning ought to lead to greater achievement.

The experiment tested 1,051 Dutch 7th-9th graders studying either Dutch, Biology, History, or Economics over the course of one academic year. Assignment to static or adaptive practice was assigned by classroom, and students were blind to condition. All students received the same instruction (a hybrid of a digital environment and traditional paper textbook) and all homework was the same. The independent variable was implemented only through extra practice; students were asked to practice at least 15 minutes per week, but any more practice than that was taken at their own initiative.  
We might expect that, because students are practicing at their own initiative, they will use the adaptive program less, given that the problems will likely be more difficult. The proportion of students who used the practice module did not differ between conditions, hovering around 90% in both cases. Students in the adaptive condition did indeed work more difficult problems and they also practiced a little bit longer per session…but they worked fewer problems than students in the static condition. Presumably, more difficult problems required more time per problem.

Nevertheless, they showed no advantage on a summative test. In fact, better prepared students (those who had passed the summative test before the experiment began) were slightly negatively impacted by the adaptive regimen compared to the static. (There was no effect for students who had failed the last summative test.)

Post-experimental interviews showed that students did not know whether their practice had been adaptive or static, and showed no difference in students’ attitudes towards the practice.

Why was there not a positive effect of adaptive testing?

One possibility is low dosage. The intervention was only 15 minutes per week and although students could have practiced more, few did. At the same time, the intervention lasted an entire school year, the N was fairly large, and an effect was observed (in the unexpected direction) for the better prepared students.

Another possibility is that the program was effective in getting challenging problems to students, but ineffective in providing instruction. Students in the adaptive condition saw more difficult problems, but they got a lot of them wrong. Perhaps they needed more support and instruction at that point, so the potential benefit of stretching their knowledge and skills was not realized.

Another possibility is that the adaptive group would have shown a benefit on a different outcome measure. As the authors note, the summative test was more like the static practice than the adapative practice. Perhaps the adapative group would have shown a benefit in their readiness and ability to learn in the next unit.

​This result obviously does not show that adaptive practice is a bad idea, or cannot be made to work well. It simply adds to the list of ideas that sound like they are more or less foolproof that turn out not to be: think spiral curriculum, or electronic textbooks. Thinking and learning is simply too complicated for us to confidently predict how a change in one variable will affect the entire cognitive and conative systems.
 

 

 
There’s a new blog post over at The 74 commenting on a “finding” that I’ve seen reported in other places (e.g., Inc and Forbes).
There are two parts to the claim.

1. “[Valedictorians] do well, but they don’t go on to change the world or lead the world.” Elsewhere these behaviors are characterized as those of "disruptors."
2. “School rewards people who follow the rules, not people who shake things up,”

This blog post would make a good final exam question for an undergraduate course in experimental methods. (If you like, head on over and see if you can find the problems in the claims.)

Problem #1: The evidence offered for the claim that valedictorians do not become “disruptors” is that a study of 81 valedictorians showed few or none became disruptors. To draw the conclusion “valedictorians don’t become disruptors” you need to show that fewer valedictorians become disruptors relative to other achievers e.g., non-valedictorians in the top quartile, or better, compare valedictorians to all students sorted by grade quartile. That few valedictorians become disruptors is expected--the baserate is low (i.e., very few people in any group would be expected to be disruptors). 

The second bit of evidence offered is that a study analyzing 700 millionaires found that their average college GPA was 2.9. First, It’s not obvious that status as a millionaire means you’re a disruptor. Second, if the criterion for disruption is income, well, it’s well-known that GPA predicts income.

Problem #2: The author not only assumes a relationship between two variables (status as a valedictorian & status as a disruptor) based on inadequate evidence, but also claims to understand the causal relationship; both are caused by a third variable, conformity. It’s great fun to propose causal mechanisms when you haven’t measured the relevant construct, but absent other evidence, it ought to be thought of in just those terms: fun, merriment, whimsy. If the relationship actually exists, we can have equal fun proposing other causal relationships; disruptors are bad at assessing risks but valedictorians are good at assessing risks; gaining status as a valedictorian makes people buy into societal norms; disruptors don’t do very well in school because they aren’t very smart—that’s why they take big risks.

See, isn’t this fun?

Maybe the book is better. If so, this is a case of careless reporting. Either way, it’s a case of careless thinking.

​The last decade has seen a huge upsurge in researcher interest in the consequences of pre-k education. That’s due, in part, to the steady increase over the last fifty years in the number of children enrolled in pre-k. In the last twenty years, that increase has been driven by children enrolled in public programs. 

The increase in publicly funded programs naturally enough sparks interest among policymakers as to whether these programs work.

That’s not an easy research question. It’s hard to track children in future years as their families move, it’s more difficult to construct reliable assessments for younger children, and there’s less agreement about what constitutes successful outcomes in pre-k than in higher grades.

Perhaps most troubling, it’s not obvious what the counterfactual is; in other words, you’d like to compare the outcome for a child if he goes to pre-k compared to the outcome for the same child if he doesn’t go to pre-k. That’s obviously impossible, so you’ll compare the child to another child who doesn’t go to pre-k….but what does that child do? Fifty years ago it was a good bet that “didn’t go to pre-k” meant the child was at home with his mother. Today, he might go to a home daycare, or be cared for by a relative. To what we should compare pre-k just isn’t so obvious.

In addition, the answers to these questions are clouded by political factors. Ones assessment of the efficacy of public pre-k programs has the potential to be influenced by one’s preconceptions of the efficacy of government programs more generally, and by preconceptions as to whether such programs ought to be within the role of government, whatever their efficacy.

Some researchers have countered the latter set of concerns by noting that effective pre-k programs can more than pay for themselves; children who attend effective pre-k programs will be less likely to drop out of school, more likely to end up in high paying jobs (and so pay more taxes), are less likely to be incarcerated, to need public assistance, and so on. Some researchers claim that pre-k programs return ten dollars or more for each dollar spent.

The pressing questions are: (1) are these claims accurate; (2) what are the characteristics of a “high quality” pre-k program; (3) can governments create and sustain pre-k programs with these features at scale?

Two groups of researchers recognized the need to bring together existing research and to provide policymakers with some answers that are, insofar as is possible, objective and devoid of political ax-grinding. This academic world is small enough that it was inevitable that each should learn about the other, and they chose to join forces. The result is a report, The Current State of Scientific Knowledge on Pre-Kindergarten Effects.

The heart of the report is a consensus statement. I offer four key conclusions from that statement here, verbatim, in red, with brief comments of my own after each conclusion. 

Studies of different groups of preschoolers often find greater improvement in learning at the end of the pre-k year for economically disadvantaged children and dual language learners than for more advantaged and English-proficient children.

That’s the counterfactual at work. Rich and poor kids would have different experiences if they were not in pre-k, with poor kids having fewer opportunities for an enriching environment than the wealthy kids.

Pre-k programs are not all equally effective. Several effectiveness factors may be at work in the most successful programs. One such factor supporting early learning is a well implemented, evidence-based curriculum. Coaching for teachers, as well as efforts to promote orderly but active classrooms, may also be helpful.

This is not a surprise…the curriculum matters, and providing training and direction to teachers helps. In the details of the report, curricular comparisons are pretty rough-cut: whole-child vs. skills-based (i.e., math, literacy or both). Whole-child curricula have not been successful in developing literacy, math, or socio-emotional skills…but it also sounds like a bit of a basket category.

Children’s early learning trajectories depend on the quality of their learning experiences not only before and during their pre-k year, but also following the pre-k year. Classroom experiences early in elementary school can serve as charging stations for sustaining and amplifying pre-k learning gains. One good bet for powering up later learning is elementary school classrooms that provide individualization and differentiation in instructional content and strategies.

This is one of the most important points. It’s saying that the oft-cited Perry and Abcederian preschool results are atypical. Absent continued intervention, you should expect fadeout of the pre-k benefit. I’ve blogged about relevant studies before, but the main point is intuitive. Academic and social outcomes are a product not just of school experiences, but also of home and other out-of-school experiences. If those out-of-school experiences are not especially enriching, children benefit (to a greater or lesser degree) from substituting pre-k experiences. The out-of-school experiences continue to matter after pre-k.

When you spell it out, the counter-assumption sounds a little strange: children may be behind their peers in important knowledge and skills by age four, but with a good year or two of pre-k they catch up, and can keep pace with their wealthier peers thereafter. This assumption makes sense if you ascribe an outsize importance to the first few years of development, e.g., as Freud did. But that assumption isn’t right. Outside-of-school experiences continue to matter after age six.

Convincing evidence shows that children attending a diverse array of state and school district pre-k programs are more ready for school at the end of their pre-k year than children who do not attend pre-k. Improvements in academic areas such as literacy and numeracy are most common; the smaller number of studies of social-emotional and self-regulatory development generally show more modest improvements in those areas.

There’s more than one way to do this sucessfully, and the public sector can get it right. There are probably three reasons (at least) that evidence is better for literacy and numeracy than for socio-emotional learning. First, we know better how to teach numbers and letters because we’ve been at it longer. Second, there’s less happening at home that might conflict with the learning happening at school when it comes to these skills.  Third, the relative contributions of environment vs. heritable factors (e.g., temperament) is probably larger for literacy and numeracy. 

In addition to the consensus statement, the report includes brief but meaty analyses of questions important to pre-k policy, for example:

• How can scale-up be improved? (Training and monitoring at a level of detail similar to that used during initial design.)
• Is the economic return really 10 dollars or more, per dollar spent? (That figure may apply to small scale programs operating in the 1960’s. Today, with a scaled up program expect more like 2-4 dollars per dollar spent).
• Should pre-k be targeted or universal? (It depends on the circumstances in the state or district—both are effective).

If you have even a passing interest in pre-k, I recommend this report to you. 

​As digital devices have decreased in price, they have become more available to more children. The impact of this availability on children’s social lives have been debated with vigor, often with gloomy foreboding. The concern centers on the possibility that online activities are absorbing so much of children’s time that little is left for other worthy pursuits, e.g., face-to-face conversations.

​But data to inform these opinions have been lacking. We know children who own digital devices (and now, that’s most of them) spend a great deal of time interacting with them—estimates for teens are around 10 hours per day or more. To some, it’s self-evident that must carry a cost, and the cost is assumed to be social. Kids are isolated by digital devices, kids no longer know how to speak face to face, and so on (e.g., here.)

But this was really punditry and speculation. Hard data were lacking, especially hard data to which causality could be ascribed. That is, we might see kids who were socially withdrawn who spent a lot of time in online social pursuits, but such a correlation would be hard to interpret. Were online social interactions replacing face-to-face interactions and causing social isolation, or would this child be withdrawn in any event, and online communication is actually a more social activity than this child would otherwise engage in?

Clearly, a true experiment would help clear the matter up: take 1,000 middle-schoolers, give 500 of them a computer, and see what happens over the course of a school year. Well, darned if someone didn’t do that (Fairlie & Kalil, 2017).

The experimenters administered a survey at the start and the end of the school year. They also had access to administrative data regarding school participation.

It should be noted that children in the control group did have access to computer time at school and elsewhere, and some families purchased computers on their own during the year. The researchers tracked these confounds as best they could. Children given the computers did indeed spend more computer time per week than control kids.

The results:
Friends: The results showed that kids given computers did not report communicating with or hanging out with their friends less…in fact, they reported spending more time with friends.
Social groups: Giving kids a computer had no impact on the probability that they would be part of a sports team, club, or music group.
School participation: There was also no effect of home computers on the number of days absent from school (or tardy), or days suspended.
Competing activities: Self-reported TV time, homework time and leisure reading were unaffected.
Social networking: Children with a home computer were more likely to have a social network page and reported spending more time on social networks. There was also a statistically nonsignificant increase in the probability of reporting cyberbullying, a result that is difficult to interpret because the overall mean was so low (less than 1%).

A few caveats of these conclusions should be borne in mind. First, the study only lasted for one school year. Second, having a smart phone, with the constant access it affords, may yield different results. Third, children were given a computer, but not Internet access. Some kids had it anyway, but the more profound effects may come from online access.

All that said, I am less frightened than some by the threat that digital technologies will eat children’s minds, or making them anti-social zombies. I wrote The Reading Mind before this study was published, but as I put together the data, I suggested that digital activities were not replacing reading, and that’s true for two reasons. First, reading provides a different sort of pleasure than gaming or social networking. If you like reading, that pleasure is only available by reading. Second, digital technology has not reduced reading for most kids because most kids don’t read anyway.

This later point is the most salient to me, and has most influenced how I raise my own kids. It’s not that most digital technologies are so terrible, but most of what my kids can do online is less preferable to me than what they can do offline. I’d rather they make something, take a bike ride, or read a book. But if they horse around on the computer, that’s no worse (or better) than watching Say Yes to the Dress, my ten-year-old's latest television infatuation.

My real concern about digital technology use in teens is hard to quantify. When I was a teen I, like most, probably assigned too much value to the opinions of my peers.  They necessarily stopped influencing me when I got off the school bus, and I was influenced mostly by my parents and two sisters. I don’t relish the thought of children taking their peer groups home with them in their pockets, influencing them 24/7, and diminishing the impact of their families. 

Suppose a teacher asks students to read a text and he wants to be sure that children notice X, Y, and Z about the text. One strategy would be to pose questions before they read, the answers to which are X, Y, and Z. The problem with this strategy is that it increases the chances they will notice and remember X, Y, and Z, and decreases the chances they will notice everything else (e.g., Pressley et al, 1990).

The interpretation of this phenomenon is straightforward: posing questions keys the reader’s attention to certain content, and they pay less attention to everything else. Indeed, when researchers put in boldface type information in the text that was not prequestioned, but would later appear on the test, the disadvantage vanished (Richland et al, 2009). The boldface drew attention to the content that would have otherwise been skimmed.

In some cases, the teacher may view that as a worthwhile tradeoff, but that’s probably infrequent—she’d prefer the boost to X, Y, and Z occur without the cost. 

A new paper (Carpenter & Toftness, 2017) may suggest a strategy to avoid the problem, although the experiment actually tested memory for video content. 

The researchers tested 85 undergraduates, each of whom watched a video lasting about eight minutes. The video was divided into three segments describing the original settlement of Easter Island, religious practices there, and the arrival of outsiders. The researchers generated four questions about each segment. Half of the subjects were asked to guess at the answer to two randomly selected questions about the upcoming segment. The control group simply pushed a button to continue on with the video. At the end of the video, subjects in both groups answered all 12 questions about the video in random order.

​Just as in prior experiments using text rather than video, asking people about specific information before seeing the video made it more likely they would learn that information once they watched the video. (That’s the tallest bar at far left). 

More interesting is that the prequestioned group was also more likely to successfully answer questions for which they had not been precued. 

Why was the cost not observed? Carpenter & Toftness emphasize that a reader controls the pace of reading; the reader can skip over content that she deems less important, and read again content that matters more. The viewer does not control the pace of video; important content might pop up at any time, and once it’s past it cannot be reviewed. So the viewer is more likely to attend closely to the whole thing. 

The researchers note that this attention hypothesis can help explain other instances in the research literature where the prequestioning deficit for other content is not observed. For example, Pressely et al (1990) asked subjects to rate each paragraph for interest. Little & Bjork (2016) showed that non-prequestioned information got a boost if it was mentioned in a prequestion, although not the target to-be-learned information. 

So in the final analysis, can teachers pose prequestions in a way that boosts memory for targeted content but doesn’t incur a cost for everything else? 

​In principle, yes. With the right type of material (boldfaced, or video), you're good, and asking for interest judgments works too. But of course none of these may be practicable as the teacher envisions the lesson plan. 

This work suggests that a teacher could devise another strategy that uses prequestions without cost--a mental task that requires attention to all content, not just the prequestioned would do the trick. True in principle, but the bottom line on prequestions a the moment seems to be “proceed with caution.” 

It's been about ten years since college students on most campuses began to take notes on a laptop in lecture classes. So that's about 10 years of professors fretting about it.

It seems obvious that students are, at the least, themselves distracted during lectures--supporting anecdotal evidence can be collected by popping into the back of any large lecture hall and observing laptop content; you'll see plenty of social media websites open, students responding to messages or email, and a surprising number of people shopping.

​More insidious, some data indicate that using a laptop is distracting to students sitting behind the multi-tasking student.

But most data on the consequences of laptop use are correlational; for example, students who report taking notes on laptops earn lower grades than those who don't, and observational data show that students have non-course related software open during class (e.g., Kraushaar & Novak, 2010). The obvious problem is that laptop use may not be causing low grades; laptop use may be more appealing to those who would earn lower grades anyway. 

Two new studies using different methods suggest that laptop use does, indeed, incur a cost to students.

In one (Carter et al., 2017) students in sections of a required introductory economics course were randomly assigned to use a laptop or to refrain from doing so. In a clever twist, some classrooms were asked to take notes on tablets with the tablet lying flat, hence reducing the likelihood that the screen would distract neighboring students.

The final examination was the same across sections, so scores on the exam served as the outcome measure for the study. Laptop (or tablet) use was associated with lower scores, d = -.18 (controlling for demographic variables, GPA, and ACT scores at admission). It is noteworthy that all of the effect was carried by performance on the multiple choice and short-answer portion of the exam, with no effect on essay grade. The researchers noted that essay grading is not objective and professors might center their grades on student performance. 

We can have greater confidence that laptop use is really causal here, given that class sections were randomly assigned to use them or not. But there are some other caveats of interpretation. It's possible that teachers teach differently when they know their students are taking notes on devices. We should also be cautious in generalizing beyond the institution (West Point) and the class (economics). For example, perhaps economics requires the drawing of many figures and graphs, a process that's clumsy on a device. Perhaps the same experiment conducted in an English literature class would show no effect. Or maybe laptop use offers a small benefit to most students, but there's a minority for whom the forced use of a laptop incurs a big cost.

The second study (Patterson & Patterson, 2017) uses a different, complimentary method that avoids these problems (but has others).

​The researchers noted that, at their institution, different classes required laptops (15%), forbade them (2%), or made them optional (83%). The researchers reasoned that a student who had a laptop-optional class might be nudged into using one if she had a laptop-required class the same day. After all, she would already have the laptop with her, so why not use it? And indeed, their survey showed that students were about 20% more likely to use a laptop under those circumstances.

And students were 48% less likely to use laptop in a laptop-optional class if they had another class the same day that forbade laptop use. 

So researchers compared grades in a class where some students are biased to use laptops and others were biased not to use them--biased, not forced, so if a student believes he really takes better notes on a laptop, he can use it. And both the laptop-user and non-user are in the same class, with the same opportunities to be distracted by other students, and experiencing the same lecture from the professor. 

Researchers analyzed about 5,500 grades in lap-top optional courses from undergraduate and masters students over the course of six semesters. All were enrolled at the same liberal arts college. 

The effect of having a laptop-required course at another time was d = -.32 to -.46.  Having a laptop-prohibited course the same day was associated with a positive effect, d = .14 to .25. (These effects controlled for gender, ethnicity, age, course load, course schedule difficulty, major, and GPA.)

Patterson & Patterson found that the negative effect was larger for weaker students--in fact, there was little or no effect for stronger students. They also found that laptop use (or more precisely, having a laptop-required course the same day) had a larger negative effective in quantitative classes. 
​
So we have two studies--a randomized control trial, and a quasi-experimental design--to add to the correlational studies showing that students are better off taking notes by hand than doing so on a laptop. None is perfect, but different designs have different flaws. This desirable state of affairs is referred to as converging operations--different types of experiments all point to the same conclusion. ​

In closing, it's worth remembering that this study does not concern all classroom laptop use, but rather one function: taking notes in a lecture course. Still, considering that most time in introductory college courses is spent listening to lectures, the impact of this work ought to be consequential. 

Once children are fluent decoders, the most frequent problem in reading is poor comprehension due to a failure to make inferences. Even seemingly straightforward anaphoric inferences can elude these students: they might read “Bob gave Tamisa some of his snack because she was hungry” and still be unsure of the referent for “she.” Other inferences require bridging information from long term memory, and these are still more challenging. For example, “Kevin said he was cold. Zeke gave him his coat.” Even if these two sentences are understood, each on its own, deeper comprehension entails making the (probably accurate) inference that Zeke gave Kevin the coat because Kevin said he was cold, which requires knowing that putting on a coat is something one does when cold.

Educators have sought to improve inferencing. In some cases they can teach students reading comprehension strategies: create a summary, for example, or create a graphic organizer. The task provides some structure that will prompt the student to make the necessary inferences. Alternatively, students might be taught more directly to make inferences, usually by instruction to elaborate on what they read in the text,  to use cues in the text that provide clues to inferencing (e.g., words like “because,” or “so,”), and to monitor their comprehension.

​A new meta-analysis of inference instruction (Elleman, 2017) shows that it’s quite effective, but carries some important caveats…ones that I’ve mentioned before.

The overall mean effect size on comprehension of inference instruction was a healthy g = .58. Elleman also evaluated separately comprehension due to inferences, and literal comprehension, and observed a difference moderated by ability. 

As the figure shows, both skilled and less skilled readers improve in inference-making ability, but inference instruction provides a boost to understanding of things stated explicitly in the text only for less skilled readers.
​
Still more interesting to me was the report that the amount of instructional time had no impact on the effectiveness of the intervention, which I’ve shown in the graph below, compiled from a data table in the paper.

Each dot represents a study condition. Two things are notable: first, most studies entail rather little practice, but nevertheless show a large effect. Second, more practice does not lead to a larger effect.

​This finding is important because it provides an important clue to the mechanism by which this instruction helps. Practice usually helps, especially early in training. The curve looks like this

Performance improves with practice, and the curve is negatively accelerating. You get the most bang for the buck from practice early in training. The data from this meta-analysis don’t show either effect. It’s true that the range is relatively small…most studies use very little practice, so it’s harder to observe any effect of practice. That still doesn’t explain why you get such a big effect with very little practice.

​This failure to observe a practice effect is more understandable if the relationship of practice and performance for inference generation look more like this: 

The plot would look like this if what kids learn during inference instruction is easy to learn, and easy to implement, but carries a one-time benefit. For example, this instruction might prompt children to better understand the importance of making the effort to coordinate meaning across sentences. It might teach them to look for cues to inference possibilities like the word “because,” or time cues like “later.”

As Elleman notes, theories of reading have centered on the knowledge from long-term memory needed to make inferences (Kintsch, 1998) and/or the working memory capacity needed to hold information in mind simultaneously so meanings can be compared (Daneman & Carpenter, 1980). Neither of these is going to change over the course of 10 hours of instruction.

And that’s the practical implication of this meta-analysis. There’s a big benefit to inference instruction, but all of the benefit accrues rapidly. There seems to be no point in spending extended classroom time on the practice.

Similar results have been observed in meta-analyses of studies examining the impact of reading comprehension strategy instruction. Gail Lovette and I (2014) reported on nine meta-analyses of typically developing readers and of readers either identified with a reading disability, or at risk. In each case, the pattern was the same: a large effect with few hours of instruction, and no benefit to more instruction. (This piece was published as a commentary in Teachers College Record and seems to have disappeared from the website. Email me if you’d like a copy.)

This interpretation is also consistent with theories of reading comprehension. Inferences are situation specific: you can’t really teach how to make inferences because the inference to be made depends on the content of the text. Rather, you can teach them to seek and use some cues (probably a causal connection here) and you can teach them that it’s important to make inferences. But making them requires broad background knowledge in long term memory.

That is an essential goal to improve reading comprehension, but it is a goal requiring years of planning, not hours. 

Tomorrow, March 18, is Trisomy-18 awareness day. It’s important to me because one of my daughters, Esprit, has Trisomy-18. In the spirit of the day I’m going to offer just a little background for those who are unfamiliar with it, but focus mostly on one interaction small children typically have with Esprit—staring at her.

By way of background, Trisomy-18 is a chromosome disorder. Each cell in the human body has 23 pairs of chromosomes, strands of DNA. Trisomy means that there are three copies, not two, of one of the pairs. Three copies of the 21st chromosome is Trisomy-21, also called Downs Syndrome. Three copies of the 18th chromosome give you Trisomy-18, also called Edwards Syndrome. (So now you know why they picked the eighteenth of March—3/18—as Trisomy-18 awareness day. )

There’s no particular cause—it’s a fluke. There’s also no cure, and over 90% of the children born with Trisomy-18 die within the first year.

Esprit is unusual for still being here at age 13, but her profile is typical of Trisomy-18 in other ways. She cannot walk or speak, and she learned to sit upright about a year ago. Cognitively, she’s like a one-year old on many dimensions. 

My goal here is not to raise awareness about the medical side of Trisomy-18—if you’ve read to this point, you’ve come close to the end of my knowledge—but rather to consider the manner in which you are most likely to encounter a child with Trisomy-18; out and about with your own child. Older kids (and their parents) will sneak a surreptitious glance at Esprit. Many adults will smile and some will approach her, always with warmth.

But kids aged two to six are generally flummoxed and show it. Parents are usually not prepared to respond to their child’s curiosity and bafflement.

​Your five-year-old will notice that Esprit doesn’t look like other children. She has facial features typical of Trisomy-18 kids. Her head is small, her ears low-set, her chin recedes, and her eyelids droop, so she usually looks sleepy. 

But your child wouldn’t be paying attention to these facial features, because others are much more noticeable. Esprit is tiny for her age, weighing just fifty pounds, and she’s usually in a large wheelchair that provides support for her back. She also wears a TLSO around her midsection, and orthotics on her feet.

To a five-year old, this is a sight.

So usually he stares. (I'll call the child "he" to keep pronouns unambiguous, cf Esprit.) That makes the parent uncomfortable, and they try to call the child away, distract him, so he’ll unlock his gaze. The parent doesn’t say “don’t stare;” doing so would acknowledge that he’s staring, and that there’s something to stare at. Eventually, the parent might drag or chivy the child away, with the child looking back the whole time, staring.

I appreciate that you don’t want your child to stare, but ignoring his interest or trying to get the hell away doesn’t work well and sends your child the message that there’s something wrong here. And admonishing the child “it’s not nice to stare” once out of earshot will not make him accept a disabled kid as part of the shopping mall crowd next time he sees one. Kids this age are too young for that. (If there's any effect, it will be to make him a sneakier starer.)

Here’s an alternative. Encourage your child to add a social element to the staring. It’s natural and unobjectionable that he’s curious about a child who looks different. Staring feels wrong because interaction with another person demands some outward acknowledgement that there is a fellow human in front of you. You can gape at a skyscraper or a sunset, but no matter how interesting another person is to behold and for whatever reason, you must give social signals that you recognize that they are not an object.

Once you add social signals, staring doesn’t feel like staring. Staring while smiling, for example, seems perfectly appropriate. Sure, a five-year old’s voluntary smile is comically phony, but who cares? It’s the thought that counts. Or encourage your child to say “hi,” or to wave. Any of these changes the dynamic from “observation of a spectacle” to a bid for social interaction. (It will also thrill Esprit.)

More outgoing kids will ask questions, usually of my wife or I rather than Esprit. Please don’t shush them, and please don’t worry about what they will ask. Parents (and strangers) don’t expect social graces at this age, so we know we’ll hear “what’s wrong with her?” or “what’s that? (pointing to her orthotic). We’re very used to talking with children about Esprit’s disability. Questions are a way of initiating social interaction. They’re great.

​Every family’s experience is different, of course, and I’d never claim to speak for all parents of disabled children. But the next time your little one stares at someone different, give the social signal strategy a try. Let me know how it goes.