There was a brief, lively thread on Twitter over the weekend concerning the definition of learning. To tip my hand here at the outset, I think this debate—on Twitter and elsewhere--is a good example of the injunction that scientists ought not to worry overmuch about definitions. That might seem backwards—how can you study learning if you aren’t even clear about what learning is? But from another perspective don’t we expect that a good definition of learning might be the result of research, rather than a prerequisite? 

The Twitter thread began when Old Andrew asked whether John Hattie’s definition (shown below) was not “really terrible."

I'll first consider this definition (and one or two others) as our instincts would dictate they be considered. Then I'll suggest that's a bad way to think about definitions, and offer an alternative. 

Hattie's definition has two undesirable features. First, it entails a goal (transfer) and therefore implies that anything that doesn’t entail the goal is not learning. This would be….weird. As Dylan Wiliam pointed out, it seems to imply that memorizing one’s social security number is not an example of learning. 

The second concern with Hattie’s definition is that it entails a particular theoretical viewpoint; learning is first shallow, and then later deep. It seems odd to include a theoretical perspective in a definition. Learning is the thing to be accounted for, and ought to be independent of any particular theory. If I’m trying to account for frog physiology, I’m trying to account for the frog and it's properties, which have a reality independent of my theory. 

The same issue applies to Kirschner, Sweller and Clark's definition, "Learning is a change in long-term memory." The definition is fine in the context of particular theories that specify what long term memory is, and how it changes. Absent that, it invites those questions: “what is long term memory? What prompts it to change?” My definition of learning seems to have no reality independent of the theory, and my description of the thing to be explained changes with the theory.

It’s also worth noting that Kirscher et al’s definition does not specify that the change in long term memory must be long-lasting…so does that mean that a change lasting a few hours (as observed in repetition or semantic priming) qualifies? Nor does their definition specify that the change must lead to positive consequences…does a change in long term memory that results from Alzheimer’s disease qualify as learning? How about a temporary change that’s a consequence of transcranial magnetic stimulation? 

I think interest in defining learning has always been low, and always for the same reason: it’s a circular game. You offer a definition of learning, then I come up with a counter-example that fits your definition, but doesn’t sit well with most people’s intuitions about what “learning” means, you revise your definition, then I pick on it again and so on. That's what I've done in the last few paragraphs, and It’s not obvious what’s gained. 

The fading of positivism in the 1950s reduced the perceived urgency (and for most, the perceived possibility) of precise definitions. The last well-regarded definition of learning was probably Greg Kimble's in his revision of Hilgard & Marquis’s Conditioning and Learning, written in 1961: “Learning is a relatively permanent change in a behavioral potentiality that occurs as a result of reinforced practice,” a formulation with its own problems.

Any residual interest in defining learning really faded in the 1980s when the scope of learning phenomena in humans was understood to be larger than anticipated, and even the project of delineating categories of learning turned out to be much more complicated than researchers had hoped. (My own take (with Kelly Goedert) on that categorization problem is here, published in 2001, about five years after people lost interest in the issue.)

I think the current status of “learning” is that it’s defined (usually narrowly) in the context of specific theories or in the context of specific goals or projects. I think the Kirschner et al were offering a definition in the context of their theory. I think Hattie was offering a definition of learning for his vision of the purpose of schooling. I can't speak for these authors, but I suspect neither hoped to devise a definition that would serve a broader purpose, i.e., a definition that claims reality independent of any particular theory, set of goals, or assumptions. (Edit, 6-28-17. I heard from John Hattie and he affirmed that he was not proposing a definition of learning for all theories/all contexts, but rather was talking about a useful way to think about learning in a school context.)

​This is as it should be, and neither definition should be confused with an attempt at a all-purpose, all-perspectives, this-is-the-frog definition.

NPR, the Daily Mail, and other outlets are trumpeting the results of a study published  in Computers and Human Behavior: The spin is that digital devices leave kids emotionally stunted. But that conclusion is not supported by the study which is, in fact, pretty poorly designed.

Researchers examined kids' ability to assess non-verbal emotion cues from still photos and from video scenes from which dialog had been removed. These assessments were made pre- and post-intervention.

The intervention is where things get weird. The press has it that the main intervention was the removal of electronic devices from children's lives for five days. In fact, the experimental group went to a sort of educational nature camp call the Pali Institute. While control subjects went to their regular school, experimental subjects participated in activities like these:

This study could almost serve as a test question in an undergraduate research methods course. In the results section, the authors conclude "We found that children who were away from screens for five days with many opportunities for in-person interaction improved significantly in reading facial emotion." As should be obvious from the Table, there were a host of differences between what the experimental kids and the control kids experienced.

In the discussion the authors do allow "We recognize that the design of this study makes it challenging to tease out the separate effects of the group experience, the nature experience, and the withdrawal of screen-time." But then go on to say "but it is likely that the augmentation of in-person communication necessitated by the absence of digital communication significantly contributed to the observed experimental effect." That's a mere wish. We in fact cannot draw any conclusions about the source of effect.

It's a shame that news outlets are not more discriminating in how they report this sort of work. 

This column originally appeared on RealClearEducation.com on July 10, 2014.

When I talk to educators about research, their most common complaint (by a long shot) is that they are asked to implement new interventions (a curriculum, a pedagogical technique, a software product, whatever), and are offered no reason to do so other than a breezy “all the research supports it.” The phrase is used as a blunt instrument to silence questions. As a scientist I find this infuriating because it abuses what ought to be a serious claim—research backs this—and in so doing devalues research. It’s an ongoing problem (see Jessica & Tim Lahey’s treatment here) that’s long concerened me.

In fact, the phrase “research supports it” invites questions. It implies that we can, in a small way, predict the future. It claims “if we do X, Y will happen.” If I take this medication, my ear infection will go away. If we adopt this new curriculum, kids will be more successful in learning math. Saying “research supports it” implies that you know not only what the intervention is, but you have at least a rough idea of what outcome you expect, the likelihood that it will happen, and when it will happen.

I offer the following list of rights for educators who are asked to change what they are doing in the name of research, whether it’s a mandate handed down from administrator to teacher or from lawmaker to administrator.

1.       The right to know what is supposed to improve. What problem is being solved? For example, when I’ve been to schools or districts implementing a one-to-one tablet/laptop policy, I’ve always asked what it’s meant to do. The modal response is a blank look followed by the phrase “we don’t want our kids left behind.” Behind in what? In what way are kids elsewhere with devices zooming ahead?

2.       The right to know the means by which improvement will be measured. How will we know things are getting better? If you’re trying to improve students’ understanding of math, for example, are you confident that you have a metric that captures that construct? Are you sure scores on that metric will be comparable in the future to those you’re looking at now? How big an increase will be deemed a success?

3.       The right to know the approximate time by which this improvement is expected. A commitment to an intervention shouldn’t be open-ended. At some point we must evaluate how it’s going.

4.       The right to know what will be done if the goal is or is not met. Naturally, conditions may change, but let’s have a plan. If we don’t meet our target, will we quit? Keep trying for a while? Tweak it?

5.       The right to know what evidence exists that the intervention will work as expected. Is the evidence from actual classrooms or is it laboratory science (plus some guesswork)? If classrooms, were they like ours? In how many classrooms was it tried?

6.       The right to have your experience and expertise acknowledged. If the intervention sounds to you and your colleagues like it cannot work, this issue should be addressed in detail, not waved away with the phrase “all the research supports it.” The fact that it sounds fishy to experienced people doesn’t mean it can’t work, but whoever is pitching it should have a deep enough understanding of the mechanisms behind the intervention to be able to say why it sounds fishy, and why that’s not a problem.

This list is not meant to dictate criteria that must be met before an intervention should be tried, but rather what information ought to be on the table. In other words, the information provided in each category need not unequivocally support the intervention for it to be legitimate. For example, I can imagine an administrator admitting that the research support for an intervention is absent, yet mounting a case for why it should be tried anyway.

This list should also be considered a work in progress. I invite your additions or emendations.

This column was originally published on RealClearEducation.com on May 13, 2014.

Last week I discussed the case Carl Wieman made that education research and physics have more in common than people commonly appreciate. I mentioned in passing Wieman’s suggestion that random control trials—often referred to as the “gold standard” of evidence—are not the be-all and end-all of research, and that qualitative research contributes too. But I didn’t say how it contributes, and neither did Wieman (at least in the paper I discussed). Here, I offer a suggestion.

Qualitative research is usually contrasted with quantitative research. In quantitative research the dependent variable—that is, the outcome—is measured as a quantity, usually using a measurement that is purportedly objective. In qualitative research, the outcome is not measured as a quantity, but as a quality.

For example, suppose I’m curious to know what students think about their school’s use of technology. In a quantitative study, I might develop a questionnaire that solicits student’s opinions about, say, the math and reading software they’ve been using, and also asks students to compare them to the paper versions they used last year. I’d probably try to get most or all of the students in the school to respond. I would end up with ratings which I can treat as quantitative data.  

Or I could do a qualitative study, in which I use focus groups to solicit their opinions. The result of this research is not numeric ratings, but transcripts of what people said. I’d try to find common themes in what was said. Because focus groups are time-consuming, I’d probably talk to just a handful of students.

People who criticize qualitative research treat it as a weak version of quantitative methods. For example, a critic would point out that you can’t trust the results of the focus groups, because I might have, by chance, interviewed students with idiosyncratic views. Another problem is that focus groups are not very objective; responses are a product of the dynamic between the interviewer and the subjects, and among the subjects themselves.

These complaints are valid, or would be if we hoped to treat the focus group results the way we treat the outcomes of other experiments. I suggest we should not.

Here’s a simple graphic I used in a book to describe how science works. We begin with observations about the world. These can come from our casual, everyday observations or from previous experiments. We then try to synthesize those observations into a theory; we try to find simplifying rules that describe the many observations we’ve made. This theory can be used to generate a prediction, something we haven’t yet observed, but ought to. Then we test the prediction, and that test leads to a new observation, a new fact about the world. And we continue around the circle, always seeking to refine the theory.

The criticism of qualitative research is that it does not provide a very reliable test of a theory. But I think it’s better viewed as providing a new observation of the world.

An advantage of qualitative over quantitative data is the flexibility in how its collected. If I want to know what students think of the new technology instruction and I create a quantitative scale to measure it, I’m really guessing that I know the important dimensions of student opinion. I may try to capture their views on effectiveness and ease-of-use, for example, but what students really care about is the fact that so many websites are blocked.

Qualitative studies allow the subject to tell you what he thinks is important. For example in this qualitative study, students suggested that the schools policy on blocking websites affected their relationships with teachers—they felt untrusted. Maybe that’s the kind of thing a researcher would have been looking for, but I doubt it.

In addition, qualitative data tend to be richer. I’m letting the subject describe in his or her own words what she thinks, rather than asking her to select from reactions that I’ve framed.

Naturally, either type of research—qualitative or quantitative--can be poorly conducted. Each has its own rules of the road. It’s important to judge the quality of research by its own set of best-practice rules, and to judge it by how well it fulfils the purpose to which it is suited.

That’s why I believe qualitative research has an undeserved bad reputation. It is judged by standards of quantitative research, and deemed unable to serve purposes that ought to be served by quantitative research. But I agree with Wieman that qualitative research, well-conducted, makes a valuable contribution, one to which quantitative research is ill-suited.

This post first appeared at RealClearEducation.com on May 6, 2014

As someone who spends most of their time thinking about the application of scientific findings to education, I encounter plenty of opinions about the scientific status of such efforts. Almost always, a comparison is made between the rigor of “hard” sciences and education. What varies is the accompanying judgment: derision for education researchers or despondency about the difficulty of the enterprise. In a recent article, physicist Carl Wieman (2014) offers a different perspective on the issue, suggesting that the difference between research in education and in physics is smaller than you might guess.

In education, Wieman is best known for refining and popularizing techniques to have college students better engage in large lecture courses. He started his career as a physicist, producing work that culminated in a Nobel prize. So he has some credentials in talking about the work of “hard” scientists.

Wieman begins by making clear what he takes to be the outcome of good science: predictive power. Can you use the results of your research to predict with some accuracy what will happen in a new situation? A common mistake is to believe that in education one ought to be able to predict outcomes for individual students; not necessarily so, any more than a physicist must be able to predict the behavior of each atom. Prediction in aggregate—a liter of gas or a school of children—is still an advance.

Wieman’s other points follow from his strong emphasis on prediction.

First, it follows that “rigorous” methods are any that contribute to better prediction. You don’t state in the absolute that randomized controlled trials are better than qualitative research. They provide different types of information in a larger effort to allow prediction.

Second, the emphasis on prediction frames the way one thinks about the messiness of research. Education research is often portrayed as inherently messy because there are so many variables at play. Physics research, in contrast, is portrayed as better controlled and more precise because there are many fewer variables that matter. Wieman argues this view is a misconception.

Physics seems tidy because you’ve probably only studied physics in textbooks, where everything is worked out: in other words, where no extraneous variables are discussed as possibly mattering. When the work that you study was first being conducted, it was plenty messy: false leads were pursued, ideas that (in retrospect) are self-contradictory were taken seriously, and so on. The same is true today: the frontiers of physics research is messy. “Messy” means that you don’t have a very good idea of which variables are important in gaining the predictive power that characterizes good science.

Third, Wieman suggests that bad research is the same in physics and education. Research is bad when the researcher has failed to account for factors that, based on prior research, he or she should have known to include. There is plenty of bad research in the hard sciences. People aren’t stupid; it’s just that science is hard.

I agree with Wieman that differences in “hardness” are mostly illusory. (That’s why I’ve been putting the term in quotation marks.) The fundamentals of the scientific method don’t differ much wherever they are applied. I also agree that people (usually people uninvolved in research) are too quick to conclude that, compared to other fields, a higher proportion of education research is low-quality. Come to a meeting of the Society for Neuroscience and I’ll show you plenty of studies that were poorly conceived, poorly controlled, or were simply wheel-spinning and will be ignored.

Wieman does ignore a difference between physics and education that I take to have important consequences: physics (and other basic sciences) strive to describe the world as it is, and so strive to be value-neutral. Education is an applied science; it is in the business of changing the world, making it more like it ought to be. As such, education is inevitably saturated with values.

Education policy would, I think, benefit, with a greater focus on the true differences between education and other varieties or research, and a reduced focus on the phantom differences in rigor.

Reference:

Wieman, C. E. (2014). The similarities between research in education and research in the hard sciences. Educational Researcher, 43, 12-14.  

Last week Dave Grissmer and I published an op-ed on universal pre-k. We didn’t take it as controversial that government support for pre-K access is a good idea. As Gail Collins noted, when President Obama mentioned early education in his State of the Union address, it was one of the few times John Boehner clapped. Even better, there are good data indicating that, on average, state programs help kids get ready to learn math and to read in Kindergarten (e.g., Gormley et al, 2005; Magnuson et al, 2007).

Dave and I pointed out that the means do show gains, but state programs vary in their effectiveness. It’s not the case that any old preschool is worth doing, and that’s why everyone always says that preschool must be “high quality.” But exactly how to ensure high quality is not so obvious.

One suggestion we made was made was to capitalize on what is already known. The Department of Education has funded preK research for decades. Dave and I merely claimed that it had yielded useful information. Let me give an example here of the sort of thing we had in mind.

A recent study (Weiland & Yoshikawa, 2013) reported research that was notable in this respect: important decisions and procedures concerning the programs were made by the people and in the way such decisions will likely to be made as state preK programs expand or are initiated. The district was Boston Public Schools, and they offer preK for any child of age—there is no restriction based on income. The district:  

1.       picked the curriculum.
2.       figured out how to implement the curriculum at scale without any input from its developers.
3.       developed its own coaching program for teachers, meant to ensure that the curricula were implemented effectively.

The second and third points are especially important, as the greatest challenge in education research has been bringing what look like useful ideas to scale.  It’s not certain why that’s so, but one good guess is that as you scale up, the people actually implementing the curriculum have little or no contact with the person who developed it. So it’s harder to tell exactly how it’s supposed to go.

Naturally, schools and classrooms will want to tweak the program here and there to make it a better fit for their school or classroom. They will use their judgment as to which changes won’t affect the overall integrity of the program, but the voice of the developer of the curriculum is probably important in this conversation.

Boston Public Schools picked Opening the World of Literacy for their prereading and language program; there were few data for the program, and they were somewhat mixed. For mathematics, they picked Building Blocks, which had both more research and a stronger track record of success.

Weiland and Yoshikawa measured the progress of 2,018 children in 238 classrooms during the 2008/09 school year. They found moderate to large gains in language, pre-reading, and math skills. There was even a small effect in executive function skills, although the two curricula did not target these directly. Interestingly (and in contrast to other findings) they found no interaction with household income; poor and wealthy children showed the same benefit. There were some interactions with ethnicity: children from Hispanic homes showed larger benefits than others on some measures.

There are questions that could be raised. The comparison children were those who had just missed the age cut-off to attend the preschool. So those children are, obviously, younger, and might be expected to show less development during those 9 months than older children. Another objection concerns what those control kids were doing during the year. The researchers did have data on this question, and reported that many were in setting that typically do not offer much opportunities for cognitive growth, e.g., center-based care (although the researchers argued that Massachusetts imposes stricter regulations for quality on such settings than most states do.)

Despite these caveats, this study represents the kind of thing Dave and I had in mind when we said the Department of Education should make communicating research findings to states a priority. Boston faced exactly the problem that many districts will face, they solved it using their own limited resources as districts will have to, and by all appearances, it’s been a success.

References:

Gormley, W. T., Gayer, T., Phillips, D., & Dawson, B. (2005). The effects of universal pre-K on cognitive development. Developmental Psychology, 41, 872–884.

Magnuson, K., Ruhm, C., & Waldfogel, J. (2007). Does prekindergarten improve school preparation and per-formance? Economics of Education Review, 26,33–51.

Weiland, C. & Yoshikawa, H. (2013). Impacts of a prekindergarten program on children’s mathematics, language, literacy, executive function, and emotional skills. Child Development, 84, 2112-2130.

The cover story of latest New Republic wonders whether American educators have fallen in blind love with self-control. Author Elizabeth Weil thinks we have. Titled “American Schools Are Failing Nonconformist Kids: In Defense of the Wild Child” the article suggests that educators harping on self-regulation are really trying to turn kids into submissive little robots. And they do so because little robots are easier to control in the classroom.

But lazy teachers are not the only cause. Education policy makers are also to blame, according to Weil. She writes that “valorizing self-regulation shifts the focus away from an impersonal, overtaxed, and underfunded school system and places the burden for overcoming those shortcomings on its students.”

And the consequence of educators’ selfishness? Weil tells stories that amount to Self-Regulation Gone Wild. A boy has trouble sitting cross-legged in class—the teacher opines he should be tested because something must be wrong with him. During story time the author’s daughter doesn’t like to sit still and to raise her hand when she wants to speak. The teacher suggests occupational therapy.

I can see why Weil and her husband were angry when their daughter’s teacher suggested occupational therapy simply because the child’s behavior was an inconvenience to him. But I don’t take that to mean that there is necessarily a widespread problem in the psyche of American teachers. I take that to mean that their daughter’s teacher was acting like a selfish bastard.

The problem with stories, of course, is that there are stories to support nearly anything. For every story a parent could tell about a teacher diagnosing typical behavior as a problem, a teacher could tell a story about a child who really could do with some therapeutic help, and whose parents were oblivious to that fact.

What about evidence beyond stories?

Weil cites a study by Duncan et al (2007) that analyzed six large data sets and found social-emotional skills were poor predictors of later success.

She also points out that creativity among American school kids dropped between 1984 and 2008 (as measured by the Torrance Test of Creative Thinking) and she notes “Not coincidentally, that decrease happened as schools were becoming obsessed with self-regulation.”

There is a problem here. Weil uses different terms interchangeably: self-regulation, grit, social-emotional skills. They are not same thing. Self-regulation (most simply put) is the ability to hold back an impulse when you think that that the impulse will not serve other interests. (The marshmallow study would fit here.) Grit refers to dedication to a long-term goal, one that might take years to achieve, like winning a spelling bee or learning to play the piano proficiently. Hence, you can have lots of self-regulation but not be very gritty. Social emotional skills might have self-regulation as a component, but it refers to a broader complex of skills in interacting with others.

These are not niggling academic distinctions. Weil is right that some research indicates a link between socioemotional skills and desirable outcomes, some doesn’t. But there is quite a lot of research showing associations between self-control and positive outcomes for kids including academic outcomes, getting along with peers, parents, and teachers, and the avoidance of bad teen outcomes (early unwanted pregnancy, problems with drugs and alcohol, et al.). I reviewed those studies here. There is another literature showing associations of grit with positive outcomes (e.g., Duckworth et al, 2007).

Of course, those positive outcomes may carry a cost. We may be getting better test scores (and fewer drug and alcohol problems) but losing kids’ personalities. Weil calls on the reader’s schema of a “wild child,” that is, an irrepressible imp who may sometimes be exasperating, but whose very lack of self-regulation is the source of her creativity and personality.

But irrepressibility and exuberance is not perfectly inversely correlated with self-regulation. The purpose of self-regulation is not to lose your exuberance. It’s to recognize that sometimes it’s not in your own best interests to be exuberant. It’s adorable when your six year old is at a family picnic and impulsively practices her pas de chat because she cannot resist the Call of the Dance. It’s less adorable when it happens in class when everyone else is trying to listen to a story.

So there’s a case to be made that American society is going too far in emphasizing self-regulation. But the way to make it is not to suggest that the natural consequence of this emphasis is the crushing of children’s spirits because self-regulation is the same thing as no exuberance. The way to make the case is to show us that we’re overdoing self-regulation. Kids feel burdened, anxious, worried about their behavior.

Weil doesn’t have data that would bear on this point. I don’t either. But my perspective definitely differs from hers. When I visit classrooms or wander the aisles of Target, I do not feel that American kids are over-burdened by self-regulation.

As for the decline in creativity from 1984 and 2008 being linked to an increased focus on self-regulation…I have to disagree with Weil’s suggestion that it’s not a coincidence (setting aside the adequacy of the creativity measure). I think it might very well be a coincidence. Note that scores on the mathematics portion of the long-term NAEP increased during the same period. Why not suggest that kids improvement in a rigid, formulaic understanding of math inhibited their creativity?

Can we talk about important education issues without hyperbole? 

References

Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: perseverance and passion for long-term goals. Journal of personality and social psychology, 92(6), 1087.

Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., ... & Japel, C. (2007). School readiness and later achievement. Developmental psychology, 43(6), 1428.

One of the great intellectual pleasures is to hear an idea that not only seems right, but that strikes you as so terribly obvious (now that you've heard it) you're in disbelief that no one has ever made the point before.

I tasted that pleasure this week, courtesy of a paper by Walter Boot and colleagues (2013).

The paper concerned the adequacy of control groups in intervention studies--interventions like (but not limited to) "brain games" meant to improve cognition, and the playing of video games, thought to improve certain aspects of perception and attention.

To appreciate the point made in this paper, consider what a control group is supposed to be and do. It is supposed to be a group of subjects as similar to the experimental group as possible, except for the critical variable under study.

The performance of the control group is to be compared to the performance of the experimental group, which should allow an assessment of the impact of the critical variable on the outcome measure.

Now consider video gaming or brain training. Subjects in an experiment might very well guess the suspected relationship between the critical variable and the outcome. They have an expectation as to what is likely to happen. If they do, then there might be a placebo effect--people perform better on the outcome test simply because they expect that the training will help just as some people feel less pain when given a placebo that they believe is a analgesic.

The standard way to deal with that problem is the use an "active control." That means that the control group doesn't do nothing--they do something, but it's something that the experimenter does not believe will affect the outcome variable. So in some experiments testing the impact of action video games on attention and perception, the active control plays slow-paced video games like Tetris or Sims.

The purpose of the active control is that it is supposed to make expectations equivalent in the two groups. Boot et al.'s simple and valid point is that it probably doesn't do that. People don't believe playing Sims will improve attention.

The experimenters gathered some data on this point. They had subjects watch a brief video demonstrating what an action video game was like or what the active control game was like. Then they showed them videos of the measures of attention and perception that are often used in these experiments. And they asked subjects "if you played the video game a lot, do you think it would influence how well you would do on those other tasks?"

And sure enough, people think that action video games will help on measures of attention and perception. Importantly, they don't think that they would have an impact on a measure like story recall. And subjects who saw the game Tetris were less likely to think it would help the perception measures, but were more likely to say it would help with mental rotation.

In other words, subjects see the underlying similarities between games and the outcome measures, and they figure that higher similarity between them means a greater likelihood of transfer.

As the authors note, this problem is not limited to the video gaming literature; the need for an active control that deals with subject expectations also applies to the brain training literature.

More broadly, it applies to studies of classroom interventions. Many of these studies don't use active controls at all. The control is business-as-usual.

In that case, I suspect you have double the problem. You not only have the placebo effect affecting students, you also have one set of teachers asked to do something new, and another set teaching as they typically do. It seems at least plausible that the former will be extra reflective on their practice--they would almost have to be--and that alone might lead to improved student performance.

It's hard to say how big these placebo effects might be, but this is something to watch for when you read research in the future.

Reference

Boot, W. R., Simons, D. J., Stothart, C. & Stutts, C. (2013). The pervasive problems with placebos in psychology: Why active control groups are not sufficient to rule out placebo effects. Perspectives in Psychological Science, 8, 445-454.

Which of these learning situations strikes you as the most natural, the most authentic?

1) A child learns to play a video game by exploring it on his own.
2) A child learns to play a video game by watching a more experienced player.
3) A child learns to play a video game by being taught by a more experienced player.

In my experience a lot people take the first of these scenarios to be the most natural type of learning—we explore on our own. The third scenario has its place, but direct instruction from someone is a bit contrived compared to our own experience.

I’ve never really agreed with this point of view, simply because I don’t much care about “naturalness” one way or the other. As long as learning is happening, I’m happy, and I think the value some people place on naturalness is a hangover from a bygone Romantic era, as I describe here.

Now a fascinating paper by Patrick Shafto and his colleagues (2012) (that’s actually on a rather different topic) leads to implications that call into doubt the idea that exploratory learning is especially natural or authentic.

The paper focuses on a rather profound problem in human learning. Think of the vast difference in knowledge between a new born and a three-year-old; language, properties of physical objects, norms of social relations, and so on. How could children learn so much, so rapidly? 

As you're doubtless aware, from the 1920's through the 1960's, children were viewed by psychologists as relatively passive learners of their environment. More recently, infants and toddlers have been likened to scientists; they don't just observe the environment, they reason about what they observe.

But it's not obvious that reasoning will get the learning done. For example, in language the information available for their observation seems ambiguous. If a child overhears an adult comment “huh, look at that dog,” how is the child to know whether “dog” refers to the dog, the paws of the dog, to running (that the dog happens to be doing), to any object moving from the left to the right, to any multi-colored object etc.?

Much of the research on this problem has focused on the idea that there must be innate assumptions or biases on the part of children that help them make sense of their observations. For example, children might assume that new words they hear are more likely to apply to nouns than to adjectives.

Many models using these principles have not attached much significance to the manner in which children encounter information. Information is information.

Shafto et al. point out why that's not true. They draw a distinction between three different cases with the following example. You’re in Paris, and want a good cup of coffee.

1) You walk into a cafe, order coffee, and hope for the best.
2) You see someone who you know lives in the neighborhood. You see her buying coffee at a particular cafe so you get yours there too.
3) You see someone you know lives in the neighborhood. You see her buying coffee at a particular cafe. She sees you observing her, looks at her cup, looks at you, and nods with a smile

In the first case you acquire information on your own. There is no guiding principle behind this information acquisition. It is random, and learning where to find good coffee will slow going with this method.

In the second scenario, we anticipate that the neighborhood denizen is more knowledgeable than we--she probably knows where to get good coffee. Finding good coffee ought to be much faster if we imitate someone more knowledgeable than we. At the same time, there could be other factors at work. For example, it's possible that she thinks the coffee in that cafe is terrible, but it's never crowded and she's in a rush that morning.

In the third scenario, that's highly unlikely. The woman is not only knowledgeable, she communicates with us; she knows what we want to know and she can tell us that the critical feature we care about is present. Unlike scenario #2,  the knowledgeable person is adjusting her actions to maximize our learning. 

More generally, Shafto et al suggest that these cases represent three fundamentally different learning opportunities; learning from physical evidence, learning from the observation of goal-directed action, and learning from communication.

Shafto et al argue that although some learning theories assume that children acquire information at random, that's likely false much of the time. Kids are surrounded by people more knowledgeable than they. They can see, so to speak, where more knowledgeable people get their coffee.

Further, adults and older peers often adjust their behavior to make it easier for children to draw the right conclusion. Language is notable in its ambiguity-“dog” might refer to the object, its properties, its actions—but more knowledgeable others often do take into account what the child knows, and speak so as to maximize what the child can learn. If an adult asked “what’s that?”  I might say “It’s Westphalian ham on brioche.” If a toddler asked, I ‘d say “It’s a sandwich.”

One implication is that the problem I described—how do kids learn so much, so fast—may not be quite as formidable as it first seemed because the environment is not random. It has a higher proportion of highly instructive information. (The real point of the Shafto et al. paper is to introduce a Bayesian framework for integrating these different three types of learning scenarios into models of learning.)

The second implication is this: when a more knowledgeable person not only provides information but tunes the communication to the knowledge of the learner, that is, in an important sense, teaching.

So whatever value you attach to “naturalness,” bear in mind that much of what children learn in their early years of life may not be the product of unaided exploration of their environment, but may instead be the consequence of teaching. Teaching might be considered a quite natural state of affairs.

EDIT: Thanks to Pat Shafto who pointed out a paper (Csibra & Gergely) that draws out some of the "naturalness" implications re: social communication. 

Reference
Shafto, P., Goodman, N. D. & Frank, M. C. (2012). Learning from others: The consequences of psychological reasoning for human learning. Perspectives in Psychological Science, 7, 341-351.

A teacher from the UK has just written to me asking for a bit of clarification (EDIT: the email came from Sue Cowley, who is actually a teacher trainer.)

She says that some people are taking my writing on the experiments that have tested predictions of learning styles theories (see here) as implying that teachers ought not to use these theories to inform their practice.

Her reading of what I've written on the subject differs: she thinks I'm suggesting that although the scientific backing for learning styles is absent, teachers may still find the idea useful in the classroom.

The larger issue--the relationship of basic science to practice--is complex enough that I thought it was worth writing a book about it. But I'll describe one important aspect of the problem here.

There are two methods by which one might use learning styles theories to inspire ones practice. The way that scientific evidence bears on these two methods is radically different.

Method 1: Scientific evidence on children's learning is consistent with how I teach.

Teachers inevitably have a theory--implicit or explicit--of how children learn. This theory influences choices teachers make in their practice. If you believe that science provides a good way to develop and update your theory of how children learn, then the harmony between this theory and your practice is one way that you build your own confidence that you're teaching effectively. (It is not, of course, the only source of evidence teachers would consider.)

It would seem, then, that because learning styles theories have no scientific support, we would conclude that practice meant to be consistent with learning styles theories will inevitably be bad practice.

It's not that simple, however. "Inevitably" is too strong. Scientific theory and practice are just not that tightly linked.

It's possible to have effective practices motivated by a theory that lacks scientific support. For example, certain acupuncture treatments were initially motivated by theories entailing chakras--energy fields for which scientific evidence is lacking. Still, some treatments motivated by the theory are known to be effective in pain management.

But happy accidents like acupuncture are going to be much rarer than cases in which the wrong theory leads to practices that are either a waste of time or are actively bad. As long as we're using time-worn medical examples, let's not forget the theory of four humors.

Bottom line for Method 1: learning styles theories are not accurate representations of how children learn. Although they are certainly not guaranteed to lead to bad practice, using them as a guide is more likely to degrade practice than improve it.

Method 2: Learning styles as inspiration for practice, not evidence to justify practice.

In talking with teachers, I think this second method is probably more common. Teachers treat learning styles theories not as sacred truth about how children learn, but as a way to prime the creativity pump, to think about new angles on lesson plans.

Scientific theory is not the only source of inspiration for classroom practice. Any theory (or more generally, anything) can be a source of inspiration.

What's crucial is that the inspirational source bears no evidential status for the practice.

In the case of learning styles a teacher using this method does not say to himself "And I'll do this because then I'm appealing to the learning styles of all my students," even if the this was an idea generated by learning styles. The evidence that this is a good idea comes from professional judgment, or because a respected colleague reported that she found it effective, or whatever.

Analogously, I may frequently think about Disneyland when planning lessons simply because I think Disneyland is cool and I believe I often get engaging, useful ideas of classroom activities when I think about Disneyland. Disneyland is useful to me, but it doesn't represent how kids learn.

Bottom line for Method 2: Learning styles theories might serve as an inspiration for practice, but it holds no special status as such; anything can inspire practice.

The danger, of course, lies in confusing these two methods. It would never occur to me that a Disneyland-inspired lesson is a good idea because Disneyland represents how kids think. But that slip-of-the-mind might happen with learning styles theories and indeed, it seems to with some regularity.