Am I stupid if I can't turn on my stove? The picture below (or one very similar) appears in most textbooks on human factors psychology.

The arrangement of controls is spatially incompatible with the arrangement of stove elements, so if I want to turn on the back left element, I may very well turn on the front left one.

What's notable is that this stove likely came with an instruction book, describing which knob goes with which burner.

But something about that feels wrong. It feels like the designer of the stove should have known how my mind works, and taken that into account, rather than shrugging and saying "well, it's in the manual. It's not my fault if you don't read the manual."

The stove reminds me of value-added measures of teacher effectiveness.

Even the staunchest boosters of value-added measures agree that they should not be the whole story, that there should be multiple measures of teacher effectiveness. But I'm afraid that asking people to remember that fact is a little like asking people to remember which knob goes with which burner on their stove. It's not that people can't do it, but you are swimming upstream of the mind's biases.

To be clear, I don't think that there are data to prove this contention, but let me describe why I'm guessing it's true.

We're talking about a case of missing information: you tell people: "Teacher Smith's value-added score is X. By the way, value-added scores are incomplete as a measure of teacher effectiveness"

How do people interpret information that they know to be incomplete? It varies with the situation. Sometimes they assume the missing information is positive. ("I haven't heard that the roads are closed, so I guess all's well.") Sometimes they assume missing information is negative ("He left 'prior experience' blank, so I guess he doesn't have any.")

And sometimes missing information is forgotten or discounted. My guess--and I emphasize that it's a guess--is that will be the case here. I make this guess in part by analogy to the evaluation of college applicants.

A student's high school record has lots of "soft" components, the values of which are tricky to evaluate: participation in sports and clubs, leadership positions, recommendations from teachers. . .. even a student's grade point average must be evaluated in light of the difficulty of the courses taken and the competitiveness of the high school.

But then there's the SAT. It has the gloss of being numeric, and it is easy to make comparisons across students. Make no mistake, I believe that the SAT does what it's supposed to do--predict success in the freshman year of college. But it's often interpreted to be much more meaningful than that. That's the problem.

I'm afraid that value-added measures will have the same problem. They are produced via a fancy formula, they make it simple to make comparisons, and they are numeric, which can lead one to conclude that they are more precise than they really are.  And at this point, we don't even have any of the other "soft" measures to round out the picture of teacher effectiveness.

I don't think value-added measures are meaningless. But handing people value-added measures with the bland warning "these are incomplete" is like giving me a stove with a bad mapping plus an instruction booklet.

The solution to the stove problem is straightforward

The solution to teacher evaluation is not straightforward, and I won't attempt to resolve it in a blog posting.

My purpose here is simply to highlight the problem in publishing value-added data for individual teachers, with the caveat "these measures are incomplete." I predict that caveat will go unnoticed or be forgotten.

18th century scientist G. C. Lichtenberg understood the importance of prior knowledge to reading: “A book is like a mirror: if an ass looks in, you can’t expect an apostle to look out.”

I came across this quotation in Alan Jacob's lively book The Pleasures of Reading in an Age of Distraction.

Lichtenberg understood the importance of prior knowledge to reading, a subject on which there is considerable scientific evidence. I've written about it before, and will again, I'm sure. (And if reading's not your thing, here's the video.)

Ulric “Dick” Neisser has passed away at age 83.

Neisser is sometimes called the father of cognitive psychology due to a book he published in 1967, titled Cognitive Psychology. The field was already well under way by that date, but Cognitive Psychology did much to make the theoretical foundations and the experimental framework explicit. That served both to define the field, and to help train new students. (It's less often mentioned that Neisser repudiated this framework in a 1976 books, Cognition and Reality, in which he adopted a more Gibsonian view of perception.)

Neisser was not just a theoretician, but a gifted experimentalist. Among other important findings, he conducted an experiment showing the people focusing on a complex video scene failed to notice a woman with an open umbrella traverse the scene, anticipating Simon & Chabris's now-famous gorilla video. In memory research, Neisser did important work in showing that “flashbulb” memories, although held with great confidence, are not terrible accurate.

Neisser spent most of his career at Cornell, and died in Ithaca.

I admit that until a few years ago, learning that a school asked their students to meditate prompted me to roll my eyes. It struck me as faddish and meant to appeal to parents rather than something meant to help students. But I’m not sneering anymore.

The last five or ten years has seen a burgeoning research literature on the cognitive benefits of mindfulness meditation—that style of meditation in which one focuses one’s thoughts on the present moment and emphasizes a open, non-judgmental attitude towards thoughts and sensations.

Most practitioners engage in mindfulness meditation for its effects on overall feelings of well-being. From a cognitive point of view, the daily practice in the management and control of attention might yield benefits for students. This sort of attentional control is positively associated with academic outcomes. (An article I wrote on the topic can be found here.)

What do the data on meditation and attentional control look like?

The truth is that it’s a bit early to tell. A recent review (Chiesa, Calati, & Serretti, 2011) concluded that meditation training did lead to improvements in controlled attention, but the authors warned that the effects were inconsistent.

The results might be inconsistent because the benefits to attention only accrue after significant practice--more practice then volunteers are willing to engage in for the sake of a study. But even when examining long-time meditators, the benefits to attention are inconsistent.

Another possibility is that meditation doesn’t make attentional control any more effective, but it does make it less taxing, which might be consistent with reports of improved well-being. There are electrophysiological data (e.g., Moore, Gruber, Derose & Malinowski, 2012) indicating that meditation training leads to changes in how the brain deals with attentional challenges, and that these changes reflect easier, smoother processing.

Most of the work has been done with adults, not kids.  There are at least a few studies that have used mindfulness mediation interventions with kids of middle-school age, and the authors of these studies claim these kids can learn the practice (e.g., Wall, 2005).

So at this point, the benefits of mindfulness mediation are not clear enough to make a claim that there is scientific backing for the practice in schools, if the hoped-for benefit is academic. But this is a research literature worth keeping on the radar.

Chiesa, A., Calati, R., & Serretti, A. (2011). Does mindfulness training improve cognitive abilities? A systematic review of neuropsychological findings. Clinical Psychology Review, 31, 449-464.

Moore, A., Gruber, T., Derose, J., & Malinowski, P. (2012). Regular, brief mindfulness meditation practice improves electrophysiological markers of attentional control. Frontiers in Human Neuroscience, 6, 18.

Wall, R. B. (2005). Tai chi and mindfulness-based stress reduction in a Boston Public middle school. Journal of Pediatric Health Care. 19,  230-237.

There’s a new article (Melby-Lervåg, Lyster, & Hulme, 2012) out in Psychological Bulletin (commonly considered the premier outlet for large literature reviews), which summarizes a great deal of data on phonological skills and reading.

The broad conclusions will startle no one who follows this research literature—we know phonological skills are important--but the article is notable for making a couple of finer-grain distinctions.

It is sensible that a child’s ability to understand that words are composed of sounds should be important in learning to read. After all, letters and groups of letters correspond to sound, they don’t signify meaning directly. If that weren’t true you wouldn’t be able to read nonsense words like “flotupe.” Letters signify sounds, and decoding means that kids must match letters (T) and groups of letters (TH) to sound. That means they must first appreciate that words are composed of sounds.

There has been some debate, however, as to the size of the sound unit that matters: the phoneme, or rimes and onsets. The onset refers to the consonant string that precedes a vowel sound, and the rime refers to the vowel and any consonants that follow. Hence, in the word TRIP, the onset would be /TR/ and the rime would be /IP/.

Other researchers have suggested that children must appreciate still smaller sound units—phonemes—as preparation for reading. Phonemes are individual speech sounds that cannot be further subdivided. For example, the rime /IP/ is composed of two phonemes: /I/ and /P/. Perhaps it helps if kids perceive that /IP/ is really two sounds. . .but perhaps that’s not necessary. (And indeed, perceiving that the letter P goes with the individual speech sound /P/ is no small feat, because /P/ is nearly impossible to say on its own. It’s really just a plosion of air, so it sound like you’re imitating a champagne cork popping. Parents typically add a vowel, usually saying “puh” for P.

Then again, maybe whether kids perceive onsets/rimes or phonemes is less important than their having a sizable verbal short term memory in which to manipulate and consider these speech sounds as they are learning to read.

Melby-Lervåg et al. included 235 studies in their analysis and concluded that existing research suggests that all three—rime awareness, phoneme awareness, and size of verbal short-term memory—predict word reading but the largest effect is observed with phonemic awareness.  (In fact, the predictive value of verbal short-term memory is quite small.)

The second important conclusion from this review concerns causality. All of these studies in the meta-analysis are correlational. Hence, one interpretation is that phonemic awareness is related to word reading skills because phonemic awareness is necessary for that skill. However, an equally viable interpretation is that reading changes phonemic awareness, so the association is observed because more skilled readers have undergone greater change.

Melby-Lervåg et al. separately consider longitudinal studies that measure phonemic awareness when children are still quite young and have little reading experience. Then reading ability (and especially, the rate at which this ability grows) of these same children is then measured later. Phonemic awareness remains an excellent predictor of reading skill in these studies with a mean correlation of .43; reading could not have caused phonemic awareness, because phonemic awareness was measured before kids could read. (Rime awareness was also a significant predictor in this sort of study, but not as strong, mean correlation = .29). Coupled with other data (not included in the meta-analysis) showing a positive effect for phonemic awareness training, the evidence for a causal role of phonemic awareness in learning to read is growing quite strong.

Melby-Lervåg, M., Lyster, S-A H., & Hulme, C. (2012). Phonological skills and their role in learning to read: A meta-analytic review. Psychological Bulletin, 138,  322-352.

An op-ed in today's New York times claims that the methods espoused by the "Tiger Mother" and by French parents (as reported in Pamela Druckerman's new book) are not necessary, and are a bad fit for Americans. The authors suggest that "rather than trying to emulate the strict discipline supposedly instilled by child-rearing techniques in other countries, it may be more useful to consider the science of successful parenting in general."

Uh, actually, the science of successful parenting shows that children who are high in self-control are more likely to come from homes with house rules. (Lengua, Honorado, & Bush, 2007; Schroeder & Kelley, 2010)

But let's set that aside for the moment. What do they allege that the science of successful parenting suggest? Apparently "psychologists find that children can-learn self control without externally imposed pressure." 

Well, in one sense that's surely true: kids ability to regulate their emotions, delay gratification and engage in other acts of self-control does improve as they get older, whether parents work on that skill or not. 

But merely waiting is not what the authors have in mind, of course. But there's not much reason to think that the strategies they suggest will be effective.

The strategies:
1) help the child find a hobby he or she is really excited about
2) encourage the child to engage in imaginative play
3) learn a second language (which does yield cognitive benefits, but not self-control of the sort alluded to in the rest of the article)
4) encourage aerobic exercise

It's hard to take these suggestions seriously because the descriptions are incomplete.  Imaginative play does improve self-control, but the way in which the play is orchestrated can't really be left to chance. The successful "Tools of the Mind" curriculum uses lots of imaginative play, but the drama is not left utterly up to kids. It requires a skillful teacher (and a set of ground rules as to how the drama is to be carried out) for the strategy to work. 

A hobby might help self control if the child is (as the authors say) passionate about it, and so learn that hard work is necessary for a desired payoff. But again, you're sort of leaving a lot to chance if you hope that your child will develop a hobby consonant with that, and will actually stick with it. (I'm reminded of the 13-year-old son of a friend, who calmly told his mother "Mom, don't you get it? Watching TV is my hobby. It's what I do.")

I'm not arguing for "strict parenting." I'm arguing for home factors that the science of parenting actually shows is associated with self-control in kids: parental warmth, and a predictable, organized home environment, 

I wrote an article summarizing self-control which you can find here.

Liliana J. Lengua, Elizabeth Honorado, and Nicole R. Bush, “Contextual Risk and Parenting as Predictors of Effortful Control and Social Competence in Preschool Children,” Journal of Applied Developmental Psychology 28, no. 1 (2007): 40–55.

Valarie Schroeder and Michelle L. Kelley, “Family Environment and Parent-Child Relationships as Related to Executive Functioning in Children,” Early Child Development and Care 180, no. 10 (2010): 1285–1298.

 

Title tells all.

Video here.

One strategy for thinking about interventions to boost kids success in school is to conduct the following sort of study. Step one:  measure lots of factors early in life, i.e., before kids start school. Step two: measure academic success after kids have been in school awhile (say, fourth grade). Then see which factors you measured early in life are associated with school success measured later.

Some factors are well-known, e.g., socio-economic status of the parents, and so you’d statistically remove those “usual suspects” first.

In 2007 Duncan and colleagues introduced a new method of analyzing this type of data, and they applied it to six sizable international data sets that followed kids from as early as birth to 3rd grade, focusing especially on reading and math achievement.  They concluded that early measures of math and reading, and measures of attention were significant predictors of later math and reading skills, but early social skills were not. Curiously, early math scores predicted later reading scores as well as early reading scores did.

Their conclusions, while not startling, attracted a lot of attention because the new method was deemed quite useful, and because it was applied meticulously to several large-scale datasets.

In 2010, another article was published using the same methodology, but with a startling result.

David Grissmer and his colleagues noted that three of the data sets had early measures of fine motor skills. They found that, after they statistically accounted for all of the factors that Duncan et al had examined, fine motor skills was and additional, strong predictor of student achievement.

I have to note that what the tests called “fine motor skills” strikes me as a bit odd.  Cognitive psychologists think of that as being tasks like buttoning a button, or picking something up with tweezers—i.e., requiring precise movements, usually of the fingers. But in these data sets it was tested with tasks like copying simple designs, or drawing a human figure. These are not solely motor tasks.

The fuzziness of exactly what the tasks mean may cloud the interpretation, but it doesn’t cloud the size of the effect—these tasks are a robust predictor of later math and reading achievement.

There’s plenty of speculation as to why this effect might work. Perhaps the measure of “fine motor skills” is really another way of measuring some aspect of attention. Perhaps it’s another way of measuring how well kids can understand and use space. Or the effect may be more direct; it’s commonly thought that the motor and cognitive domains are intertwined, and so practicing motor  tasks may aid cognition.

The big question: does this mean that practice of fine motor skills will boost academic achievement? Those studies are ongoing, and I hope to report on the results here before long.

Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C.,Klebanov, P., . . . Japel, C. (2007). School readiness and later achievement. Developmental Psychology, 43, 1428–1446.

Grissmer, D., Grimm, K., J., Aiyer, S. M., Murrah, W. M., & Steele, J. S. (2010). Fine Motor Skills and Attention: Primary Developmental Predictors of Later Achievement. Developmental Psychology, 46, 1008-1017.

An article in yesterday’s New York Times covered some recent research on the increasing education achievement gap between rich and poor. It’s worth a read, but it misses a couple of important points.      

     Regarding reasons for the gap, the article dwells on one hypothesis, commonly called the investment theory: richer families have more money to invest in their kids. (The article might have mentioned that richer families not only have more financial capital, but more human capital and social capital.) The article does not mention at all another major theory of the economics of educational achievement; stress theory. Kids (and parents) who live in poverty live under systemic stress. A great deal of research in the last ten years has shown that this stress has direct cognitive consequences for kids, and also affects how parents treat their kids. (Any parent knows that you’re not at your best when you’re stressed.) An open-access review article on this research can be found here.

      Another important point the article misses concerns what might be done. It ends with a gloomy quote from an expert: “No one has the slightest idea what will work. The cupboard is bare.”

     I think there is more reason for optimism, because other countries are doing a better job with this problem than we are. The OECD analyzes the PISA results by reported family SES. In virtually every country, high SES kids outperform low SES kids. But in some countries, the gap is smaller, and that’s it’s not just countries that have smaller income gaps.

      Economic inequality within a country is often measured with a statistic called the Gini coefficient which varies from 0 (everyone has the same net worth) to 1 (one person has all the money, and the other has nothing). Rich children score better than poor children in countries with large Gini coefficients (like the US) and the rich outscore the poor in countries with lower Gini coefficients (like Norway). Being poor predicts lower scores everywhere, but the disparity of wealth means more in the US than it does in other countries. What’s significant is that the relationship between income and test performance is stronger in the US than it is in most countries. (The US has the 3rd strongest relationship between income and student performance in Science and 10th highest for math, in the 2006 PISA results).

         Some countries, (e.g., Hong Kong), despite an enormous disparity between rich and poor, manage to even the playing field when the kids are at school. The US does a particularly poor job at this task; wealthy kids enjoy a huge advantage over poor kids. People generally argue that the US is different than Hong Kong, we’re a large, heteroogenous country, and so forth. All true, but the defeatist attitude won’t get us anywhere. We need more systematic study of how those countries solve the problem.

     It's a funny thing about the use of new technologies in schools: it's not only seen as inevitable, it's often described as necessary because today's students are digital natives.
     But for at least some technologies, the evidence supporting that contention is weak.
     An article in Computers & Education by Woody, Daniel, & Baker (2010) replicated other studies in showing that college students preferred studying from traditional textbooks rather than etextbooks, and also reported no correlation between previous experience with ebooks and how much students liked etextbooks.
     Some technology boosters have suggested that previous findings of student indifference to etextbooks is due to their novelty--once students get used to them, the argument goes, they will like them. Woody et al. suggest that their finding casts doubt on this explanation.

Woody, W. D., Daniel, D. B. & Baker, C. A. (2010). E-books or textbooks: Students prefer textbooks. Computers & Education, 55, 945-948.