Daniel Willingham--Science & Education
Hypothesis non fingo
  • Home
  • About
  • Books
  • Articles
  • Op-eds
  • Videos
  • Learning Styles FAQ
  • Daniel Willingham: Science and Education Blog

Responding to a Study You Just KNOW Is Wrong

5/15/2020

 
Earlier this week I participated in a Zoom session organized and hosted by David Weston of the Teacher Development Trust. David asked me an interesting question, one I hadn’t considered before. He noted that learning styles theories have become a shibboleth for educators who are scientifically “in the know;” mention learning styles in any positive way and they will pounce on you with glee.

There are two problems with verbally thrashing people who say something you think lacks scientific support. First, it’s simply a bad tactic. You look like a bully and  snob; sure, the flush of self-righteousness is heady, but you’re not informing the other person, although you're pretending to. You’re not even really talking to them, you’re just enjoying yourself.

The second problem caused by the uncritical dismissal of disproven theories is that you might miss new developments. I’ve written about learning styles theories many times and my view of these theories has changed, (see here versus here) notably from excellent experimental work by David Kraemer, Josh Cuevas, and others. If you hear “learning styles” and immediately shout “Nonsense! Ridiculous!” without noticing that someone is providing new data in a well-conducted study, you miss things.

Which brings me to a study published by my colleague at UVa, Vikram Jaswal, about which I tweeted a earlier this week. 
Picture
The experiment describes an eye-tracking study in autistics who appear to be typing responses to questions on a letterboard. This sort of communication from people who cannot speak has a long and terrible history. In a previous incarnation called facilitated communication, an assistant held/supported the arm of the respondent (or in some other way touched them) as he or she typed. Research made it clear that the communication was actually coming from the assistant, although the assistant might not be aware of this influence.

The story of facilitated communication has become a staple in psychology methods courses, along with Clever Hans, and it also generated interest in a particular type of unconscious social influence and the methods by which we consciously interpret the reasons for our own behavior. (Which in turn provided Dan Wegner, my colleague at UVa that the time, and a lover of puns, the chance to publish a paper titled Clever Hands.)

The more recent version of this technique, employed in Jaswal's study, has the assistant holding the letterboard, but not touching the person who’s typing. This removes one way the assistant might be authoring the communication, but not all; the assistant might subtly indicate which letter is to be selected.

The study I tweeted about examined eye movements and pointing in a small (N=9) group of autistics who regularly use this method. The predictions are straightforward; if you think people are responding to cues, that’s a 26-choice response time task, and RTs in that sort of task would be slow, and people would make a lot of errors. You’d also predict that they would look at the assistant fairly often. If, in contrast, you think that participants are actually typing themselves, you’d expect fluent typing, you’d expect eye movements to lead finger movements without checking what the assistant was doing, and perhaps most interesting, you’d expect that response times would be slow at the start of a new word (for a multiword response) or at the beginning of the second morpheme in a compound word (like “scarecrow”). This  effect us observed in touch typists, and are due to motor planning processes.

The authors claimed that the data were consistent with the interpretation that those typing were agentic—at least part of the behavior was self-generated, rather than being fully determined by external cues. It’s the first such demonstration, which helps explain why it was published in a prestigious journal.

The response to my Tweet was a series of Tweets that were highly critical of…lots of things, and were reminiscent of the sort of thing David Weston asked me about regarding learning styles. I’ll provide just a very small sample of them.
​
These two tweets suggested that the lead author (Jaswal) had an association with some people who do terrible things. 
Picture
Picture
Jaswal doesn't run "The Tribe," btw, and never has. He taught an undergrad class which collaborated with The Tribe, and that fact is described in the "Competing Interests"  section of the paper. 

This person wanted the authors to conduct a different experiment. 
Picture
There was also a good deal of discussion about the fact that facilitated communication has been discredited…which the introduction of the paper points out, and which draws the distinction between the method they use and facilitated communication.

All of these tweets have one thing in common: they don’t address the study. You can debate whether the researchers should have run a different study, and just how terrible some people who use this method are or aren’t, but there the data sit, waiting to be explained.

​Some critics did try to address issues with the data.

These two watched the video provided by the authors as a supplementary material and reckoned they could code them more accurately than the coders, who analyzed pointing and eye movements frame by frame. Coders analyzed more than 142,000 frames. 
Picture
Picture
As near as I can tell, these tweeters are unaware of criteria for deciding what counts as the target of an eye movement versus gaze travelling through a location on its way to a target (dwell time > 98 ms).

This next bit would be an important criticism, except this person too appears not to have read the methods section, which describes how it was determined which letter the speller pointed to. 
Picture
​This tweet came closer to mark by calling attention to the sample size.
Picture
But small sample sizes are routine in certain types of work, especially neuropsychology. If you can’t get many people with the desired characteristics, you test who you can test. You recover statistical power by administering many trials, which lets you conduct statistical analyses person by person. This is a routine method in psychophysics. But the small N is a valid concern in that it calls into question generalizability. If we take the conclusions at face value, should we find them interesting if they might apply to just a tiny fraction of autistics?

I've picked out tweets that addressed methods, but for the most part, people didn’t engage with the actual study to discredit it. They attacked the experimenter and his associates, they broadly said it’s obvious this can’t work, they said the method has been discredited before.

If they really wanted to shoot for the soft spot of this study, they should have gone after things like the calculation of the simulation of the percentage of points to correct letters preceded by a fixation of that letter if fixations had been random—that was used as a baseline for the analysis that supported a key conclusion of the study, and the method the authors used is probably open to debate.

Now, what do I really think of this study? Of course this study should be replicated, and ideally in a different lab. That’s always the case; researchers may have made a mistake, equipment have been flukey, who knows. Equally obviously, the data are consistent with agency, nothing more—it’s not a test of whether a therapeutic technique or a communication method work. It’s also not a test of whether there was absolutely no influence by the assistant holding the board. The authors note that there certainly was…she sometimes finished words or interrupted, sort of the way speakers do. Their claim was that the data can’t be explained by influence alone.

Back to my conversation with David Weston about learning styles. When we feel sure we know something, disconfirming data pose a problem. You have three choices. The first of the three is the worst, and it’s mostly what we saw here; you castigate the study as terrible and obviously stupid but don’t provide any substantive evidence regarding problems with the method, analysis, or interpretation.  The second is to engage with the substance of the research and critique it. But that takes a lot of time and expertise. We saw some attempts at such criticism here, and we saw transparent failure to actually read the study, as well as lack of expertise. Which brings me to the third response. You have a feeling the conclusion is probably wrong because it conflicts with a whole lot of other theory and data. But you don’t know the particulars of what’s wrong with this experiment. So you ignore it.

Lest you think I'm suggesting that people just shut up, I'll tell you that I respond in this third way all the time. I see a study that I think doesn't square with a lot of other theory and data and I think "that's probably wrong." And I ignore it. If someone replicates it or if the study becomes  a big deal, I'll get worked up, but not before. 

It's not bad poker, folks. A hell of a lot of studies don't replicate, as we all know.

I'm guessing the critics of Jaswal's study would say they can't ignore this study because the stakes are so high. It's my perception that there's no little indignation in many of these tweets. As in every education debate I've seen, each side feels that they are motivated by what's best for students whereas the other side is motivated by greed and evil, filtered through stupidity and stubbornness. 

Which brings me back to the start of this blog. I suggested that pleasure lies behind the righteous indignation people apply to the learning styles issue. Emotion was at play here. It wasn’t positive emotion in this case, but the outcome was the same. Nobody learned anything, and nobody was convinced. 
Douglas Hainline link
5/15/2020 10:44:35 am

I couldn't, by any chance, persuade you to run for President, could I?

Although I recall reading what Adlai Steveson replied, after one of his campaign speeches, when a supporter came up to him and gushed, "Oh, Governor Stevenson, I'm sure every thinking American will vote for you!" and he supposedly said, "Yes ma'am, but I need a majority."

seanon jones
5/15/2020 12:10:05 pm

Thank you, Daniel Willingham. I completely agree that the reactive responses add little to the conversation.

Harriett Janetos
5/15/2020 04:46:16 pm

"There are two problems with verbally thrashing people who say something you think lacks scientific support. First, it’s simply a bad tactic. You look like a bully and snob; sure, the flush of self-righteousness is heady, but you’re not informing the other person, although you're pretending to. You’re not even really talking to them, you’re just enjoying yourself."

Thanks for saving me a lot of money on psychoanalysis! This is me--I'll work on it. Recently, Robert Pondiscio referred to himself as a "bore and a scold", and that also resonated.

Know thyself!

Harriett Janetos
5/15/2020 04:57:53 pm

And thanks, too, for the links to the learning styles articles.

Dale Webster link
5/15/2020 06:59:24 pm

Much appreciated Dr. Willingham! The types of responses you exemplified are what we often see in social media on a variety of topics. We are a society that lacks critical reading skills or won't take the time to critically read but feel compelled to then share that opinion, not thinking for one second we might be wrong or misrepresenting the information. I wish we would do better to support dialogue among friends and colleagues and be respectful in our approach rather than attacking them. As you point out replication is key. Let the data speak for itself.

Gail Brown link
5/16/2020 01:21:12 am

For your interest, and for other readers, The Reading League ran a online session this week about Facilitated Communication by Dr. Katharine Beals at Drexel University, which both debunked some research, presented their research that seemed a lot more credible? She is included in your post above... This may be available if you contact The Reading League for those who are interested. I was shocked to see that an organisation like the UN supported a strategy that didn't have an evidence-base. Katharine also mentioned Clever Hans, and the confounds - I was impressed by her knowledge and research! Hope this is helpful?

Daniel Willingham
5/16/2020 08:47:51 am

Hi Gail
I haven't seen the online session you mention...I'm guessing it probably didn't address the study I cite, because it was just published this past week. Again, it doesn't overturn everything we thought we knew! But it can't be easily dismissed either.

Gail Brown link
5/16/2020 03:40:20 pm

Agreed Dan - The Reading League is a local US group - formed from SSSR (Scientific Studies of Reading) - so they are evidence-based. That’s why I watch and learn from them - even though I can’t attend from Australia...

Katharine Beals link
5/17/2020 02:35:10 pm

Hi Gail,
Thanks for your kind words about my Reading League talk. It is still available through this link:
https://thereadingleague.uscreen.io/programs/le-may-beals-cut-1mp4-de3af4
As you probably know, there will be a live q and a this Monday, and I'm hoping people will ask questions relating to the Nature article. A number of the things I cover in my talk, as you may have noticed, are applicable, though the article did come out until a few days after I recorded my talk.
Meanwhile, I've posted a reply to Dan on my blog and on twitter.

Robert Fasso
5/18/2020 06:22:26 pm

The points you make about how we handle ourselves during disagreements are invaluable, Dan. Thinking about how we conduct ourselves and come across in these situations is vital.

But I think I missed something. In the first paragraph you mention that that one of the participants asked you a question, but what follows is a statement about learning styles becoming a shibboleth for some educators. I don't see a question. Is the question: Why do people act that way?

Daniel Willingham
5/19/2020 05:38:44 am

Robert, yes, it was something like that...tbh I can't remember the exact phrasing...

Robert Guttentag
5/20/2020 05:11:46 pm

Does Vyse's critique meet your criteria for a reasoned analysis.

https://skepticalinquirer.org/exclusive/of-eye-movements-and-autism-the-latest-chapter-in-a-continuing-controversy/

Let me also add that when someone uses a more complex approach to address an issue rather than a simpler and more obvious approach (e.g., using the kids of measures found in Jaswal's study as opposed to a simple message passing paradigm), and when that researcher chooses also chooses not to include very simple and obvious controls (e.g. literally blinding the person holding the board or putting the board on a stand), that may not absolutely prove that he conclusions are wrong, but it sure raises MAJOR red flags about whether the conclusions are warranted.

Nathanael
5/21/2020 07:20:09 am

It doesn't meet mine. He fails to make any plausible criticisms of the actual claim of the study.

This is because he's actually arguing about something different. He's worried that RPM is a bad, unreliable, or unsafe communication method. Maybe it is; he makes an argument for that.

But literally nothing he says seriously disputes the proof that the autistic people in the study were communicating their own thoughts with agency.

____

Of course, I have my own reaction because I'm working from outside evidence. There are dozens of autistic people who were non-speaking and had severe motor control problems who have since developed the ability to write and obviously have agency.

So the *default assumption* has to be that other autistic people are like Ido Kedar. It requires extraordinary evidence to assume that other autistic people have cognitive deficits rather than motor deficits -- you'd need *additional evidence* and there isn't any.

Therefore the "critique" of Vyse doesn't meet the extremely high burden of proof which anyone disputing the study's conclusion has. The burden is, and has been for years, actually on anyone who claims that the non-communicative autistic people have cognitive deficits rather than motor/communication deficits.

His advice to redo the study with even better "blinding", fine, that's always good advice.

Nathanael
5/21/2020 08:21:12 am

To add to the reasons why the burden of proof is on those claiming that the result of this study is wrong, take a look at this survey study from 2013:

https://emmashopebook.com/

You have to assume that every single survey submitter was committing intentional fraud AND that Emma Zurcher-Long doesn't actually exist, to disbelieve the underlying result here. It's possible but extremely unlikely.

In fact, based on Zucher-Long's extremely strong evidence, RPM actually works. Oh, and as for study strength? Her N=70, *after* throwing out the people who can communicate reliably through words. It's not a random sample but neither are any of the other autism studies.

So with Zucher-Long's 2017 survey study and the individual examples of her and Kedar and a dozen other *published authors* -- the burden of proof by someone claiming autistic people have cognitive deficits rather than being trapped by difficulty communication (as everyone in Zucher-Long's study said they were) is EXTREMELY HEAVY.

I have to conclude people who want to assume that autistic people have cognitive deficits are just bigots, there's nothing more to it than that.

Nathanael
5/21/2020 08:25:33 am

Also, you want to use the simplest and most obvious method for a study to see whether autistic people who can't communicate are typically intelligent? You should try asking autistic people.

Specifically all of the ones who were previously unable to communicate but now can. Like Zucher-Long does.

The simplest study method. What may be surprising is that it's having overwhelmingly consistent results (everyone else's surveys of actually autistic people are giving the same statements). Read it.

Nathanael
5/21/2020 08:28:16 am

Apologies for inaccuracy, Zucher-Long's research paper is from 2017. In case she updates the blog further this is the permalink:

https://emmashopebook.com/2017/12/09/methods-of-communication-my-research-paper/

Daniel Willingham
5/21/2020 05:47:34 am

hi robert, wow, that's really long...does he offer either an alternative account of the eyemovement data or a reason to think those data are irrelevant?

Robert E Guttentag
5/21/2020 02:18:32 pm

Yes. I can only assume that the other responder to my post (Nathanael) did not read Vyse's article.

Daniel Willingham
5/21/2020 02:19:43 pm

ah...i was kind of hoping you would tell me what it is

Nathanael
5/22/2020 02:48:33 am

Read it end to end. It's junk, though well-intentioned junk.

I'm actually digging into this deep now and I'm starting to get really really angry. There's a long history of actual fraud by the anti-FC people and FC has been proven beyond the shadow of a doubt to be genuine. Start reading.

https://dsq-sds.org/article/view/1729/1777

So apparently a "staple of psychology courses" is a pure fraud conducted by bigots. Nice, real nice. There are no standards in this field.

Please pay very close attention to this point. There are multiple people who are now typing completely unfacilitated who have testified, *unfacilitated*, that the facilitator was just preventing their arms from spasming and that they were writing everything they wrote using FC.

Further, I found reference for this too, people using FC can use a bunch of different facilitiators (and often try to have many) and their writing style is still the same.

I don't think there is any doubt now. The anti-FC stuff was an *actual fraud*. Nice work being gullible and bigoted, pysch departments.

Nathanael
5/22/2020 02:51:35 am

When I first started reading this I assumed that the claim that FC had been proven to be the facilitator speaking was actually true. It turns out it is not only false but comprehensively disproven *and* the claim was made by people who were blatantly fraudulent, who suppressed evidence they had which disproved their thesis (namely the people typing unfacilitated who stated that they were wrong).

That's sick. That is so sick. And even people like the respectable Dr. Willingham fell for it.

Kirsten Duncombe
5/22/2020 05:10:24 pm

Vyse's article almost exclusively focuses on the methodology of Jaswal's study, and outlines a clear and compelling argument that the 'causative variable' responsible for the impressive results cannot be reasonably determined.

This is because the study "did not test cause and effect... the assistant was very active throughout the process of gathering participant responses, creating the usual suspicions about the influence of the assistant". Somewhat legitimate suspicions it would appear, in light of the troubled and troubling history of Facilitated Communication techniques.

Vyse writes at length about the inability (due to built-in limitations of the design study) to disentangle participant agency from alternative factors.

Jaswal et al make clear at the outset of their paper their desire to respond to what they regard as the 'blanket dismissal of autistic assisted communication', which they see as stemming from 'controversy' regarding the assistance of another person, in particular "someone who holds the letterboard in front of the users".

The holding of the letterboard by a non-blindfolded assistant was one of several potentially confounding variables (all outlined in detail in Vyse's article) that the study's authors inexplicably retained.

You may argue that I'm 'asking for a different study', but note again that it was the authors themselves who sought to dismantle the objections of "some scientists" who "have dismissed the possibility that any nonspeaking autistic person who communicates with assistance could be conveying their own thoughts", through the mechanism of a purpose-built study,

Robert Guttentag
5/28/2020 10:18:30 am

You can find two summaries below (one that I wrote, and another by Kirsten Duncombe).

Robert Guttentag
5/23/2020 03:03:08 pm

Dan Willingham asked for a summary of Vyse's arguments.

Some of Vyse’s arguments (including quotes from the article itself) related to the general criticism that “The glaring alternative explanation for the origin of the words tapped out is created by the involvement of the assistant.”

Of course, rather that reading/accepting my summary, it would, of course, be better to read the article itself.

1. The study failed to use a direct approach to assessing agency (and in adopting their approach, they failed to incorporate easily utilized controls).

I understand that you (DW) don’t consider this a reasonable criticism, although it baffles me why you don’t.

“Questions of agency are questions of cause. An agent causes things to happen, and for hundreds of years, scientists have known that the best way to demonstrate a cause and effect relationship: identify the causal variable by systematically eliminating other possible causes. Scientists use control groups, placebos, or other experimental controls to rule out potential alternative hypotheses. This is also known as manipulating the independent variable.

The authors of this study chose not to do that. Instead they engaged in a highly detailed analysis of the dependent variable—the autistic participants’ eye movements and pointing—and concluded that, given the shape of their behavior, the participants must have been the agents (authors) of the words written. This kind of argument is reminiscent of supporters of intelligent design who claim that, due to the great intricacy of the human eye, human eyes could not be the product of evolution and must have been the work of an intelligent designer. In science, we don’t merely observe a phenomenon, throw up our hands, and say, “Well, it must have been caused by X.” We test for X and all the other possible causes.”

and

“Given that the assistant is not touching the participant, the supporters of RPM have never given an adequate explanation for why the board cannot simply be placed on a table or mounted on an easel and must instead be held—often being moved around in the air—by another person. In addition, why can’t the letterboard be replaced by a touch-sensitive tablet, eliminating the need for the assistant to call out the letters as the participant points to them on the board? “

2. The boards would bend (claimed to be the source of coding of which letter was touched) regardless of agency

“Holding a letterboard in the air creates a problem for the claim that the participant is doing the typing. Jaswal et al. said their coders could recognize a tap to the board because it bent under the pressure of the participant’s finger. I have watched the videos, and in several cases, I cannot tell whether the finger was meeting the letterboard or the board was hitting the finger. Both actions would produce a similar bend of the board, but the agency would be quite different.”

3. The longer pauses between words (taken as evidence supporting the claim that the participant, rather than the assistant, was the agent of each message) may be the result of the assistant saying each word aloud after the spelling of each word was completed.

“As to the pointing and eye movement data, the authors suggest that longer pauses between words are typical of non-autistic typists, but what is not typical of non-autistic typists is to have someone else speak each word as you type it. If one does not watch the videos, this alternative explanation might not come to mind, but it is quite possible the longer pause is simply produced by the participant stopping momentarily to listen to the assistant before starting the next word. The assistant has informed the participant by her actions that a word has been completed, and the participant pauses briefly before starting another. As a result, the pause between words is an artifact of the RPM procedure. There could be other explanations as well, but this hypothesis seems obvious.”

4. The quicker transitions between letters that form common letter sequences (takes as evidence supporting the claim that participant, rather than the assistant, was the agent of each message) may be the result of several years of rewarded practice touching common letter patterns.

“As to the quicker movement to common letter pairs versus less common letter pairs, this seems expected as well. If, after an average of over three years of practice, the participant has learned sequences of letters that will result in the assistant saying a word and moving the lesson along, then it is quite plausible that the participant has built up a store of such responses. Assuming the words taught appear in relation to their natural frequency, then one might assume that the N-to-T movement would happen more quickly than T-to-N, because the N-to-T sequence is more practiced. But even if we gra


Comments are closed.

    Enter your email address:

    Delivered by FeedBurner

    RSS Feed


    Purpose

    The goal of this blog is to provide pointers to scientific findings that are applicable to education that I think ought to receive more attention.

    Archives

    April 2022
    July 2020
    May 2020
    March 2020
    February 2020
    December 2019
    October 2019
    April 2019
    March 2019
    January 2019
    October 2018
    September 2018
    August 2018
    June 2018
    March 2018
    February 2018
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    February 2017
    November 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    December 2015
    July 2015
    April 2015
    March 2015
    January 2015
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012

    Categories

    All
    21st Century Skills
    Academic Achievement
    Academic Achievement
    Achievement Gap
    Adhd
    Aera
    Animal Subjects
    Attention
    Book Review
    Charter Schools
    Child Development
    Classroom Time
    College
    Consciousness
    Curriculum
    Data Trustworthiness
    Education Schools
    Emotion
    Equality
    Exercise
    Expertise
    Forfun
    Gaming
    Gender
    Grades
    Higher Ed
    Homework
    Instructional Materials
    Intelligence
    International Comparisons
    Interventions
    Low Achievement
    Math
    Memory
    Meta Analysis
    Meta-analysis
    Metacognition
    Morality
    Motor Skill
    Multitasking
    Music
    Neuroscience
    Obituaries
    Parents
    Perception
    Phonological Awareness
    Plagiarism
    Politics
    Poverty
    Preschool
    Principals
    Prior Knowledge
    Problem-solving
    Reading
    Research
    Science
    Self-concept
    Self Control
    Self-control
    Sleep
    Socioeconomic Status
    Spatial Skills
    Standardized Tests
    Stereotypes
    Stress
    Teacher Evaluation
    Teaching
    Technology
    Value-added
    Vocabulary
    Working Memory