A recent paper (Smith, Lewis, Hawthorne, & Hodges in press) examines whether such beliefs might account for sex differences in participation in STEM fields.
In Experiment 1, they examined how much effort graduate students in STEM fields perceived that they exerted, relative to their peers. The results showed that for women, perceived effort was inversely associated with sense of belonging. That is many women seemed to say to themselves "this is so hard for me, I must not really belong in graduate school." That perception was, in turn, associated with decreased motivation. These associations were not observed in men.
In Experiment 2, the researchers created a fictitious field (Eco-psychology) and distributed a professional-looking brochure for a graduate program in Eco-psychology to Introductory psychology students. The graduate program was subtly portrayed as either male-dominated or gender-neutral. Students were asked a number of questions about it, including how interested they were in the program and how difficult they thought they would find it, compared to "the average student." When the program was portrayed as male-dominated, women thought that they would find the program harder, and were less interested in learning more about it.
Experiment 3 used an elaborate ruse in which subjects believed they were interacting via webcam with a professor from the Eco-psychology program at University of Colorado, Boulder. The key manipulation was that the "professor" provided feedback about the subject's likely success in the program (which in this experiment was always portrayed as male-dominated). The feeling of alienation observed in Experiment 2 was observed again, but feedback from the professor could undo it; if the professor merely made effort seem normal but commenting that everyone in the program had to work hard, the gender effect disappeared.
This study mirrors some conceptually similar studies of college freshmen from historically underrepresented groups. For example, in Walton & Cohen (2011) students heard a simple message from upperclassmen emphasizing that everyone feels disoriented and concerned about whether they can really do the work when they first get to college, but that things get better. These brief messages not only made students feel better, they had an impact on students' grades. (There was no effect of the intervention on White students.)
In the larger picture, these findings should remind us of the powerful impact of beliefs on motivation.
References
Smith, J. L., Lewis, K. L., Hawthorne, L., & Hodges, S. D. (in press). When Trying Hard Isn’t Natural Women’s Belonging With and Motivation for Male-Dominated STEM Fields As a Function of Effort Expenditure Concerns. Personality and Social Psychology Bulletin.
Walton, G. M., & Cohen, G. L. (2011). A brief social-belonging intervention improves academic and health outcomes of minority students. Science, 331(6023), 1447-1451.