What aspects of background, personality, or achievement predict success in college--at least, "success" as measured by GPA? A recent meta-analysis (Richardson, Abraham, & Bond, 2012
) gathered articles published between 1997 and 2010, the products of 241 data sets. These articles had investigated these categories of predictors:
- three demographic factors (age, sex, socio-economic status)
- five traditional measures of cognitive ability or prior academic achievement (intelligence measures, high school GPA, SAT or ACT, A level points)
- No fewer than forty-two non-intellectual measures of personality, motivation, or the like, summarized into the categories shown in the figure below (click for larger image).
Make this fun. Try to predict which of the factors correlate with college GPA.
Let's start with simple correlations.
41 out of the 50 variables examined showed statistically significant correlations. But statistical significance is a product of the magnitude of the effect AND the size of the sample--and the samples are so big that relatively puny effects end up being statistically significant. So in what follows I'll mention correlations of .20 or greater.
Among the demographic factors, none of the three were strong predictors. It seems odd that socio-economic status would not be important, but bear in mind that we are talking about college students, so this is a pretty select group, and SES likely played a significant role in that selection. Most low-income kids didn't make it, and those who did likely have a lot of other strengths.
The best class of predictors (by far) are the traditional correlates, all of which correlate at least r = .20 (intelligence measures) up to r = .40 (high school GPA; ACT scores were also correlated r = .40).
Personality traits were mostly a bust, with the exception of consientiousness (r = .19), need for cognition (r = .19), and tendency to procrastinate (r = -.22). (Procrastination has a pretty tight inverse relationship to conscientiousness, so it strikes me as a little odd to include it.)
Motivation measures were also mostly a bust but there were strong correlations with academic self-efficacy (r = .31) and performance self-efficacy (r = .59). You should note, however, that the former is pretty much like asking students "are you good at school?" and the latter is like asking "what kind of grades do you usually get?" Somewhat more interesting is "grade goal" (r = .35) which measures whether the student is in the habit of setting a specific goal for test scores and course grades, based on prior feedback.
Self-regulatory learning strategies likewise showed only a few factors that provided reliable predictors, including time/study management (r = .22) and effort regulation (r = .32), a measure of persistence in the face of academic challenges.
Not much happened in the Approach to learning category nor in psychosocial contextual influences.
We would, of course, expect that many of these variables would themselves be correlated, and that's the case, as shown in this matrix.
So the really interesting analyses are regressions that try to sort out which matter more.
The researchers first conducted five hierarchical linear regressions, in each case beginning with SAT/ACT, then adding high school GPA, and then investigating whether each of the five non-intellective predictors would add some predictive power. The variables were conscientiousness, effort regulation, test anxiety, academic self efficacy, and grade goal, and each did, indeed, add power in predicting college GPA after "the usual suspects" (SAT or ACT, and high school GPA) were included.
But what happens when you include all the non-intellective factors in the model?
The order in which they are entered matters, of course, and the researchers offer a reasonable rationale for their choice; they start with the most global characteristic (conscientiousness) and work towards the more proximal contributors to grades (effort regulation, then test anxiety, then academic self-efficacy, then grade goal).
As they ran the model, SAT and high school GPA continued to be important predictors. So were effort regulation and grade goal.
You can usually quibble about the order in which variables were entered and the rationale for that ordering, and that's the case here. As they put the data together, the most important predictors of college grade point average are: your grades in high school, your score on the SAT or ACT, the extent to which you plan for and target specific grades, and your ability to persist in challenging academic situations.
There is not much support here for the idea that demographic or psychosocial contextual variables matter much. Broad personality traits, most motivation factors, and learning strategies matter less than I would have guessed.
No single analysis of this sort will be definitive. But aside from that caveat, it's important to note that most admissions officers would not want to use this study as a one-to-one guide for admissions decisions. Colleges are motivated to admit students who can do the work, certainly. But beyond that they have goals for the student body on other dimensions: diversity of skill in non-academic pursuits, or creativity, for example.
When I was a graduate student at Harvard, an admissions officer mentioned in passing that, if Harvard wanted to, the college could fill the freshman class with students who had perfect scores on the SAT. Every single freshman-- 800, 800. But that, he said, was not the sort of freshman class Harvard wanted.
I nodded as though I knew exactly what he meant. I wish I had pressed him for more information.
Richardson, M., Abraham, C., Bond, R. (2012). Psychological correlates of university students' academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138, 353-387.
What people learn (or don't) from games is such a vibrant research area we can expect fairly frequent literature reviews. It's been about a year since the last one
, so I guess we're due. The last time I blogged on this topic Cedar Riener remarked that it's sort of silly to frame the question as "does gaming work?" It depends on the game. The category is so broad it can include a huge variety of experiences for students.
If there were NO games from which kids seemed to learn anything, you probably ought not to conclude "kids can't learn from games." To do so would be to conclude that distribution of learning for all possible games and all possible teaching would look something like this.
But this pattern of data seems highly unlikely. It seems much more probable that the distributions overlap more, and that whether kids learn more from gaming or traditional teaching is a function of the qualities of each.
So what's the point of a meta-analysis that poses the question "do kids learn more from gaming or traditional teaching?
I think of these reviews not as letting us know whether kids can learn from games, but as an overview of where we are--just how effective are the serious games offered to students?
The latest meta-analysis (Wouters et al, 2013
) includes data from 56 studies and examined both learning outcomes (77 effect sizes), retention (17 effect sizes) and motivation (31 effect sizes).
The headline results featured in the abstract is "games work!" Games are reported to be superior to conventional instruction in terms of learning (d
= 0.29) and retention (d
= .36) but somewhat surprisingly, not motivation (d
The authors examined a large set of moderator variables and this is where things get interesting. Here are a few of these findings:
- Students learn more when playing games in groups than playing alone.
- Peer-reviewed studies showed larger effects than others. (This analysis is meant to address the bias not to publish null results. . . but the interpretation in this case was clouded by small N's.)
- Age of student had no impact.
But two of the most interesting moderators significantly modify the big conclusions.
First, gaming showed no advantage over conventional instruction when the experiment used random assignment. When non-random assignment was used, gaming showed a robust advantage. So it's possible (or even likely) that games in these studies were more effective only when they interacted with some factor in the gamer that is self-selected (or selected by the experimenter or teacher). And we don't know yet what that factor is.
Second the researchers say that gaming showed and advantage over "conventional instruction" but followup analyses show that gaming showed no advantage over what they called "passive instruction"--that it, the teacher talk or reading a textbook. All of the advantage accrued when games were compared to "active instruction," described as "methods that explicitly prompt learners to learning activities (e.g., exercises, hypertext training.)" So gaming (in this data set) is not really better than conventional instruction; it's better than one type of instruction (which in the US is probably less often encountered.)
So yeah, I think the question in this review is ill-posed. What we really want to know is how do we structure better games? That requires much more fine-grained experiments on the gaming experience, not blunt variables. This will be painstaking work.
Still, you've got to start somewhere and this article offers a useful snapshot of where we are. EDIT 5:00 a.m. EST
2/11/13. In the original post I failed to make explicit another important conclusion--there may be caveats on when and how the games examined are superior to conventional instruction, but they were almost never worse. This is not an unreasonable bar, and as a group the games tested pass it.
Wouters, P, van Nimwegen, C, van Oostendorp, H., & van der Spek, E. G. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology.
Advance online publication. doi: 10.1037/a0031311
If the title of this blog struck you as brash, I came by it honestly: it's the title of a terrific new paper by three NYU researchers (Protzko, Aronson & Blair, 2013
). The authors sought to review all interventions meant to boost intelligence, and they cast a wide net, seeking any intervention for typically-developing children from birth to kindergarten age that used a standard IQ test as the outcome measure, and that was evaluated in a random control trial (RCT) experiment.A feature of the paper I especially like is that
none of the authors publish in the exact areas they review. Blair mostly studies self-regulation, and Aronson, gaps due to race, ethnicity or gender. (Protzko is a graduate student studying with Aronson.) So the paper is written by people with a lot of expertise, but who don't begin their review with a position they are trying to defend. They don't much care which way the data come out. So what did they find? The paper is well worth reading in its entirety--they review a lot in just 15 pages--but there are four marquee findings.
First, the authors conclude that infant formula supplemented with long chain polyunsaturated fatty acids boosts intelligence by about 3.5 points, compared to formula without. They conclude that the same boost is observed if pregnant mothers receive the supplement. There are not sufficient data to conclude that other supplements--riboflavin, thiamine, niacin, zinc, and B-complex vitamins--have much impact, although the authors suggest (with extreme caution) that B-complex vitamins may prove helpful.
Second, interactive reading with a child raises IQ by about 6 points. The interactive aspect is key; interventions that simply encouraged reading or provided books had little impact. Effective interventions provided information about how to read to children: asking open-ended questions, answering questions children posed, following children's interests, and so on.
Third, the authors report that sending a child to preschool raises his or her IQ by a little more than 4 points. Preschools that include a specific language development component raise IQ scores by more than 7 points. There were not enough studies to differentiate what made some preschools more effective than others.
Fourth, the authors report on interventions that they describe as "intensive," meaning they involved more than preschool alone. The researchers sought to significantly alter the child's environment to make it more educationally enriching. All of these studies involved low-SES children (following the well-established finding that low-SES kids have lower IQs than their better-off counterparts due to differences in opportunity. I review that literature here.)
Such interventions led to a 4 point IQ gain, and a 7 point gain if the intervention included a center-based component. The authors note the interventions have too many features to enable them to pinpoint the cause, but they suggest that the data are consistent with the hypothesis that the cognitive complexity of the environment may be critical. They were able to confidently conclude (to their and my surprise) that earlier interventions helped no more than those starting later.Those are the four interventions with the best track record. (Some others fared less well. Training working memory in young children "has yielded disappointing results."
) The data are mostly unsurprising, but I still find the article a valuable contribution. A reliable, easy-to-undertand review on an important topic. Even better,
this looks like the beginning of what the authors hope will be a longer-term effort they are calling the Database on Raising Intelligence--a compendium of RCTs based on interventions meant to boost IQ. That may not be everything we need to know about how to raise kids, but it's a darn important piece, and such a Database will be a welcome tool.
There is a great deal of attention paid to and controversy about, the promise of training working memory to improve academic skills, a topic I wrote about here
. But working memory is not the only cognitive process that might be a candidate for training. Spatial skills
are a good predictor of success in science, mathematics, and engineering. Now on the basis of a new meta-analysis (Uttal, Meadow, Tipton, Hand, Alden, Warren & Newcombe, in press) researchers claim that spatial skills are eminently trainable. In fact they claim a quite respectable average effect size of 0.47 (Hedge's g)
after training (that's across 217 studies).
Training tasks across these many studies included things like visualizing 2D and 3D objects in a CAD program, acrobatic sports training, and learning to use a laparascope (an angled device used by surgeons). Outcome measures were equally varied, and included standard psychometric measures (like a paper-folding test
), tests that demanded imagining oneself in a landscape, and tests that required mentally rotating objects.
Even more impressive:
1) researchers found robust transfer to new tasks
2) researchers found little, if any effect of delay between training and test--the skills don't seem to fade with time, at least for several weeks. (Only four studies included delays of greater than one month.)
This is a long, complex analysis and I won't try to do it justice in a brief blog post. But the marquee finding is big news. What we'd love to see is an intervention that is relatively brief, not terribly difficult to implement, reliably leads to improvement, and transfers to new academic tasks.
That's a tall order, but spatial skills may fill all the requirements.
The figure below (from the paper) is a conjecture--if spatial training were widely implemented, and once scaled up we got the average improvement we see in these studies, how many more people could be trained as engineers?
The paper is not publicly available, but there is a nice summary here
from the collaborative laboratory responsible for the work. I also recommend this excellent article from American Educator
on the relationship of spatial thinking to math and science, with suggestions for parents and teachers.
Uttal, D. H., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., Warren, C., & Newcombe, N.S. (2012, June 4). The Malleability of Spatial Skills: A Meta-Analysis of Training Studies. Psychological Bulletin
. Advance online publication. doi: 10.1037/a0028446Newcombe, N. S. (2010) Picture this: Increasing math and science learning by improving spatial thinking. American Educator, Summer,
One of the most troubling problems concerns the promotion or retention of low-achieving kids. It doesn't seem sensible to promote the child to the next grade if he's terribly far behind. But if he is asked to repeat a grade, isn't there are high likelihood that he will conclude he's not cut out for school?Until recently, comparisons of kids who were promoted and kids who were retained indicated that retention didn't seem to help academic achievement, and in fact likely hurt. So the best practice seemed to be to promote kids to the next grade, but to try to provide extra academic support for them to handle the work. But new studies indicate that academic outcomes for kids who are retained may be better than was previously thought, although still not what we would hope.A meta-analysis by Chiharu Allen and colleagues indicates that the apparent effect of retention on achievement varies depending on the particulars of the research.
Two factors were especially important. First, the extent to which researchers controlled for possible differences between retained and promoted students. Better studies ensured that groups were matched on many characteristics, whereas worse studies just used a generic "low achiever" control group. Second, some studies compared retained students to their age-matched cohort--who were now a year ahead in school. Other studies compared retained students to a grade-matched cohort or
to the grade-matched norms of a standardized test.
Which comparison is more appropriate is, to some extent, a value judgment, but personally I can't see the logic in evaluating a kids' ability to do 4th grade work (relative to other 4th graders) when he's still in 3rd grade. The authors reported three main findings:1) studies with poor controls indicated negative academic outcomes for retained students. Studies with better controls indicated no effect, positive or negative, on retention versus promotion. 2)
When compared to students in the same grade, retained children show a short term boost to academic achievement, but that advantage dissipates in the coming years. The authors speculate that students' academic self-efficacy increases in that first year, but they come to adopt beliefs that they are not academically capable. This pattern--a one-year boost followed by loss--was replicated in a recently published study (Moser, West, & Hughes, in press).The question of whether it's best to promote or retain low-achieving students
is still open. But better research methodology is providing a clearer picture of the outcomes for these students. One hopes that better information will lead to better ideas for intervention.Allen,
C. S., Chen, Q., Willson, V. L., & Hughes, J. N. (2009). Quality of research design moderates effects of grade retention on achievement: A meta-analytic, multi-level analysis. Education Evaluation & Policy Analysis, 31,
480-499.Moser, S. E., West, S. G. & Hughes, J. N. (in press). Trajectories of math and reading achievement in low-achieving children in elementary school: Effects of early and later retention in grade. Journal of Educational Psychology.
To what extent can we trust that experimental results from a psychology laboratory will be observed outside of the laboratory? This question is especially pertinent in education. It's difficult to conduct research in classrooms. It's hard to get the permission of the administration, it's hard to persuade teachers to change their practice to an experimental practice which may or may not help students--and the ethics of the request ought to be carefully considered. The researcher must make sure that the intervention is being implemented equivalently across classrooms, and across schools. And so on. Research in the laboratory is, by comparison, easy. But it's usually assumed that you give something up for this ease, namely, that the research lacks what is usually called ecological validity.
Simply put, students may not behave in the laboratory as they would in a more natural setting (often called "the field").
A recent study
sought to test the severity of this problem. The researchers combed through the psychological literature, seeking meta-analytic studies that included a comparison of findings from the laboratory and findings from the field. For example, studies have examined the spacing effect--the boost to memory from distributing practice in time--both in the laboratory and in classrooms. Do you observe the advantage in both settings? Is it equally large in both settings? The authors identified 217 such comparisons.
Each dot represents one meta-analytic comparison (so each dot really summarizes a number of studies). What this graph shows is a fairly high correlation between lab and field experiments: .639. If it worked in the lab, it generally worked in the field: only 30 times out of 215 did an effect reverse--for example, a procedure that reliably helped in the lab turned to out to reliably hurt in the field (or vice versa).
The correlation did vary by field. It was strongest in Industrial/Organizational Psychology
: there, the correlation was a whopping .89. In social psychology it was a more modest, but far from trivial .53. And what of education? There were only seven meta-analyses that went into the correlation, so it should be interpreted with some restraint, but the figure was quite close that observed in the overall dataset: the correlation was .71.So what's the upshot? Certainly, it's wise to be cautious about interpreting laboratory effects as applicable to the classroom. And I'm suspicious that the effects for which data were available were not random: in other words, there are data available for effects that researchers suspected would likely work well in the field and in the classroom. Still, this paper is a good reminder that we should not dismiss lab findings out of hand because the lab is "not the real world." These results can replicate well in the classroom. Mitchell, G. (2012). Revisiting truth or triviality: The external validity of research in the psychological laboratory. Perspectives on Psychological Science, 7, 109-117.