I’m leery of value-added measures as a metric of individual teacher quality. Aside from the straight psychometric challenges, I’ve always worried that there’s too much that teachers give students that a VAM would miss. For example, I think of a LAUSD high school teacher who told me with considerable excitement that he had managed to get one of his kids who was near to dropping out to come to his class for five days in a row. The student wasn’t really participating in any way yet, but the teacher had a plan in mind for how to try to coax this student back to trying to engage with math just one more time, after many years of difficulty and frustration.
Who would tell this teacher “that child is a bad investment of your time?” Yet where is the VAM that will measure the value the teacher is adding to this student?
A recent paper (Master, Loeb & Wyckoff, 2017; preprint here) tried to shed light on this problem. The authors note that the short-term effects of ELA teachers are usually smaller than those of math teachers (as measured by VAM). This outcome is easy to understand from a cognitive perspective—students learn more outside of school that can be applicable to ELA tests (compared to math), and thus the contribution of a teacher and school will be relatively smaller. But the contribution of ELA and math teachers to long-term outcomes (e.g., graduation) is equivalent. Why?
One possibility is that students learn different kinds of things from each. They may learn more subject-specific content from math teachers that persist to math performance next year. But a good ELA teacher may be more likely to impart different, more persistent skills to students—they may improve their self-image as students, for example.
To examine this possibility the authors compared VAMs across years in ELA and math both within subjects and between them. In other words, if an ELA teacher was really effective, we know that effect will be observable in English class the next year. Will it be observable in math class as well?
The authors had two enormous data sets with which to investigate this question: standardized test scores from 3rd through 8th students in New York City and Miami-Dade County from 2003/04 to 2011/12.
This study, like previous studies, found that about 25 or 30% of VAM persists into the next year. In this study, those quantities were similar in math and in ELA. The startling result came when investigators used VAM in one subject to predict VAM in the other subject in subsequent years. (More precisely, they examined student achievement in year 2 in subject A, accounting for achievement in year 1 in subjects A and B, and the VAM estimates of the teachers in subjects A and B in year 1.)
As noted, about ¼ to 1/3 of the VAM in ELA from one year carried over to ELA achievement the following year. And about 46% of that effect also carried over to math achievement in Miami-Dade. In New York City, it was 70%.
But having a really effective math teacher had very little impact on ELA achievement the following year. In both districts, this carryover was around 5%.
There was good evidence that these effects persist into a third year in New York City, but in Miami-Dade the results were inconclusive, according to the researchers because measurements there were less precise.
The results point to three conclusions, given the caveats typical to this work (and often forgotten): subjects other than ELA and math were not measured, and students were not randomly assigned to teachers (and in fact, there is likely systematic bias in assignment)
First, these results help us to understand why ELA short-term VAMs are smaller than math short-term VAMs, yet the predictive value for long-run outcomes (like graduation) is the same. The ELA short-term VAMs may typically be smaller, but they contribute across subjects (which the math do not). And these cross-subject effects may last years.
Second, the authors don’t speculate much on the mechanism of transfer, but at least two routes seem plausible. First, ELA teachers may, on average, provide a bigger boost to what are usually called non-cognitive skills: self-regulation, persistence, seeing oneself as belonging in school, and so on. Second, better ELA skills—especially better reading skills like decoding, fluency, deployment of comprehension strategies, and self-monitoring of comprehension—seem likely to pay dividends in many subjects.
Third, when it comes to policy, I’ll leave the conclusion to the authors: “educators and policymakers may miss valuable information if they rely only on short-term within-subject student learning to evaluate teachers’ “value added” to student achievement.” Student achievement gains prompted by teacher X could easily be misattributed to another teacher in another subject, years later.
Who would tell this teacher “that child is a bad investment of your time?” Yet where is the VAM that will measure the value the teacher is adding to this student?
A recent paper (Master, Loeb & Wyckoff, 2017; preprint here) tried to shed light on this problem. The authors note that the short-term effects of ELA teachers are usually smaller than those of math teachers (as measured by VAM). This outcome is easy to understand from a cognitive perspective—students learn more outside of school that can be applicable to ELA tests (compared to math), and thus the contribution of a teacher and school will be relatively smaller. But the contribution of ELA and math teachers to long-term outcomes (e.g., graduation) is equivalent. Why?
One possibility is that students learn different kinds of things from each. They may learn more subject-specific content from math teachers that persist to math performance next year. But a good ELA teacher may be more likely to impart different, more persistent skills to students—they may improve their self-image as students, for example.
To examine this possibility the authors compared VAMs across years in ELA and math both within subjects and between them. In other words, if an ELA teacher was really effective, we know that effect will be observable in English class the next year. Will it be observable in math class as well?
The authors had two enormous data sets with which to investigate this question: standardized test scores from 3rd through 8th students in New York City and Miami-Dade County from 2003/04 to 2011/12.
This study, like previous studies, found that about 25 or 30% of VAM persists into the next year. In this study, those quantities were similar in math and in ELA. The startling result came when investigators used VAM in one subject to predict VAM in the other subject in subsequent years. (More precisely, they examined student achievement in year 2 in subject A, accounting for achievement in year 1 in subjects A and B, and the VAM estimates of the teachers in subjects A and B in year 1.)
As noted, about ¼ to 1/3 of the VAM in ELA from one year carried over to ELA achievement the following year. And about 46% of that effect also carried over to math achievement in Miami-Dade. In New York City, it was 70%.
But having a really effective math teacher had very little impact on ELA achievement the following year. In both districts, this carryover was around 5%.
There was good evidence that these effects persist into a third year in New York City, but in Miami-Dade the results were inconclusive, according to the researchers because measurements there were less precise.
The results point to three conclusions, given the caveats typical to this work (and often forgotten): subjects other than ELA and math were not measured, and students were not randomly assigned to teachers (and in fact, there is likely systematic bias in assignment)
First, these results help us to understand why ELA short-term VAMs are smaller than math short-term VAMs, yet the predictive value for long-run outcomes (like graduation) is the same. The ELA short-term VAMs may typically be smaller, but they contribute across subjects (which the math do not). And these cross-subject effects may last years.
Second, the authors don’t speculate much on the mechanism of transfer, but at least two routes seem plausible. First, ELA teachers may, on average, provide a bigger boost to what are usually called non-cognitive skills: self-regulation, persistence, seeing oneself as belonging in school, and so on. Second, better ELA skills—especially better reading skills like decoding, fluency, deployment of comprehension strategies, and self-monitoring of comprehension—seem likely to pay dividends in many subjects.
Third, when it comes to policy, I’ll leave the conclusion to the authors: “educators and policymakers may miss valuable information if they rely only on short-term within-subject student learning to evaluate teachers’ “value added” to student achievement.” Student achievement gains prompted by teacher X could easily be misattributed to another teacher in another subject, years later.
RSS Feed
