Education Week has published a review of a new Report: ‘Do High Fliers Maintain their Altitude?: Performance Trends of Top Students’.
The Report claims to be: ‘the first ever to examine the achievement of high-performing students over time at the individual level‘.
It is written by five staff from the North-West Evaluation Association (NWEA), most of them employed by its Kingsbury Center research institute, and published by the Thomas B Fordham Institute a non-profit think tank based in Washington DC, USA.
High achievers are defined as students who scored at or above the 90th percentile against external norms, meaning that the cohort is not confined to the ‘top 10%’ at any one time, but includes all those who exceed this benchmark.
The instrument used is the NWEA Measures of Academic Progress (MAP), a set of computerised adaptive tests in maths, reading, language usage and science aligned with national and state standards. The study deploys only the maths and reading tests.
Computerised adaptive tests are instruments whose difficulty is adjusted to reflect the learner’s performance: different students encounter different questions, with the degree of difficulty determined by how well the learner has answered the preceding questions. Such tests are not confined to specific grades or year groups, but MAP tests are usually used with Grades 2-10.
The study tracked two sets of learners, one covering grades 3 to 8 (elementary and middle schools), the other grades 6-10 (middle and high schools). The sample sizes were large. In the grades 3-8 sample there were 10,000 high-achieving learners in each of maths and reading drawn from 1,500 schools in 30 different states. The grades 6-10 sample included almost 3,000 high-achievers in maths and approaching 4,500 high achievers in reading, representing over 800 schools in 28 states.
First…the extent of movement in and out of the high-achieving group over time:
- 57.3% of high-achieving 3rd grade learners in maths were still high achieving by 8th grade, compared with 55.9% in reading. And 69.9% of high-achieving 6th grade learners in maths remained so in 10th grade, while 52.4% did so in reading. These learners are given the soubriquet ‘Steady High Fliers’.
- Conversely, 42.7% of 3rd grade learners had fallen out of the high-achieving maths cohort by 8th grade, as had 44.1% of the high achievers in reading; and 30.1% of high-achieving maths students in grade 6 were no longer so in grade 10, whereas the comparable figure in reading was 47.6%. These are called ‘Descenders’ in the study.
- At the same time, the cohort of high achievers increased between grades 3-8 and grades 6-10. The percentage of High Fliers in maths increased from 12.4% to 14.1% between grades 3 and 8. The comparable figures for reading were 11.7% to 13.3%. Between grades 6-10, the percentages of High Fliers increased markedly, from 6.7% to 11% in maths and slightly, from 9.1% to 9.7% in reading. These incoming learners are called ‘Late Bloomers’ in the study.
- Looking at the whole population in each sample, there were 7% Late Bloomers and 5.3% Descenders in grades 3-8 maths, compared with 6.8% Late Bloomers and 5.2% Descenders in reading. There were 6.3% Late Bloomers in grades 6-10 maths and 2% Descenders, while in reading there were 4.9% Late Bloomers and 4.3% Descenders. So Late Bloomers outnumbered Descenders in each of the four areas.
Second…the composition of the high-achieving group:
- Minority ethnic high achievers were under-represented, ranging between 6.7% and 9.4% of high achievers in each of the four categories. However, representation remained relatively stable between grades and even tended to increase slightly.
- Boys were over-represented in maths (56-61%) and girls typically so in reading (49-53%), but boys were more likely than girls to be Descenders and girls more likely to be Late Bloomers in both maths and reading.
- As one would expect, high-achievers were much more prevalent in low-poverty schools (80-87%) than in high-poverty schools (13-18%) and, unlike minority and female students, their proportions declined between the lower and higher grades, falling 3.3% between grades 3 and 8 in maths, though only by 0.1% between grades 3 and 8 in reading. The comparative reductions between grades 6 and 10 were 2.8% in maths and 1.9% in reading. (Low poverty schools were defined as those with less than 50% of students receiving a free or reduced price lunch.)
Third…a tendency for Late Bloomers to be relatively strong achievers already, and for Descenders to continue as relatively strong achievers:
- Most Descenders remained at or above the 70th percentile of achievement, while most Late Bloomers were already relatively high attaining learners. For example, Descenders in maths between grades 3-8 still performed at the 77th percentile, while Late Bloomers at grade 8 maths performed on average at the 74th percentile in grade 3.
Fourth…mixed evidence about the relative progress of high, middle and low achievers:
- when the High Fliers were compared with middle achievers (performing between the 45th and 54th percentiles inclusive) and low achievers (performing below the 10th percentile) they were found to progress at similar rates in maths but at ‘slightly slower rates’ in reading.
- But this is later glossed by the statement that High Fliers in reading progressed ‘about half as fast from 3rd grade to 8th grade as low achievers, reducing the gap between the two groups by over a third’. Middle achievers also reduced their performance gap by around 30%. The reduction in performance gaps between grades 6 and 10 was smaller – around 25% for both middle and low achievers. The authors note that computer adaptive testing reduces the likelihood that these findings are attributable to ceiling effects of the tests.
- The span of achievement is illustrated by the fact that, in grade 8, the mean maths scores of the low achievers did not reach the mean maths scores of the high achievers in grade 3.
Finally…some provisional evidence about school effects:
A parallel analysis was undertaken which defined high achievers as within the ‘top 10%’ of maths and reading scores within their grades and schools. This was therefore a significantly different group of students, especially in high-poverty schools.
The research followed an elementary school cohort from 3rd grade to 5th grade and a middle school cohort from 6th grade to 8th grade, with a view to identifying school-level factors influencing their progress.
The methodology was imperfect, because of the unrepresentative sample and the impact of pupil mobility, but the researchers identified a general trend for High Fliers in high-poverty schools to make the same amount of progress over time as their peers in low-poverty schools, even though high-fliers in low-poverty schools performed at a higher level.
In maths, grade 3 High Fliers in low-poverty schools performed on average at the 97th percentile and remained at the 97th percentile in 5th grade, while grade 3 High-Fliers in high-poverty schools were at the 83rd percentile on average, and at the 82nd percentile in grade 5.
However, the difference between the 83rd and 97th percentiles is estimated as representing a year’s worth of progress. So there is a significant gap between the level achieved by high achievers defined against the norms for their schools, but this does not increase over time.
Both high-poverty and low-poverty schools varied significantly according to the progress made by their high achievers. The odds that a low-poverty school would achieve stronger progress are put at slightly over 50%
A more comprehensive report is promised later in 2011 containing more findings about the impact of school-level factors on high achievers.
The study offers four key messages:
- there is good news – in the sense that the proportion of High Fliers increased as a consequence of the Late Bloomers – and bad news – in that there is arguably no good reason any student to be a Descender. The report raises issues about how to prevent the falling performance of Descenders and, conversely, asks why some less strong achievers turn into Late Bloomers while others do not.
- The fact that High Fliers in reading progressed at a lower rate than their middle and low-achieving peers may be regarded positively as evidence of gap-narrowing, or as a cause for concern. The causation may be attributable to policies such as No Child Left Behind, but there is no evidence from this study to support that hypothesis.
- The methodologically questionable results from the school-level surveys may challenge the belief that low-poverty schools ‘produce greater academic gains for students than their poorer counterparts’. It is conceivable that, for High Fliers, school effects are less significant than other factors, such as their home environment.
- The progress made by Late Bloomers ‘is not staggering’ but nor is the relatively small fall in the progress of Descenders. Learners move in and out of the top 10%, but typically remain within the top third of achievers. They are nevertheless underachieving and this needs attention.
The Report suggests that accountability systems should be revised to place emphasis on the progress of high-achieving, middle-achieving and low-achieving students – not least to challenge some ‘coasting’ low poverty schools.
Differences between schools cannot be a matter of resources, otherwise the low poverty schools would invariably out-perform high-poverty institutions.’Instead the problem seems to be one of consistency in policy and practice’.