This post considers how PISA and other international comparisons data can help to make the case for national investment in gifted and talented (G&T) education.
It is timed to anticipate the PISA 2009 results for reading, maths and science, to be published on 7 December 2010. It incorporates a review of a new report from Hanushek, Peters and Woessman on ‘US Math Performance in Global Perspective’ which draws on data from the 2006 PISA round.
The publication of PISA data is now a major international event. If previous rounds are anything to go by, PISA 2009 will generate exceptionally heavy media interest around the world.
In those countries where performance improves, there will be much backslapping and self-congratulation. Relatively new governments will claim the responsibility, conveniently disregarding the significant contribution made by their predecessors.
But if improvement has been modest, there is every chance that one’s international competitors have improved at a faster rate. In those countries where performance declines, relative to other countries or relative to the country’s own performance in the 2006 round, relatively new governments will be shifting the blame onto their predecessors – and will nevertheless remain under considerable pressure to announce major new programmes to reverse their negative trend.
High achievement in PISA 2009
The PISA 2009 Assessment Framework sets out in exhaustive detail the nature of the exercise. It confirms the inclusion of an updated assessment of reading literacy and unchanged assessments of maths and science, all undertaken at age 15. For reading literacy there are five levels of proficiency; for maths and science there are six levels.
The level 6 descriptor for maths says:
‘At Level 6 students can conceptualise, generalise, and utilise information based on their investigations and modelling of complex problem situations. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understandings along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings , interpretations , arguments, and the appropriateness of these to the original situations.’
The level 6 descriptor for science is as follows:
‘At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.’
Comparative data about the performance of the highest achievers in different countries can potentially tell us a lot about the effectiveness of their education systems in providing challenge and support to their G&T learners.
One can also analyse the composition of the high achieving cohort in each of the three fields – by gender, ethnic and socio-economic background – to draw inferences about the relationship between equity and overall achievement. There is evidence from previous rounds to suggest that high equity systems tend to secure larger percentages of high achievers, a critical point that both the US and the UK need to take on board (and on which I commented briefly in this earlier post).
High achievement in England and the USA
Readers with good memories will also recall that I drew on the comparative data about students achieving the highest PISA benchmarks, in addition to data drawn from the TIMSS and PIRLS studies, when analysing the ‘excellence gap’ in England.
My overall conclusion was that:
‘…the UK/England is above average in educating its high achievers and not atypical in terms of its excellence gap, but… lags far behind the world leaders – typically the knowledge-based economies that invest most heavily in gifted education.’
It will be fascinating to see whether or not the new PISA data will reinforce this conclusion – and whether it will support similar analyses of the performance of high achievers in the US.
One such analysis – the aforementioned study by Hanushek et al – has just been published. It met with relatively little attention in the G&T community, preoccupied as it was with the impending NAGC Annual Convention in Atlanta. but it reinforces powerfully the messages only recently conveyed by the National Science Board in its Report ‘Preparing the Next Generation of STEM Innovators’ which I summarised in this post and which were prominent at the Convention.
The authors begin by noting the US Federal Government’s commitment to STEM education. While noting that the emphasis has been placed on supporting more disadvantaged students to achieve basic achievement levels, they cite the conclusion from their own earlier studies that:
‘countries with students who perform at higher levels in math and science show larger rates of increase in economic productivity than do otherwise similar countries with lower-performing students…In short, the U.S. cannot afford to neglect high performers in our quest to bring up the bottom. Performance at the top end is no less important, and improvements at both ends reinforce each other, helping to accelerate the growth in productivity of the nation’s economy.’
They choose to concentrate on maths, rather than science or reading, because their own earlier studies show that maths achievement is particularly significant to a country’s national economic competitiveness – and because there is relatively greater international consensus over the content of the maths curriculum and the order in which concepts are introduced.
They have devised a methodology to review the performance of high-achieving maths students in each US state and in 10 selected urban districts compared with the 50+ countries engaged in the PISA 2006 maths study. This involves linking performance of high achievers on the National Assessment of Education Progress (NAEP) assessment of Grade 8 maths in 2005 and high achievers on the PISA assessment of maths (Grade 9) a year later in 2006.
In the 2005 NAEP assessment, 6.04% of 8th Grade students achieved the advanced level (NAEP assesses performance at three levels – basic, proficient and advanced). Hanushek et al use PISA 2006 data to estimate the percentage of students from other countries who would have achieved the advanced level had they taken NAEP 2005, by calculating the PISA score that secures the equivalent performance level.
I am not equipped to judge whether this methodology stands up to rigorous scrutiny. This note from the US National Center for Education Statistics would suggest that there are some significant comparability issues between PISA and NAEP which Hanushek et al do not address and which might be sufficient to call their findings into question. I leave others who are better qualified to judge.
Let us for the time being give them the benefit of the doubt, for their findings ought to be seriously worrying for US educators and the US Government:
- Overall, 30 of the 56 countries undertaking PISA 2006 secured a higher percentage of advanced students than the US. Taiwan topped the table, with 28% of students achieving this level. Hong Kong, South Korea and Finland followed. Fifteen countries achieved more than twice the US figure; the other 15 include the UK (23rd);
- 18 states exceed the US average. Massachusetts and Minnesota are some way ahead of the rest – their performance would lift them into the top twenty countries and so they would out-perform the UK. The lowest-ranked states – Mississippi, New Mexico, West Virginia and Louisiana – are out-performed by countries such as Serbia and Uruguay. Results for many states are equivalent to those of developing countries;
- The research also isolates the achievement of white US students and those whose parents have a college degree, so as to test the hypothesis that the poor US showing is attributable to underachievement amongst minority ethnic students and households with lower levels of parental support. But the percentage of high achieving students from all backgrounds exceeds the US figure for white students in 24 countries (including the UK) and 16 countries exceed the US figure for those with a degree-holding parent (although the UK is not one of them). So even when selecting a favoured sub-group from the overall population of high achievers, the comparison is unfavourable to the US;
- Thirteen states exceed the US average in respect of white students, with Massachusetts, Minnesota and New Jersey leading the field. In the case of students with a degree-holding parent, fifteen states exceed the US average, and Massachusetts and Minnesota lead the way. Mississippi brings up the rear in both cases: the percentage of advanced achievers from college educated families in that state is equivalent to the percentage of high achievers from all backgrounds in Uruguay and Bulgaria;
- Performance in selective urban districts ranges from Austin and Charlotte – which are broadly comparable with the UK – to Atlanta, Los Angeles, Chicago and Washington DC. The latter are also outperformed by Bulgaria and Uruguay, but just outscore Chile, Thailand, Romania, Brazil and Mexico.
An annex to the report contains a briefer analysis for science and reading performance, though the authors attach significant health warnings and will only say that the US is outperformed by many countries on these assessments too, although the difference is not so pronounced as in maths.
In relation to science, some of their concerns about comparability and statistical error rather call into question any self-congratulation in the UK about achieving the 3rd best score on this measure, outscoring the US significantly but being outscored in turn by Massachusetts as the leading US state, as well as Finland and New Zealand . The lowest performing state on this measure is Hawaii, which scores alongside Greece and Portugal.
In relation to reading, they can only approximate the statistical comparison summarised above ‘because PISA was maladministered within the US in 2006, no PISA results are reported for that year’. However, on this measure, the US and UK score very similarly, both outscored by 14 countries. Massachusetts comfortably exceeds them both, scoring on a par with Finland, which sits just behind Korea and New Zealand. New Mexico and Oklahoma are the poorest-performing states, sandwiched between Chile and Latvia respectively.
Why are there so few high achievers in the US?
Hanushek et al are rather coy about causation, but they provide data to show that over-concentration on basic performance levels as a consequence of the No Child Left Behind legislation is not to blame: the percentage performing at advanced level has increased noticeably over the period in question.
Indeed, whereas the percentage stood at 6.04% in 2005, it had risen to 7.9% in 2009. The authors make no reference to the likely impact on their international comparisons – and it will be interesting to see from the PISA 2009 data whether other countries have improved at a similar rate.
It will also be important to look at the socio-economic background of the high-achieving cohort – it is surprising that the authors have not done so directly in this analysis, rather than relaying on the broader proxy measure of having a parent with a college degree.
But, in terms of causality, they opine that:
‘the incapacity of American schools to bring students up to the highest level of accomplishment in mathematics is much more deep-seated than anything induced by recent federal legislation’
They suggest a multiplicity of factors may be at work, such as low aspirations and expectations, a sizeable minority ethnic population and significant immigration into the US but:
‘some of our findings point specifically to problematic elements within the nation’s schools. That even relatively advantaged groups in American society—white students and those with a parent who has a college education—do not generate a high percentage of students who achieve at the advanced level in math suggests, we submit, that schools are failing to teach students effectively… We see no sign that NCLB has been harmful to the highest-performing students. But we do fear that this policy environment leaves the impression that there is no similar need to enhance the education of those students the STEM coalition has called ‘the best and brightest’.
The situation in the US does appear dire – or at least it was so in 2005/2006. But the UK cannot afford to be any less concerned. Although some 9.0% of its students achieved the advanced level in science, that places it way behind the world’s top performers and headed by the likes of Estonia and Iceland, as well as some of our main European competitors such as France, Germany and the Netherlands.
While some of the highest rated countries, like Finland, have universally high educational standards and (until recently at least) an approach to pedagogy which rules out targeted support for gifted learners, several others – Taiwan, Hong Kong, Korea, the Netherlands – are countries that continue to invest significantly in the education of their gifted high achievers.
We urgently need a research study that examines this correlation more closely – to produce hard evidence of the impact on international comparisons of high achievers’ performance of sustained educational support for gifted learners.
There is certainly a prima facie case for arguing that it is very much in our interests to ensure that they convert their potential into high performance and so generate economic benefits for our countries in an increasingly competitive international environment.
Nations that realise this belatedly will be at a significant economic disadvantage and may never be able to catch up the ground they have lost.