I have posted a new entry in the embryonic Directory of Gifted Education Centres, on the Center for Talent Development (CTD) at Northwestern University in Illinois, USA.
I have posted a new entry in the embryonic Directory of Gifted Education Centres, on the Center for Talent Development (CTD) at Northwestern University in Illinois, USA.
This post is an initial review of what PISA 2009 tells us about the performance of gifted high achievers in England and other English-speaking countries compared with the countries at the top of the PISA 2009 rankings.
It concentrates on what we can deduce from the figures rather than causation: that will be addressed in subsequent posts. It examines:
The twelve countries and regions included in the analysis are the highest performers – Hong Hong (China), Korea, Taiwan, Finland and, for 2009 only, Shanghai (China) and Singapore – plus Australia, Canada, Ireland, New Zealand the UK and the USA.
I should state at the outset that I am not a statistician: this is a lay analysis and I apologise in advance for any transcription errors. Nevertheless, I hope it reveals some significant findings, including points which have received scant attention in the wider media coverage of the PISA results.
Background to PISA
The Programme for International Student Assessment (PISA) is a triennial OECD survey of the performance of 15 year-old students in science, mathematics and reading. Science was the main focus in 2006; reading is the main focus in 2009.
Fifty-seven countries took part in PISA 2006; a total of sixty-seven countries have taken part in PISA 2009. The effect of this increase in numbers on rankings should be borne in mind, especially the inclusion of very high-performing areas, notably Shanghai and Singapore.
It is also worth noting at the outset that PISA rankings do not reflect the overall numbers of students achieving specific levels: a small country that has a high percentage of its students achieving a high achievement level outscores a bigger country with a lower percentage of high achievers, even though the overall number of high achievers in the bigger country is greater.
PISA assesses reading, scientific, mathematical literacy. It is important to have a clear understanding of exactly what is being assessed, not least so we can understand to what extent this differs from the nature of our own national assessments.
If a country’s national assessments are congruent with PISA then it will be likely to perform much better in PISA than a similar country which is domestically focused on quite different priorities.
According to the PISA 2009 Assessment Framework:
‘Reading literacy…is defined in terms of students’ ability to understand, use and reflect on written text to achieve their purposes…the capacity not just to understand a text but to reflect on it, drawing on one’s own thoughts and experiences. In PISA, reading literacy is assessed in relation to the:
• Text format…continuous texts or prose organised in sentences and paragraphs…non-continuous texts that present information in other ways, such as in lists, forms, graphs, or diagrams… a range of prose forms, such as narration, exposition and argumentation…both print and electronic texts…these distinctions are based on the principle that individuals will encounter a range of written material in their civic and work-related adult life (e.g. application, forms, advertisements) and that it is not sufficient to be able to read a limited number of types of text typically encountered in school.
• Reading processes (aspects): Students are not assessed on the most basic reading skills, as it is assumed that most 15-year-old students will have acquired these. Rather, they are expected to demonstrate their proficiency in accessing and retrieving information, forming a broad general understanding of the text, interpreting it, reflecting on its contents and reflecting on its form and features.
• Situations: These are defined by the use for which the text was constructed. For example, a novel, personal letter or biography is written for people’s personal use; official documents or announcements for public use; a manual or report for occupational use; and a textbook or worksheet for educational use. Since some groups may perform better in one reading situation than in another, it is desirable to include a range of types of reading in the assessment items.
Mathematical literacy… is concerned with the ability of students to analyse, reason, and communicate ideas effectively as they pose, formulate, solve, and interpret solutions to mathematical problems in a variety of situations. The PISA mathematics assessment has, so far, been designed in relation to the:
• Mathematical content: This is defined mainly in terms of four overarching ideas (quantity, space and shape, change and relationships, and uncertainty) and only secondarily in relation to curricular strands (such as numbers, algebra and geometry).
• Mathematical processes: These are defined by individual mathematical competencies. These include the use of mathematical language, modelling and problem-solving skills…
• Situations: These are defined in terms of the ones in which mathematics is used, based on their distance from the students. The framework identifies five situations: personal, educational, occupational, public and scientific.
However, a major revision of the PISA mathematics framework is currently underway in preparation for the PISA 2012 assessment.
Scientific literacy… is defined as the ability to use scientific knowledge and processes not only to understand the natural world but to participate in decisions that affect it. The PISA science assessment is designed in relation to:
• Scientific knowledge or concepts: These constitute the links that aid understanding of related phenomena. In PISA, while the concepts are the familiar ones relating to physics, chemistry, biological sciences and earth and space sciences, they are applied to the content of the items and not just recalled.
• Scientific processes: These are centred on the ability to acquire, interpret and act upon evidence. Three such processes present in PISA relate to: 1) describing, explaining and predicting scientific phenomena, 2) understanding scientific investigation, and 3) interpreting scientific evidence and conclusions.
• Situations or contexts: These concern the application of scientific knowledge and the use of scientific processes applied. The framework identifies three main areas: science in life and health, science in Earth and environment, and science in technology.’
Defining high achievers in PISA
PISA performance scales are designed so that the average student score in OECD countries is 500 or thereabouts. Student performance is divided into 6 proficiency levels (only 5 for reading in PISA 2006), defined in terms of the competences demonstrated by students achieving that level.
In PISA 2006 in reading, the highest proficiency level 5 was achieved by 8.6% of OECD students with a lower score limit of 625.6. In PISA 2009 a level 6 was introduced (lower score limit of 698.3) which was achieved by 0.8% of OECD students. Levels 5 and 6 combined (lower score limit of 625.6) was achieved by 7.6% of OECD students. This analysis assumes therefore that levels 5 and 6 together in 2009 can be compared with level 5 in 2006.
We can conclude that overall higher level performance in OECD countries fell by 1.0% between 2006 and 2009. This may well be attributable to changes in the level of demand in the assessment framework rather than an overall dip in performance.
‘typically require the reader to make multiple inferences, comparisons and contrasts that are both detailed and precise. They require demonstration of a full and detailed understanding of one or more texts and may involve integrating information from more than one text. Tasks may require the reader to deal with unfamiliar ideas, in the presence of prominent competing information, and to generate abstract categories for interpretations. Reflect and evaluate tasks may require the reader to hypothesise about or critically evaluate a complex text on an unfamiliar topic, taking into account multiple criteria or perspectives, and applying sophisticated understandings from beyond the text. A salient condition for access and retrieve tasks at this level is precision of analysis and fine attention to detail that is inconspicuous in the texts.’
And tasks at level 5:
‘that involve retrieving information require the reader to locate and organise several pieces of deeply embedded information, inferring which information in the text is relevant. Reflective tasks require critical evaluation or hypothesis, drawing on specialised knowledge. Both interpretative and reflective tasks require a full and detailed understanding of a text whose content or form is unfamiliar. For all aspects of reading, tasks at this level typically involve dealing with concepts that are contrary to expectations.
In PISA 2006, science level 6 was achieved by 1.3% of OECD students and required a lower score limit of 707.9. Level 5 and above was achieved by 9.0% requiring a lower score of 633.3.
In 2009, these figures were: level 6 achieved by 1.1% of OECD students with a lower score limit of 707.9; level 5 and above achieved by 8.5% of OECD students with a lower score limit of 633.3.
The science framework does not seem to have changed significantly between the two assessments, so we can provisionally identify a small overall dip in higher level performance between 2006 and 2009.
Level 6 students can:
‘consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.
At Level 5, students can:
‘identify the scientific components of many complex life situations, apply both scientific concepts and knowledge about science to these situations, and can compare, select and evaluate appropriate scientific evidence for responding to life situations. Students at this level can use well-developed inquiry abilities, link knowledge appropriately and bring critical insights to situations. They can construct explanations based on evidence and arguments based on their critical analysis.
In PISA 2006 mathematics, level 6 was achieved by 3.3 % of OECD students with a lower score limit of 669.3 and level 5 and above by 13.3% of OECD students with a lower score of 607.
In PISA 2009, level 6 was achieved by 3.1% of OECD students with a lower score limit of 669.3 and level 5 and above by 12.7% of OECD students with a lower score of 607.
As with science, the framework does not appear significantly changed and so we can provisionally identify a small drop overall in the proportion of OECD students achieving these higher levels.
The PISA 2009 rubric says:
‘At ‘Level 6 students can conceptualise, generalise, and utilise information based on their investigations and modelling of complex problem situations. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understandings along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings, interpretations, arguments, and the appropriateness of these to the original situations.
‘At Level 5 students can develop and work with models for complex situations, identifying constraints and specifying assumptions. They can select, compare, and evaluate appropriate problem solving strategies for dealing with complex problems related to these models. Students at this level can work strategically using broad, well-developed thinking and reasoning skills, appropriate linked representations, symbolic and formal characterisations, and insight pertaining to these situations. They can reflect on their actions and formulate and communicate their interpretations and reasoning.’
Comparing PISA 2006 and 2009 results by country for all participants
Table 1 below compares average scores by country in PISA 2006 and PISA 2009. These are essentially the headline figures which attract most media attention and they are included here primarily for the purposes of comparison.
However, it is worth drawing attention to some key points arising from the table:
Comparing Percentages of High Achievers in PISA2009 and PISA2006
Table 2 compares the percentages of high achievers in each of our 12 countries who achieved the higher levels in reading, maths and science in 2006 and 2009 respectively.
|Level 6||Levels 5+6||Level 5||Level 6||Levels 5+6||Level 6||Levels 5+6||Level 6||Levels 5+6||Level 6||Levels 5+6|
As we have noted above, performance across all OECD countries fell slightly across the board between 2006 and 2009. Insofar as this is not attributable to changes to the assessment frameworks, we might reasonably note that the OECD’s effort in producing PISA has not of itself resulted in improved performance across OECD countries for high achievers over this 3-year period.
Comparing ranks for high achievers versus all achievers, 2006 and 2009
The third and final table compares rank positions for high achievers and all achievers in 2006 and 2009 respectively.
This comparison could also be undertaken on the basis of percentages achieving the different levels and/or the average scores achieved, but the rankings are more readily available and are a reasonable guide to changes in the relative performance of countries, if not absolute changes.
|Level 6||Levels 5+6||All||Level 5||All||Level 6||Levels 5+6||All||Level 6||Levels 5+6||All||Level 6||Levels 5+6||All||Level 6||Levels 5+6||All|
The analysis above provides some detailed pointers for future support for high achievers, but what overall assessment can we offer for each of our English-speaking countries?
In PISA 2006, Australia achieved high overall rankings in reading (7) and science (8) and a relatively good ranking in maths (13). It fell two places in the rankings in all three areas in PISA 2009, although its average score increased slightly in maths, was unchanged in science and fell significantly in maths.
In 2006, its ranking for high achievers (levels 5 and 6) was slightly higher than its overall ranking in science, but not in reading or maths. By 2009, this remained true of science and had became true of reading as well.
The percentage of higher achievers (levels 5 and 6) in reading has increased significantly between 2006 and 2009, but the equivalent percentages in maths and science remain largely unchanged, except for small improvements for the highest achievers (level 6).
Moving forward, the priorities for Australia are likely to be improvement in maths across the board and probably for relatively low achievers in reading and science.
PISA 2006 showed Canada achieving very highly overall in reading (4) and science (3) and highly in maths (7). In 2009 it fell two places in reading (6), three places in maths (10) and five places in science (8), although average scores remained unchanged in maths and fell somewhat in science and reading.
Its 2006 rankings for high achievers were significantly lower than its overall ranking in maths and science but identical in reading. In 2009, there was still little difference in the relative rankings for science and reading and now little difference in maths either, although the change in maths is attributable to a fall in overall ranking rather than an improvement for high achievers.
The percentage of higher achievers has declined in reading and in science between 2006 and 2009 but has increased slightly in maths.
Canada has a relatively ‘balanced scorecard’ and will likely continue to focus on improving its results in all three areas and all achievement levels, though maths may be a relatively higher priority.
Ireland’s overall rankings from PISA 2006 were high for reading (6) and mid-table for maths (22) and science (20). In PISA 2009 its ranking for science remained unchanged (20) but fell very significantly in maths (32) and especially reading (21). Average scores also fell significantly in maths and reading and were unchanged in science.
The 2006 rankings for higher achievers showed very little difference to overall rankings in science and reading, but somewhat lower relative rankings for high achievers in maths. The position is similar in 2009, and there is marked slippage down the rankings in reading – and to a lesser extent maths – for higher achievers as well as for all achievers.
The percentage of higher achievers has fallen significantly in maths and reading and slightly in science.
For the future, Ireland will need to reverse its downward trend in maths and reading while not neglecting improvements in science. It needs to focus on all levels of achievement, including its higher achievers.
In PISA 2006, New Zealand achieved a very high overall ranking in reading (5), a high ranking in science (7) and a relatively high ranking in maths (11). In PISA 2009, it slipped 2 places in reading and maths but retained its position in science. Average scores were unchanged in reading, fell slightly in maths and increased slightly in science.
Rankings for higher achievers in 2006 were significantly higher than overall rankings in science and slightly higher in reading and maths. By 2009 the difference between science rankings had closed somewhat, but this is attributable to slippage in the higher achieving rankings. In maths the position is broadly unchanged, but in reading the relatively higher ranking of the higher achievers is now more pronounced.
In terms of the percentages achieving higher levels, there has been little relatively change in reading, maths or science.
New Zealand is another country with a relatively ‘balanced scorecard’ but its higher achievers seem to be doing relatively well and it may wish to concentrate more on lower end of the achievement spectrum.
The UK achieved good to mid-table rankings in PISA 2006 for science (14), reading (17) and maths (24). In PISA 2009 it fell slightly in science (16) and maths (27) and significantly in reading (25). Average scores fell slightly in all three areas.
In 2006, rankings for higher achievers were significantly higher than overall rankings in science and reading, but very similar in maths. This continues to be the case in 2009 with the decline shared across achievement levels.
The percentage achieving higher levels has fallen significantly between 2006 and 2009 in science and maths, and fallen slightly in reading.
The UK has to improve in all three areas, but particularly maths and reading. High achievers must be a priority in maths especially, but effort is required across all levels of achievement to ensure that lower achievers do not improve at the expense of their higher-achieving peers.
The PISA 2006 overall rankings for the US were low to mid-table in science (29) and maths (35). No result was declared for reading because of problems with the administration of the assessment. The PISA 2009 outcomes show that the US has improved its ranking by six places in science (23) and four places in maths (31) while it achieved a ranking of 17 in reading. Average scores increased significantly in both maths and science.
2006 rankings for higher achievers were much higher than the overall ranking in science and slightly higher in maths. By 2009, the gap had narrowed in science and maths. In reading higher achievers are ranked significantly higher than the overall ranking.
The percentage achieving higher levels is little changed in science between 2006 and 2009 but there is a significant improvement in maths.
The US is moving in broadly the right direction but has to continue to improve in all three areas, especially maths. This evidence suggests that the focus should be predominantly on lower achievers – except in maths where there is a problem across the board – but, as with the UK, care is needed to ensure that higher achievers are not neglected as a result.
The UK and the US are therefore in very similar positions, but whereas the UK needs to arrest a downward trajectory, the US is already moving in the right direction.
There is an agenda for improvement in all these countries, should they choose – as the UK has done – to align their priorities firmly with those assessed by PISA and other international comparisons studies.
And this analysis has also shown that there is clear room for improvement in the performance of other world leaders, such as Finland, Hong Kong and Korea: we should take with a big pinch of salt the news headlines that say we need only emulate them to be successful.
This post considers how PISA and other international comparisons data can help to make the case for national investment in gifted and talented (G&T) education.
It is timed to anticipate the PISA 2009 results for reading, maths and science, to be published on 7 December 2010. It incorporates a review of a new report from Hanushek, Peters and Woessman on ‘US Math Performance in Global Perspective’ which draws on data from the 2006 PISA round.
The publication of PISA data is now a major international event. If previous rounds are anything to go by, PISA 2009 will generate exceptionally heavy media interest around the world.
In those countries where performance improves, there will be much backslapping and self-congratulation. Relatively new governments will claim the responsibility, conveniently disregarding the significant contribution made by their predecessors.
But if improvement has been modest, there is every chance that one’s international competitors have improved at a faster rate. In those countries where performance declines, relative to other countries or relative to the country’s own performance in the 2006 round, relatively new governments will be shifting the blame onto their predecessors – and will nevertheless remain under considerable pressure to announce major new programmes to reverse their negative trend.
High achievement in PISA 2009
The PISA 2009 Assessment Framework sets out in exhaustive detail the nature of the exercise. It confirms the inclusion of an updated assessment of reading literacy and unchanged assessments of maths and science, all undertaken at age 15. For reading literacy there are five levels of proficiency; for maths and science there are six levels.
The level 6 descriptor for maths says:
‘At Level 6 students can conceptualise, generalise, and utilise information based on their investigations and modelling of complex problem situations. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understandings along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings , interpretations , arguments, and the appropriateness of these to the original situations.’
The level 6 descriptor for science is as follows:
‘At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.’
Comparative data about the performance of the highest achievers in different countries can potentially tell us a lot about the effectiveness of their education systems in providing challenge and support to their G&T learners.
One can also analyse the composition of the high achieving cohort in each of the three fields – by gender, ethnic and socio-economic background – to draw inferences about the relationship between equity and overall achievement. There is evidence from previous rounds to suggest that high equity systems tend to secure larger percentages of high achievers, a critical point that both the US and the UK need to take on board (and on which I commented briefly in this earlier post).
High achievement in England and the USA
Readers with good memories will also recall that I drew on the comparative data about students achieving the highest PISA benchmarks, in addition to data drawn from the TIMSS and PIRLS studies, when analysing the ‘excellence gap’ in England.
My overall conclusion was that:
‘…the UK/England is above average in educating its high achievers and not atypical in terms of its excellence gap, but… lags far behind the world leaders – typically the knowledge-based economies that invest most heavily in gifted education.’
It will be fascinating to see whether or not the new PISA data will reinforce this conclusion – and whether it will support similar analyses of the performance of high achievers in the US.
One such analysis – the aforementioned study by Hanushek et al – has just been published. It met with relatively little attention in the G&T community, preoccupied as it was with the impending NAGC Annual Convention in Atlanta. but it reinforces powerfully the messages only recently conveyed by the National Science Board in its Report ‘Preparing the Next Generation of STEM Innovators’ which I summarised in this post and which were prominent at the Convention.
The authors begin by noting the US Federal Government’s commitment to STEM education. While noting that the emphasis has been placed on supporting more disadvantaged students to achieve basic achievement levels, they cite the conclusion from their own earlier studies that:
‘countries with students who perform at higher levels in math and science show larger rates of increase in economic productivity than do otherwise similar countries with lower-performing students…In short, the U.S. cannot afford to neglect high performers in our quest to bring up the bottom. Performance at the top end is no less important, and improvements at both ends reinforce each other, helping to accelerate the growth in productivity of the nation’s economy.’
They choose to concentrate on maths, rather than science or reading, because their own earlier studies show that maths achievement is particularly significant to a country’s national economic competitiveness – and because there is relatively greater international consensus over the content of the maths curriculum and the order in which concepts are introduced.
They have devised a methodology to review the performance of high-achieving maths students in each US state and in 10 selected urban districts compared with the 50+ countries engaged in the PISA 2006 maths study. This involves linking performance of high achievers on the National Assessment of Education Progress (NAEP) assessment of Grade 8 maths in 2005 and high achievers on the PISA assessment of maths (Grade 9) a year later in 2006.
In the 2005 NAEP assessment, 6.04% of 8th Grade students achieved the advanced level (NAEP assesses performance at three levels – basic, proficient and advanced). Hanushek et al use PISA 2006 data to estimate the percentage of students from other countries who would have achieved the advanced level had they taken NAEP 2005, by calculating the PISA score that secures the equivalent performance level.
I am not equipped to judge whether this methodology stands up to rigorous scrutiny. This note from the US National Center for Education Statistics would suggest that there are some significant comparability issues between PISA and NAEP which Hanushek et al do not address and which might be sufficient to call their findings into question. I leave others who are better qualified to judge.
Let us for the time being give them the benefit of the doubt, for their findings ought to be seriously worrying for US educators and the US Government:
An annex to the report contains a briefer analysis for science and reading performance, though the authors attach significant health warnings and will only say that the US is outperformed by many countries on these assessments too, although the difference is not so pronounced as in maths.
In relation to science, some of their concerns about comparability and statistical error rather call into question any self-congratulation in the UK about achieving the 3rd best score on this measure, outscoring the US significantly but being outscored in turn by Massachusetts as the leading US state, as well as Finland and New Zealand . The lowest performing state on this measure is Hawaii, which scores alongside Greece and Portugal.
In relation to reading, they can only approximate the statistical comparison summarised above ‘because PISA was maladministered within the US in 2006, no PISA results are reported for that year’. However, on this measure, the US and UK score very similarly, both outscored by 14 countries. Massachusetts comfortably exceeds them both, scoring on a par with Finland, which sits just behind Korea and New Zealand. New Mexico and Oklahoma are the poorest-performing states, sandwiched between Chile and Latvia respectively.
Why are there so few high achievers in the US?
Hanushek et al are rather coy about causation, but they provide data to show that over-concentration on basic performance levels as a consequence of the No Child Left Behind legislation is not to blame: the percentage performing at advanced level has increased noticeably over the period in question.
Indeed, whereas the percentage stood at 6.04% in 2005, it had risen to 7.9% in 2009. The authors make no reference to the likely impact on their international comparisons – and it will be interesting to see from the PISA 2009 data whether other countries have improved at a similar rate.
It will also be important to look at the socio-economic background of the high-achieving cohort – it is surprising that the authors have not done so directly in this analysis, rather than relaying on the broader proxy measure of having a parent with a college degree.
But, in terms of causality, they opine that:
‘the incapacity of American schools to bring students up to the highest level of accomplishment in mathematics is much more deep-seated than anything induced by recent federal legislation’
They suggest a multiplicity of factors may be at work, such as low aspirations and expectations, a sizeable minority ethnic population and significant immigration into the US but:
‘some of our findings point specifically to problematic elements within the nation’s schools. That even relatively advantaged groups in American society—white students and those with a parent who has a college education—do not generate a high percentage of students who achieve at the advanced level in math suggests, we submit, that schools are failing to teach students effectively… We see no sign that NCLB has been harmful to the highest-performing students. But we do fear that this policy environment leaves the impression that there is no similar need to enhance the education of those students the STEM coalition has called ‘the best and brightest’.
The situation in the US does appear dire – or at least it was so in 2005/2006. But the UK cannot afford to be any less concerned. Although some 9.0% of its students achieved the advanced level in science, that places it way behind the world’s top performers and headed by the likes of Estonia and Iceland, as well as some of our main European competitors such as France, Germany and the Netherlands.
While some of the highest rated countries, like Finland, have universally high educational standards and (until recently at least) an approach to pedagogy which rules out targeted support for gifted learners, several others – Taiwan, Hong Kong, Korea, the Netherlands – are countries that continue to invest significantly in the education of their gifted high achievers.
We urgently need a research study that examines this correlation more closely – to produce hard evidence of the impact on international comparisons of high achievers’ performance of sustained educational support for gifted learners.
There is certainly a prima facie case for arguing that it is very much in our interests to ensure that they convert their potential into high performance and so generate economic benefits for our countries in an increasingly competitive international environment.
Nations that realise this belatedly will be at a significant economic disadvantage and may never be able to catch up the ground they have lost.
Part I – STEM & G&T Education in the USA
Recent Federal Activity
Science, Technology, Engineering and Maths (STEM) education has been a prominent issue in the United States for several years. The STEM Education Coalition’s website provides an impressive list of relevant reports produced between 1996 and 2008.
More recently, the White House has sought to co-ordinate activity to support STEM education through the Educate to Innovate campaign launched in 2009. Educate to Innovate has three broad objectives:
In September 2010, the President announced an expansion of Educate to Innovate through Change the Equation, the name given to a new not-for-profit organisation established by the business community to help in ‘elevating STEM education as a national priority essential to meeting the economic challenges of this century’.
Change the Equation also has three stated goals:
To coincide with this announcement, the President’s Council of Advisors in Science and Technology (PCAST) – a group of 20 of the USA’s leading scientists and engineers established by Obama in April 2009 – published a Report to the President setting out wide-ranging policy proposals for improving STEM education.
The press release marking publication singles out some of the most significant recommendations aimed at federal government, including that they should:
The National Science Board
Not to be outdone, Some four months previously, another body, the National Science Board – the Governing Board of the National Science Foundation and Policy Advisors to the President and Congress – had published its own report which is the main focus of this post.
The NSB has 25 members from university and industry appointed by the President. The Board is responsible, along with the Director, for administering the activities of the National Science Foundation, established in 1950 to, inter alia, ‘recommend and encourage the pursuit of national policies for the promotion of research and education in science and engineering’.
The NSB had formerly published, in October 2007, a ‘National Action Plan for Addressing the Critical Needs of the US Science, Technology, Engineering and Mathematics Education System’ . This sets out wide-ranging recommendations including the formation of a National Council for STEM education and development by the NSF of a ‘national roadmap’ for STEM education.
It subsequently offered a set of recommendations on STEM education to the incoming Obama administration in January 2009 which includes the proposal that:
‘The Federal Government should ensure that we are developing the talents of all children who have the potential to become STEM innovators or excellent STEM professionals.’
This is further developed in the NSB publication ‘Preparing the Next Generation of STEM Innovators: Identifying and Developing Our Nation’s Human Capital’ which effectively joins together the G&T education and STEM agendas in the USA, pointing to the importance of ‘STEM innovators’, defined by the Report as:
‘those individuals who have developed the expertise to become leading STEM professionals and perhaps the creators of significant breakthroughs or advances in scientific and technological understanding.’
The focus of the Report can probably be traced to the fact that one of the current members of the National Science Board is Camilla P Benbow from Vanderbilt University, a psychologist and gifted education specialist perhaps best-known for her involvement with the Study of Mathematically Precocious Youth (SMPY).
The Report draws on the work of an ad hoc Task Group led by Benbow which was formed in August 2008 and an expert panel discussion held in August 2009. The Task Group was asked to identify strategies to increase the number of future STEM innovators and set out recommendations for how the NSF and federal partners might support their development.
The Arguments Advanced in the NSB Report
The Report begins by tracing interest in the development of American scientific talent back to the Second World War and the Sputnik wake-up call delivered by the Russians in the 1950s. It argues that this national sense of urgency was dissipated by the 1970s but needs urgently to be revived, given the strength of international competition.
It quotes PISA 2006 data showing that US 15 year-olds above the 90th percentile were rated only 30th in the world on maths literacy and 13th on science literacy. Furthermore, in the 2007 TIMSS study, the percentage of US 8th graders achieving the advanced benchmark in maths (6%) was dwarfed by Taiwan (45%), South Korea (40%) and Singapore (also 40%).
It also draws on data showing that the best qualified US students have for decades been disinclined to pursue undergraduate and postgraduate qualifications in STEM subjects. It references the well-known evidence base for the argument that US G&T education is neglected, including the NAGC’s ‘State of the States‘ review and the 2007 Achievement Trap report, which drew attention to the underachievement of gifted learners from minority ethnic and socio-economically disadvantaged backgrounds.
The Report cites survey and research evidence to support the contention that ‘intellectual talent often generates attitudes ranging from ambivalence to outright hostility’, going on to highlight the significant scope that exists to improve the identification and development of STEM talent:
‘More often than not, across the educational ecosystem, we see a patchwork of individual, often ad hoc provisions implemented and funded at the local level: these approaches have been instrumental for many of today’s STEM innovators and should continue. In addition, a coherent, long-term, state- or Nation-wide plan to develop the next generation of leaders in STEM is also needed. Our Nation has too often left to chance the fate of those with exceptional talent rather than ensuring widespread, systematic and appropriate opportunities to flourish.’
The NSB’s Recommendations
The Report recommends a set of policy actions and a research agenda for three ‘keystone recommendations’. While I accept that much remains unknown in the territory, I am not sure it will serve the US to invest too high a proportion of its scarce resources in generating further research studies. I have the ex-policy maker’s preference for concrete action to change the situation described in the report, so I will concentrate solely on the policy recommendations.
First, the NSB argues that the country should provide opportunities for excellence in the form of ‘co-ordinated, proactive, sustained formal and informal interventions to develop the abilities of its most talented students’ by:
Second, the US should cast a wide net to identify and develop all abilities amongst all student demographics by:
Third, the country should foster a supportive ecosystem that celebrates excellence and innovation through a positive and inclusive culture by:
What Impact will the Report Have?
The Report concludes:
‘The Board firmly believes that the recommendations set forth in this report will help to ensure a legacy of continued prosperity for the United States and engender a renewed sense of excellence in our education system, benefiting generations to come.’
Well, maybe, but this is clearly a crowded area in US policy-making with a whole host of federal interests involved. Arne Duncan’s statement on the publication of the President’s Council Report acknowledges the contribution and involvement of the NSF amongst others.
But it is not yet clear whether the NSB will be successful in persuading the federal Government to invest in G&T education, albeit with a STEM focus.
I am not well-versed in how these things are handled on the other side of the Atlantic but, had I been the federal official charged with developing national STEM education policy, I could have wished for a somewhat clearer report with:
For I fear that, while much of the Report makes eminent good sense, it is insufficiently specific about how its recommendations can be delivered.
Time will tell whether the Report will influence directly the content of US national policy on STEM education. If it can do so, it is perhaps the best hope for the reform of US G&T education, especially if the vital connection between STEM support and narrowing the ‘excellence gap’ can be sustained.
In Part 1, I sought to:
In a nutshell, the research tells us that, while mixed ability settings tends to favour the lowest achievers and setting/tracking is typically better for the highest achievers, when we look at the impact on the attainment of all pupils there is little difference in overall outcomes, because the benefits and disbenefits broadly balance out.
But, compared with mixed ability scenarios, setting/tracking is more likely to increase attainment differences between low achievers and high achievers and, because disadvantaged learners are more prevalent in lower sets/tracks, setting/tracking is relatively more likely to increase the achievement gap between rich and poor.
I argued that, when we develop gifted education policy – and even when we advocate for gifted learners – it behoves us to understand and accept this reality, rather than taking a partial view that concentrates solely on the benefits for the group whose interests we exist to serve.
The fundamental point is this: it cannot be right to insist on improvements in G&T education which can only be achieved at the expense of other learners. Instead of a simplistic either/or debate, we need to look at more sophisticated hybrid solutions that retain the benefits of the two different models and mould them into a better ‘third way’.
The more promising elements of such a ‘third way’ will begin to emerge from deeper reflection on the research findings.
Setting versus tracking/streaming
If we look at the issue from a UK perspective, there is seemingly broad professional consensus that streaming is less effective than setting. Streaming typically depends on crude distinctions that fail to recognise the full profile of a learner’s strengths and weaknesses. So it is quite likely that learners – even many gifted learners – will find themselves relatively under-challenged in some subjects and relatively over-challenged in others.
Conversely, setting enables schools to tailor their initial selection criteria and pedagogical approach to the subject in question. It is quite conceivable that some subjects will use sets while others remain mixed ability – that is standard practice in many English schools. It may also be possible to run sets and mixed ability groups in parallel in the same subject – eg a set for high achievers and/or a set for low achievers with other learners taught in mixed ability groups.
Setting also provides greater flexibility for schools to move pupils when necessary, in recognition that children develop at different rates in different subject areas, with lags and spurts at different times. By offering a more flexible, dynamic model, they are better able to respond quickly when a learner needs extra challenge or greater support in one particular part of the curriculum.
This is not to say that setting inevitably delivers these advantages – rather that effective setting practice does so. As with the identification of gifted learners, it can be all too easy for poorer schools to undertake setting as a once-and-for-all judgement rather than an arrangement kept under regular review and informed by careful analysis of individual pupils’ progress.
I am not sure to what extent setting (maybe we should call it ‘subject-specific tracking’) has been adopted in the US, though it clearly exists in some areas as an alternative to the cruder standard tracking approach. Setting does appear to offer more inherent advantages than the cruder alternative, but there is no reason in principle why elements of both could not be combined into a workable hybrid.
And we should not discard entirely the possibility of a workable model based exclusively on tracking. Advocates of both tracking and of setting are right to assert that, given the right kind of implementation, it ought to be possible to utilise either technique to raise standards for all learners while simultaneously narrowing achievement gaps. The big – and seemingly unanswered – question is how to realise that in practice.
How Big are the Attainment Differences Between High- and Low-Achievers?
Within the research canon, the scale of the performance gap between different tracks/sets is disputed territory. This is probably because of the statistical difficulties involved in isolating the effect of the pedagogy from the effect of prior achievement attributable to gender, ethnic and socio-economic background (amongst other variables).
More recent research suggest that the gap is probably less significant than was earlier believed. Some argue that it is negligible, but there does not seem to be a reliable body of evidence to support that contention. And, even if there were, this does not impact on the ‘sorting hat’ effect that is potentially so detrimental to those from disadvantaged backgrounds (an effect that we know well because it also undermines the effective identification of G&T populations the world over).
Teacher Quality and Deployment in Tracks and Sets
One might reasonably hypothesise that performance gaps between upper and lower tracks, streams and sets are much less likely to appear in schools which ensure that their more experienced subject-specialists are more regularly deployed at the lower end of the spectrum.
Conversely, given the impact of teacher quality on pupil performance, it would be no surprise to find that schools which deployed their best teachers at the upper end were more likely to see performance gaps widen.
This is not to suggest that it would be appropriate to load all the worst teachers into the upper sets, but merely that the distribution might be weighted more towards the lower end than it is now in a typical school. For it is an inconvenient truth often ignored by gifted educators that, by and large, high-achieving gifted learners tend to benefit from the typical distribution of teaching talent across sets or tracks.
It may or may not be attributable to teacher quality, but there does seem to be some research evidence that students in upper sets and tracks typically encounter a more enlightened pedagogy, with more opportunities for student engagement and interaction, whereas lower sets more often experience a more didactic ‘remedial’ teaching style.
Teacher Quality and Deployment in Mixed Ability Settings
In schools that follow a mixed ability approach, there is a different kind of issue about the distribution of pedagogical skill. In this context, the most important consideration becomes the effectiveness with which teachers can cater for widely differing needs within a single classroom.
While the best teachers may be able to differentiate their challenge and support across a mixed-ability class of 30 or more students, the critical issue is whether this can be achieved consistently by all teachers. This is straying into yet another contentious research area – and not one that I have yet explored in any detail.
So I am not equipped to challenge the orthodox anecdotal view – often heard from the parents of gifted learners in particular – that this is a bridge too far for many teachers, particularly if the school cannot afford to reduce the class to a more manageable size. So, they argue, these teachers need the pedagogical ‘crutch’ of a narrower achievement range within which to operate.
In passing, we should not forget that effective differentiation remains an issue even within a smaller top set or upper track. There may be a tendency, particularly amongst inexperienced staff, to assume that it is more acceptable to teach to the middle in a top set, so underserving those who are struggling to stay with the set and those who need extra challenge.
We must also acknowledge potential subject differences. It seems to be a widespread view amongst professionals that setting is potentially more valuable in the linear subjects such as maths and modern languages where the acquisition of subject knowledge is critical to progress, whereas in English and arts subjects it is not quite so essential to divide pupils on the basis of past achievement. Whether this view is reliably supported by the research literature is more open to question.
The case for and against detracking seems to hinge on this issue of teachers’ capacity to differentiate effectively in mixed ability settings. If they are not up to mark, it is of course the pupils at either extreme who are most likely to suffer, including the high-achieving gifted learners.
And it is high-achieving learners from disadvantaged backgrounds that are most vulnerable. They are more likely to be found in schools in disadvantaged urban areas: precisely the schools that cannot attract a critical mass of high quality teachers able to differentiate effectively in a mixed ability setting. It is another inconvenient truth that the better teachers are found disproportionately in more advantaged areas and schools, despite the redistributive impact of initiatives like Teach for America/Teach First.
This means that – contrary to the conclusions above about the overall negative impact of sets on performance gaps – detracking may unintentionally widen the excellence gap we have discussed in previous posts on this blog.
In my view, detracking is unlikely to succeed as a crude stand-alone adjustment. It would be much more likely to work as part of a sophisticated suite of reforms that supplement the support available for all learners, but especially those at either end of the achievement spectrum, whether that be by means of in-class ability grouping, pull-out for catch-up and extension, targeted 1:1 tuition, additional enrichment, independent learning opportunities…and so on.
The critical risk that proponents of gifted education must guard against is that, when such a suite is designed and implemented, all the funding and support is targeted at the lower achievers, while comparable support for the high achievers is neglected on the mistaken assumption that they will succeed regardless. That is against the spirit of personalised education and is manifestly unfair and inefficient.
A more sophisticated approach
Too often, the debate about tracking/setting and mixed ability settings is over-simplified into a straightforward either/or choice between two mutually exclusive options. But in reality the real issue is how effectively a school deploys its professional and para-professional resources and a rich pedagogical armoury to personalise learning for all its pupils.
Schools need to think carefully about the needs of all their learners and to devise solutions that will enable them to deliver improved standards for all (excellence) and a narrowing of achievement gaps (equity). The latter must not be secured by holding high-level achievement steady while the poorer performers catch up.
In the English context, as we move towards greater institutional autonomy, this requires each and every school to secure the necessary expertise to make such decisions with confidence, to evaluate their reforms properly and to adjust them in the light of the evidence.
The trick, as always, is to provide a broad, flexible and non-bureaucratic review framework that supports the weaker schools without limiting the options of the stronger ones. In an ideal world this would be mediated by a School Improvement Partner or similar, but it should also be capable of direct use by the school’s staff and governors, possibly in partnership with another school, so providing some degree of objective external scrutiny.
On the surface, the politicians seem confident in schools’ ability to deliver and are freeing them from several constraints upon their autonomy; but it would appear that they also continue to hanker after prescription in certain areas – perhaps those where they hold strong personal beliefs, or those (like setting) where they know that opinion in schools is divided. For I would be wrong to give the impression that setting in English schools is a ‘done deal’.
Once again there seems to be a dearth of reliable national data. But a crude estimate can be derived from OFSTED 2007/08 observation data quoted in a 2009 PQ answer. This suggests that only around 15% of primary lessons are organised on the basis of ability, whereas the comparable figure for the secondary sector is 45%. So, even in the secondary sector, the small majority of lessons is still likely to be mixed-ability.
There is a case for including within a review framework an expectation that setting will be considered, but only as part of a much wider-ranging assessment of school organisation, teacher deployment and classroom pedagogy. It doesn’t make sense to consider setting in splendid isolation.
As part of this process, schools would need to satisfy themselves that a decision to introduce setting would not inhibit them from narrowing achievement gaps. They might be invited to consider setting as a default element of their strategy if they have no evidence to suggest that an alternative arrangement would result in better achievement outcomes for their high achievers and their low achievers, especially (but not exclusively) those from disadvantaged backgrounds.
Some Innovation Pointers
Such a framework should overtly encourage schools to experiment and innovate, to evaluate carefully and to learn from each other’s experience.
Ironically, while mainstream US opinion seems relatively wedded to tracking, US educators (and Australian educators too) have experimented much more thoroughly than England with hybrid solutions, innovative classroom grouping techniques and other promising approaches to organisation and differentiation.
Let us hope that the greater autonomy promised to schools here allows them to be equally innovative. They might begin by giving more serious attention to the potential of vertical grouping – where pupils are organised on the basis of achievement, but across year groups rather than within them. With one or two honourable exceptions vertical grouping is not deployed here as a means of improving differentiation, but rather as a pragmatic solution to small classes in rural schools.
And we could do with significantly more English school experience of cluster grouping, where the G&T pupils (or low achievers for that matter) are concentrated in a single mixed ability class with a specially trained teacher rather than being dispersed across a range of classes, so relatively isolated from opportunities to learn with their peers.
For the research demonstrates two things above all else:
After a recent #gtchat on this subject and follow-up posts from fellow participants, I felt an urge to set out my own perspective on this vexed question.
I cannot pretend to have reviewed the voluminous research undertaken on both sides of the Atlantic, let alone worldwide, but what I have digested leads me to the comments below.
Before we summarise the research evidence, let us begin with three important pieces of context.
First – On Excellence and Equity
Regular readers will know my view that the balance between these two principles has a powerful impact on the nature of gifted education policy at all levels of our various education systems.
Like non-identical twins, excellence and equity are often found together. Dressed in various guises, they regularly find their way into politicians’ statements of their wider educational objectives. For example, England’s Secretary of State for Education maintains that his Government’s top educational priorities are ‘raising standards’ (aka excellence) and ‘narrowing gaps’ (aka equity). It wouldn’t take much effort to unearth similar statements from most of his opposite numbers around the developed world.
Politicians are often rather coy when asked to say whether one of these twins is more important to them than the other. They tend to be given equal billing regardless of party political viewpoint.
But, given a longer term perspective, they can be imagined more accurately as sitting at either end of a policy seesaw. At any particular time, one of the pair is usually in the ascendant, but it always seems as though that elevated status inevitably results in the other gaining ascendancy in its turn, though perhaps not until the next administration, or even the next but one.
Please excuse the dreadful mixing of metaphors but, as with G&T education – and other critical issues like school admission – ability grouping is sensitive territory. It forms part of the troubled borderlands of education policy where the twin principles of excellence and equity often clash and have to be reconciled (or, failing that, to coexist in a state of some tension).
Second – On Prescription and Autonomy
The squaring of excellence and equity within these educational badlands can be attempted within the declared education policy of the authority, state or country concerned, or it can be devolved to schools. This applies to ability grouping, gifted education or any similarly contentious issue where the twin issues are wrestling for ascendancy.
The second option – devolution – seemingly enables politicians to sidestep the issue, which is often attractive because the alternative centralised prescriptive approach risks leading them into hot water and/or requires a commitment of additional resources that they do not have to spare.
But it is not feasible to admit such reasons openly, so devolution tends to be justified on the grounds that the administration should not be fettering the discretion of the professionals. In the case of ability grouping, they will argue that it cannot be right for them to trespass in the secret garden of pedagogy (to use a phrase that will resonate with English educationalists). That is rightly the province of the school, which must be free to reach decisions without meddling governmental interference, taking full account of its unique circumstances and the particular needs of its pupils.
Both prescription and autonomy have their weaknesses. In the case of autonomy, the problem is that a government is typically accountable to an electorate for its education policies and will need at some point to demonstrate success in achieving its stated priorities. If too many schools act in a way that runs counter to those priorities, the government must either admit failure or – much more likely – use evidence selectively to put the best possible spin on affairs.
A sensible administration will not trust solely to their powers of communication however. If there is a chance that the collective decisions taken in schools will not deliver the outcome they stand for, they will also want to deploy the various ‘policy levers’ available to surreptitiously encourage schools towards the ‘right’ decision. Examples of such levers are the distribution of funding, or the inclusion of relevant issues within inspection and quality assurance processes, or a set of identified priorities for teachers’ professional development.
Sometimes there are inherent contradictions in a government’s policy that need to be reconciled. This is a problem regardless of whether prescription or autonomy is in the ascendant. In the case of autonomy, there is a strong risk of mixed messages, schools are divided over the best approach and policy levers are introduced that pull in different directions.
On occasions there can be policy contradictions and a conflict between prescription and autonomy. Before the recent General Election, the senior Conservative partners in what has become the Coalition Government committed to freeing teachers and schools from government interference (autonomy) and narrow gaps in performance between advantaged and disadvantaged learners (prescription, equity) and ensure that setting became much more prevalent in schools (prescription, excellence).
It will take some sleight of hand to pull off that particular trick, as we shall see. But maybe the commitment to setting has fallen by the wayside – it did not feature in the Coalition Government’s priorities for education reform and we wait to see whether it will reappear in the forthcoming Schools White Paper.
Third – On Terminology
A further similarity between ability grouping and G&T education is that different terminology is used in different parts of the world. This makes it much more likely that discussants will become disputants, because they do not share a common pedagogical language.
In the US context, one most often encounters the term ‘tracking’ and its rather inelegant antithesis ‘detracking’.
As I understand it, ‘tracking’ is typically used to mean dividing pupils into different educational programmes – either on the basis of ability or past achievement – which apply across all or most of the school experience. The equivalent term in the UK is ‘streaming’.
I cannot find recent and reliable data to illustrate how prevalent tracking is in the US. Some of the research studies imply that it has been almost universal in US high schools, at least until ‘detracking’ began in earnest around a decade ago. Conversely, while streaming was once popular in England, it now seems comparatively rare. It seems to survive in a few comprehensive schools that are competing with selective schools and so opt to establish a ‘grammar stream’ but is otherwise discarded as poor practice.
It is also unclear to what extent US tracking practice has morphed into what we in the UK call ‘setting’ – the grouping of pupils according to ability or achievement in specific subjects. As far as I can see, the stereotypical generic and tripartite tracking model in the US still seems common, but I stand to be corrected. Setting is much more prevalent in UK secondary schools, especially in the so-called ‘linear subjects’, and has also increased in primary schools, predominantly in the core subjects of English and maths.
‘Detracking’ is the process of dispensing ‘tracking’ in favour of exclusively mixed ability classes. The online literature implies that this is happening in many schools and states across the US but, yes you’ve guessed it, reliable data seems rather thin on the ground.
Tracking, setting and streaming all take place across the school, or at least across an entire year group within a school. They are therefore clearly distinct from ‘selection’ – which is the shorthand term we in the UK apply to approaches that sort pupils into different schools – and to ability grouping at the classroom level.
So, with our terminological differences out of the way, we can move on to consider what the vast body of research actually tells us about the effectiveness of tracking, detracking, streaming and setting.
Top Line Conclusions from the Research
The first and most obvious point is that the research conclusions are contested and – to be brutally honest – not always entirely objective. Put very crudely, those with an equity focus tend to marshal the arguments in such a way that helps them conclude that tracking/setting/streaming is detrimental to learners from disadvantaged backgrounds.
Conversely, those with an excellence perspective (including most gifted education researchers) tend to work towards the conclusion that it is markedly beneficial to high achievers, if not to G&T learners per se.
Both groups tend to downplay the evidence that does not fully support their position.
For a more balanced view I turned to an extensive and objective 2005 literature review for England’s education ministry. This concludes that the research worldwide can be interpreted as follows:
The terminology is tailored for a UK audience but these conclusions apply in equal measure to tracking versus mixed ability settings. Since the gains of high achievers are offset by the losses to low achievers – and middle achievers tend to do broadly the same in either scenario – neither setting nor tracking has little overall impact on the raising of standards (excellence).
However, compared with mixed ability teaching, setting and tracking tend to increase performance gaps between high achievers and low achievers per se. Moreover, because pupils from disadvantaged backgrounds are found disproportionately in the lower sets or tracks, this serves to widen socio-economic achievement gaps (equity), as well as some ethnic minority achievement gaps and the gender gap between girls and boys.
In Part 2 we will look more closely at some of the issues beneath these headlines – playing back into the mix the wider contextual issues above. But, for now, my point is this: we, as an international community of gifted educators, ought to be prepared to accept this as the broad baseline consensual position, rather than holding on to an alternative and partial version of reality that is focused too narrowly and exclusively on the needs of gifted learners.