International benchmarking is all the rage in education policy, so I have been examining the arguments for and against, as well as contemplating its potential application to gifted education.
Benchmarking is comparing one country’s educational performance against others, with the intention of applying to it policy solutions adapted from the countries that achieve the best educational outcomes.
It is highly attractive to policy-makers, but the education research community is not quite as readily convinced.
Comparative educators have long wrestled with the key question whether policies can be transferred across national borders or are inextricably linked with the social and cultural environment in which they originated.
And there are several other problems associated with the benchmarking process, especially when it involves the derivation of policy steers from international assessment studies.
We will take a closer look at benchmarking, its popularity and the criticism it attracts, before considering to what extent it has been applied to gifted education and the case for taking this further.
I have divided the post into three parts, in recognition of its length and complexity. I’m afraid there’s not much coverage of gifted education until Part Three, but I hope the wider treatment is of interest!
Definitions and Approaches to Benchmarking
The OECD’s Programme for International Student Assessment (PISA) has become the best-known international benchmarking tool. This triennial publication compares the changing performance of education systems, over time and relative to each other, by testing a sample of 15 year-olds in literacy, maths and science.
The first assessment took place in 1997. Results from the fifth round, PISA 2009, were published in December 2010 and planning for PISA 2012 is already under way. The number of participating countries is steadily increasing: 74 took part in PISA 2009 compared with 58 three years earlier.
PISA treads a fine line between proclaiming itself the world’s foremost education benchmarking tool and making – or, more often, allowing others to make – somewhat over-ambitious claims about its value to national policy-makers.
For the OECD is typically careful to say that countries should use the results to inform their decisions, rather than slavishly transferring entire policies from the highest-scoring countries: PISA is just one source of evidence rather than a complete answer.
But sometimes the PISA ‘brand’ is advertised in such a way that national politicians and educationalists are beguiled into thinking that it offers them something more than a set of indicators to factor into their wider analysis.
The OECD press release for the 2009 Report comes a little too close to encouraging such a response:
‘PISA aims to help countries see how their school systems match up globally with regard to their quality, equity and efficiency. The best performing education systems show what others can aspire to, as well as inspire national efforts to help students to learn better, teachers to teach better, and school systems to become more effective.’
I have yet to find a PISA publication that explicitly sets out what other evidence, alongside PISA, it would be wise for national policy makers to take into account, but we shall see later that it will on occasion take care to acknowledge the limitations of PISA-based benchmarking.
One example is to be found in a September 2007 publication by McKinsey and Company, ‘How the World’s Best Performing School Systems Come Out on Top’.
This contains a foreword by Andreas Schleicher, Head of the Indicators and Analysis Division in the Education Directorate of OECD and so Director of PISA. He acknowledges the problem of causation:
‘measuring performance does not of itself lead to insights about what policy and practice can do to help students to learn better’.
before admitting that McKinsey is breaking new ground in linking together quantitative results from PISA and qualitative commentary, so enabling:
‘policy makers to learn about the features of effective systems without copying systems in their entirety’.
But he continues in such a way that we are inclined to transfer our unrealistic expectations away from PISA and onto McKinsey’s broader methodology:
‘By enabling policy-makers to examine their own education systems in the light of the best performing systems that set the standards of what can be achieved, the report provides policy-makers with a unique tool to bring about improvements in schooling….Comparative analyses of this kind will become ever more important as the best-performing education systems, not simply improvement by national standards, will increasingly become the yardstick for success.’
McKinsey examines 25 school systems around the world including 10 of the top PISA performers. The qualitative dimension of their study is derived from a literature survey and interviews with a range of experts and policy-makers.
Rather strangely, curriculum and pedagogy are set aside because ‘these subjects are well-debated in the literature’ and the focus is laid exclusively on the infrastructure of the school system.
This results in an incomplete picture of system effectiveness and, by failing to take into account key elements, inevitably distorts that picture. (We shall see later that proponents of curriculum benchmarking recognise the need for a holistic treatment and this argument surely applies both ways.)
In the absence of curriculum and pedagogy, McKinsey conclude that three policy areas have the most impact, regardless of the culture in which they are applied. These three ‘key drivers’ are:
- getting the right people to become teachers
- developing them into effective teachers and
- equipping the system to provide the best possible teaching to every learner.
The third of these is most relevant to the focus of this post. But it takes the reader only so far and no further.
We learn that high-performing systems have high expectations of what every child can achieve and then monitor performance against those expectations, intervening when they are not being met.
We are told that ‘the best systems have produced approaches to ensure that the school can compensate for the disadvantages resulting from the students’ home environment’.
This not strictly true. A more accurate statement can be found in the Executive Summary of the PISA 2009 Report, Volume 2:
‘Socio-economic disadvantage has many facets and cannot be ameliorated by education policy alone, much less in the short term….However, even if socio-economic background itself is hard to change, PISA shows that some countries succeed in reducing its impact on learning outcomes.’
Finally, these systems ensure that resources are targeted at those students who need them most, but this rather begs the question which students do need them most.
If the highest-performing systems have high expectations of all learners, then presumably all under-achievers are equally deserving, regardless of whether or not they are low achievers? We are never told.
Three years on, PISA has become more explicit in its own claims for how it can help policy makers. The 2010 publication ‘Strong Performers and Successful Reformers in Education: Lessons from PISA for the United States’ suggests that PISA can:
- show what achievements are possible in comparable education systems
- be used to set national improvement targets and reform trajectories
- allow countries to link their national assessments to PISA assessments
- help countries to validate their progress, defending themselves against allegations that improvements are attributable to falling standards
- assist countries to optimise their existing policies or consider alternative approaches.
It advances a ‘Framework for Analysis’ based on the principle that there is a continuum of approaches to education reform related to countries’ economic development.
It argues that developing countries with limited resources will be likely to have lower levels of literacy and so may choose to:
‘invest more heavily in educating well a small elite to lead the country’s industries and government operations while allocating remaining resources for teachers with little training. When teacher quality is so low, governments may also prescribe to teachers very precise job requirements, instructing teachers what to do and how to do it.’
Conversely, more advanced economies tend to focus on competing in the global economy by extending universally the education that was formerly provided only to the elite.
They need to recruit teachers from amongst the best graduates but, to do so, they must remove ‘bureaucratic command-and-control systems’ and replace them with ‘professional norms of control’ giving more professional discretion to teachers and ‘greater latitude in developing student creativity and critical thinking skills that are important to knowledge-based economies’.
All countries are said to lie somewhere along this continuum, which is presented as having the following elements:
- Economic development: from impoverished, pre-industrial low-wage to high value-added, high wage
- Teacher quality: from a few years more than lower secondary to high level professional knowledge workers
- Curriculum, instruction and assessment: from basic literacy, rote learning to complex skills, creativity
- Work organisation from hierarchical, authoritarian to flat, collegial
- Accountability: from primary accountability to authorities to primary accountability to peers and stakeholders
- Student inclusion: from the best students must learn at high levels to all students must learn at high levels
Although countries can make progress on one or more of these elements independently of the others, this is likely to pose problems:
‘adjusting only one or two dimensions at a time without concern for a more co-ordinated adaptation of the system as a whole risks tampering with the equilibrium that pervades successful systems.’
This rather confirms the inadequacy of McKinsey’s 2007 publication. While the elements of the continuum seem quite reasonable, one is left wondering about their empirical basis. We shall encounter a somewhat different list later in the argument.
‘a means for economies to analyse internal performance compared to that of other economies, identify processes and approaches of high performing education systems, and collaborate with other member economies… to learn about successful school improvement measures.’
The purpose is to:
‘improve the economic security and social well-being of member economies…to identify and replicate a set of promising policies to achieve a turnaround in the performance of consistently low-performing schools, school systems, and/or new schools.’
According to APEC, the benchmarking process involves six key steps:
- Form an expert group to identify criteria for selecting benchmark sites, develop benchmarking protocols and take decisions.
- Examine the international literature on effective school improvement strategies, including any evaluations conducted by the host countries.
- Using specified criteria, identify high performing economies with promising education policies for addressing persistently low-performing schools:
- Specify their protocols for describing and solving the identified problems.
- Identify an expert in the host country to conduct a case study.
- Bring experts together to discuss findings and extrapolate practices for adoption.
This offers us a welcome example of a more decentralised approach to benchmarking educational practice: it is not essential to base a benchmarking exercise on PISA and/or any other existing international comparisons studies.
Not to be outdone by the OECD, the World Bank is now adopting educational benchmarking.
It is currently consulting on a new Education Strategy for 2020, called ‘Learning for All’ which continues its commitment to supporting countries to achieved the education Millennium Development Goals (MDGs) building on the progress made to date but also
‘makes a significant shift toward the development of knowledge and skills that will drive people’s employability, productivity and well-being, and countries’ competitiveness and economic growth’.
The new strategy involves working to improve countries’ education systems, looking beyond inputs like schools, teachers and books, to focus on improving accountability and results.
The overview of the strategy includes a commitment to benchmarking:
‘At the country and global levels, the Bank will help develop a high-quality knowledge base to guide education systems reform. These efforts will include… new System Assessment and Benchmarking tools currently being developed. These system tools will provide detailed analysis of countries’ capacities throughout the education system, from ECD [early child development?] and teacher policy to tertiary education and skills development.
In each of these dimensions, the system tools will assess the “missing middle” of intermediate outcomes, providing information about where the results chain is breaking down. By benchmarking progress against international best practices, the tool will both highlight areas of strength and weakness and identify successful reformers [sic] that can serve as models in specific areas of education’.
These new tools are incorporated in a programme called System Assessment and Benchmarking for Education Results (SABER), which the World Bank describes as:
‘a comprehensive toolkit of system diagnostics to examine education systems and their component policy domains against global standards, best practices, and in comparison with policies and practices of countries around the world’.
SABER will look across the education policy spectrum and will review systems for equity and inclusion, as well as opportunities for tracking and extending learning, both of which may conceivably encompass gifted education (though there is as yet no reference to them doing so).
For each policy domain, the Bank will produce:
- A conceptual framework to identify the intended policy goals, the policy levers in place to secure those goals and the indicators that measure progress;
- Diagnostic tools to help assess performance on policy goals, drawing on available evidence and outcomes in the highest performing education systems;
- Country reports giving a snapshot of performance including a 1-page report card;
- Case studies to illustrate how countries have improved their performance;
- A website to share the information; and
- ‘A global education benchmarking tool drawing on a database of education leading indicators across all of the critical policy domains to give a comprehensive picture of what’s going well and what can be reformed for any country to get better results from their education system.’
In Part Two we shall look at how benchmarking is becoming influential in the United States and in England, and at the reservations that have been expressed about it.